Add related.entity to cloudtrail integration

[romulets]

tldr; Adding related.entity to cloudtrail is part of the initial steps of CDR (epic overview). That enables our customers to better correlate events based on entities and enhances the investigation workflows for the Cloud Security use case.

ECS related.entity PR

Background

Elastic Cloud Security Team has been focusing, this past year, on Cloud Detection and Response (CDR). One of the first steps towards the CDR vision is to enhance investigation workflows for the Cloud Security use-case in SIEM.

As part of enhancing investigation workflows it's necessary to be able to correlate events and entities. Meaning, if an alert is triggered on the ec2 instance i-000000000, it is of great value to easily be able to search all the events related to that entity, across multiple indices, with one query. Therefore we are working on extracting entities and enabling them to be correlated.

Why related.entity

With this background, we've researched a few options on what would be the best approach to enable such feature (discussions https://github.com/elastic/security-team/issues/10026 and https://github.com/elastic/security-team/issues/9798, and outcomes https://github.com/elastic/security-team/issues/10152), and arrived at the ecs field related.

Based on the related description:

This field set is meant to facilitate pivoting around a piece of data.

Some pieces of information can be seen in many places in an ECS event.
To facilitate searching for them, store an array of all seen values to their
corresponding field in related..

To add a broad related.entity field that can hold any needed identifier to pivot data on seems to be well fitted. This would enable customers to simply run related.entity: "i-000000000" and get all the hits to that specific cloud resource.

What is an entity?

An "entity" in our context refers to any discrete component within an IT environment that can be uniquely identified and monitored. This broad term encompasses both managed and unmanaged elements.

The term "entity" is broader than the current set of available fields under related. Although ip, user and hosts can be identities, there is a lack of space to represent messaging queues, load balancers, storage systems, databases and others. Therefore the proposal to add a new field.

Related issues

• Closes https://github.com/elastic/security-team/issues/9867

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[romulets]

Leaked access key has been revoked!

[andrewkroh]

Can you please add a link to the ECS related.entity PR in the description.

Can you post before/after results of the benchmark for this pipeline (see elastic-package benchmark pipeline -d cloudtrail -v, although I'm not sure how stable / repeatable these are at such low event counts).

[andrewkroh]

I'm surprised in the amount of variance the benchmark command gives. Running it many times yields less deviation. It seems like this will cause about a 20% decrease in throughput for the pipeline. I think it would be worth running another comparison where we use script instead of foreach + append + mustache and see if that is any more performant.

Metric
Description	Main	PR	PR + Defer Deduplication	Inline processors in default.yml
Commit	6045da1	f27ceca	d9bb940	d9cb32e
Average EPS	13379.47877	8633.592375	8681.124581	11405.02769
Stddev	766.3662518	446.2111618	436.8650981	701.414431
Min	10893.24619	7251.631617	7692.307692	8340.28357
Max	14705.88235	9225.092251	9398.496241	12091.89843
EPS Change to baseline		-21.56%	-21.30%	-7.97%

Methodology

Add config file to use more events in each simulation.

# aws/data_stream/cloudtrail/_dev/test/pipeline/config.yml
---
num_docs: 10000

Run the benchmark a few times.

for i in {1..30}; do elastic-package benchmark pipeline -v -d cloudtrail --report-format json --report-output file; done

Copy the EPS rate from the results.

jq '.test[0].result[1].value' /Users/akroh/code/elastic/integrations/build/benchmark-results/* | pbcopy

Cleanup.

rm -f /Users/akroh/code/elastic/integrations/build/benchmark-results/*

Paste data into spreadsheet.

---

(edit 1: added another column to the table where processors were inlined)

[romulets]

Thanks a lot for the contributions and tips @andrewkroh!

I ran the benchmarks with 100 cycles, because 15 was still not giving consistent results, with 100 I could get it (and it's fast enough to reproduce).

On this spreadsheet you can find the raw data and stats.

The results:

Code	main	PR	Three brackets	Painless Script
Commit	076cfec	3f5ac7f	36baf08	8d85643
Average EPS	16209.72785	10723.17927	10930.95625	13152.65734
Stddev	953.7813683	565.3165502	524.9144185	879.7901167
Min	11111.11111	9090.90909	9345.79439	9708.73786
Max	17543.85965	11494.25287	11494.25287	14285.71429
EPS AVG Change to baseline	-	-33.85%	-32.57%	-18.86%
EPS Max Change to baseline	-	-34.48%	-34.48%	-18.57%

With that, we can see that the painless script has less impact on performance.

I committed the fastest solution to this PR.

Obs: Right now, I'm sorting the entities list only for tests purposes, there is no need to do so on production data. That consumes a some processing and it's unfortunate to have it for testing purposes only. What I usually do is to sort data on the test layer, but on this case that's not possible. Let me know if you have other ideas about it @andrewkroh .

[andrewkroh]

@romulets Can you please perform one more test where you inline the processors into default.yml. See my updated comment in #11115 (comment). Possibly that combined with your usage of script could yield even better results.

---

Obs: Right now, I'm sorting the entities list only for tests purposes, there is no need to do so on production data. That consumes a some processing and it's unfortunate to have it for testing purposes only.

We could propose a change to elastic-package to handle this. How much does the sorting cost?

[romulets]

@andrewkroh Indeed, the gains are relevant. I committed the changes.

Updated benchmarks:

Code	main	PR	Three brackets	Painless Script	Inline in default.yml
Commit	076cfec0	3f5ac7f0	36baf08b	8d85643d	3f819532
Average EPS	16209.72785	10723.17927	10930.95625	13152.65734	15102.77816
Stddev	953.7813683	565.3165502	524.9144185	879.7901167	772.4165627
Min	11111.11111	9090.90909	9345.79439	9708.73786	12195.12195
Max	17543.85965	11494.25287	11494.25287	14285.71429	16129.03226
EPS AVG Change to baseline		-33.85%	-32.57%	-18.86%	-6.83%
EPS Max Change to baseline		-34.48%	-34.48%	-18.57%	-8.06%

[andrewkroh]

LGTM. Thanks for going to the performance testing to get to an optimized version.

[devamanv]

Left a couple of comments. Rest looks good to me.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 7dca73f

History

• 💚 Build #16458 succeeded 083df8f
• 💚 Build #16399 succeeded 29afaca
• 💚 Build #16289 succeeded 197a92b
• 💚 Build #16217 succeeded 325ab7c
• 💔 Build #16215 failed 0947aa6572aab30292ddcda53b0f92f4dd62963a
• 💚 Build #16185 succeeded 3f8195320dda65dc7542e85a0dc789e422e83211

cc @romulets

[elastic-sonarqube]

Quality Gate failed

Failed conditions
8.1% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[elastic-vault-github-plugin-prod]

Package aws - 2.25.0-preview01 containing this change is available at https://epr.elastic.co/search?package=aws

[efd6]

I think this should have been 2.26.0-preview01.

[elastic-vault-github-plugin-prod]

Package aws - 2.26.0-preview01 containing this change is available at https://epr.elastic.co/search?package=aws