{m365_defender,microsoft_defender_endpoint}: Add mapping and transform for CDR workflows #14809

kcreddy · 2025-08-05T07:10:28Z

Proposed commit message

Add support for CDR Cloud Native Vulnerability Management (CNVM)[1] workflow by 
adding necessary mappings and latest transform.

Also fixes agent degradation from httpjson's empty cursor happening in 
log data stream using "ignore_empty_value". Similar to [2].

[1] https://www.elastic.co/guide/en/security/current/vuln-management-overview.html
[2] https://github.com/elastic/integrations/pull/14525

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Run pipeline tests:
m365_defender:

--- Test results for package: m365_defender - START ---
╭───────────────┬───────────────┬───────────┬───────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE       │ DATA STREAM   │ TEST TYPE │ TEST NAME                                         │ RESULT │ TIME ELAPSED │
├───────────────┼───────────────┼───────────┼───────────────────────────────────────────────────┼────────┼──────────────┤
│ m365_defender │ vulnerability │ pipeline  │ (ingest pipeline warnings test-vulnerability.log) │ PASS   │    477.276ms │
│ m365_defender │ vulnerability │ pipeline  │ test-vulnerability.log                            │ PASS   │  104.08175ms │
╰───────────────┴───────────────┴───────────┴───────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: m365_defender - END   ---
Done

microsoft_defender_endpoint:

--- Test results for package: microsoft_defender_endpoint - START ---
╭─────────────────────────────┬───────────────┬───────────┬───────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE                     │ DATA STREAM   │ TEST TYPE │ TEST NAME                                         │ RESULT │ TIME ELAPSED │
├─────────────────────────────┼───────────────┼───────────┼───────────────────────────────────────────────────┼────────┼──────────────┤
│ microsoft_defender_endpoint │ vulnerability │ pipeline  │ (ingest pipeline warnings test-vulnerability.log) │ PASS   │ 434.634375ms │
│ microsoft_defender_endpoint │ vulnerability │ pipeline  │ test-vulnerability.log                            │ PASS   │  96.629375ms │
╰─────────────────────────────┴───────────────┴───────────┴───────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: microsoft_defender_endpoint - END   ---
Done

Run system tests:
m365_defender:

--- Test results for package: m365_defender - START ---
╭───────────────┬───────────────┬───────────┬───────────┬────────┬─────────────────╮
│ PACKAGE       │ DATA STREAM   │ TEST TYPE │ TEST NAME │ RESULT │    TIME ELAPSED │
├───────────────┼───────────────┼───────────┼───────────┼────────┼─────────────────┤
│ m365_defender │ vulnerability │ system    │ default   │ PASS   │ 2m13.608947917s │
╰───────────────┴───────────────┴───────────┴───────────┴────────┴─────────────────╯
--- Test results for package: m365_defender - END   ---
Done

microsoft_defender_endpoint:

--- Test results for package: microsoft_defender_endpoint - START ---
╭─────────────────────────────┬───────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE                     │ DATA STREAM   │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├─────────────────────────────┼───────────────┼───────────┼───────────┼────────┼───────────────┤
│ microsoft_defender_endpoint │ vulnerability │ system    │ default   │ PASS   │ 39.797611416s │
╰─────────────────────────────┴───────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: microsoft_defender_endpoint - END   ---
Done

Related issues

Screenshots

elastic-vault-github-plugin-prod · 2025-08-05T07:37:28Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

…nt-cdr

elasticmachine · 2025-08-05T14:27:38Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

kcreddy · 2025-08-05T14:38:03Z

...s/microsoft_defender_endpoint/elasticsearch/transform/latest_cdr_vuln/fields/base-fields.yml

@maxcold, for microsoft_defender_endpoint integration, I had to shorten the transform name as latest_cdr_vuln because if not, the system tests would throw an error during package installation:

Error: failed to setup system runner: can't install the package: there was an apply error: installation failed: can't install the package: could not zip-install package; API status code = 500; response body = {"statusCode":500,"error":"Internal Server Error","message":"action_request_validation_exception\n\tRoot causes:\n\t\taction_request_validation_exception: Validation Failed: 1: The id cannot contain more than 64 characters.;"}

Its coming from Kibana (logs below):

2025-08-05 19:18:07 [2025-08-05T13:48:07.229+00:00][WARN ][plugins.fleet] Error during execution of state "install_transforms" with status "failed": action_request_validation_exception 2025-08-05 19:18:07 Root causes: 2025-08-05 19:18:07 action_request_validation_exception: Validation Failed: 1: The id cannot contain more than 64 characters.; 2025-08-05 19:18:07 [2025-08-05T13:48:07.246+00:00][DEBUG][plugins.fleet] Created transform: logs-microsoft_defender_endpoint.latest_action-default-1.0.0 2025-08-05 19:18:07 [2025-08-05T13:48:07.311+00:00][DEBUG][plugins.fleet] Started transform: logs-microsoft_defender_endpoint.latest_action-default-1.0.0 2025-08-05 19:18:08 [2025-08-05T13:48:08.138+00:00][WARN ][plugins.fleet] Failure to install package [microsoft_defender_endpoint]: [ResponseError: action_request_validation_exception 2025-08-05 19:18:08 Root causes: 2025-08-05 19:18:08 action_request_validation_exception: Validation Failed: 1: The id cannot contain more than 64 characters.;] 2025-08-05 19:18:09 [2025-08-05T13:48:09.163+00:00][ERROR][plugins.fleet] Uninstalling microsoft_defender_endpoint-3.0.0 after error installing: [ResponseError: action_request_validation_exception 2025-08-05 19:18:09 Root causes: 2025-08-05 19:18:09 action_request_validation_exception: Validation Failed: 1: The id cannot contain more than 64 characters.;] with install type: install

Hope the shortened name is okay.

yes, it's ok I think. Don't know if we need to come up with a common way to name it or not. We had the same issue with our native transform https://github.com/elastic/integrations/blob/main/packages/cloud_security_posture/elasticsearch/transform/misconfiguration/manifest.yml and named it simply misconfiguration, don't know if it has some effect on the UX for the users

Great, thanks for the confirmation.

kcreddy · 2025-08-05T16:08:37Z

packages/m365_defender/manifest.yml

    subscription: basic
  kibana:
-    version: "^8.18.0 || ^9.0.0"
+    version: "^8.19.0 || ^9.1.0"


Currently both integrations are not added to permissions list in Elasticsearch.
I created the PR: elastic/elasticsearch#132445 to add them. Once approved and backported, I will update these versions accordingly. Until then the system tests will fail because of permissions error inside transform.

cc: @maxcold

I added run_as_kibana_system: false for now to test. Will send you the credentials for testing.

chrisberkhout

Looks good. I made a few comments in microsoft_defender_endpoint, but they apply to both.

...rosoft_defender_endpoint/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml

...s/microsoft_defender_endpoint/elasticsearch/transform/latest_cdr_vuln/fields/base-fields.yml

…nt-cdr

efd6 · 2025-08-12T06:53:54Z

CI seems to be having problems with docker; I've been seeing problems in beats too.

kcreddy · 2025-08-12T07:04:40Z

CI seems to be having problems with docker; I've been seeing problems in beats too.

Actually I thought it was because images for 8.19.2 are not available yet (maybe soon).
I need 8.19.2 for supporting permissions added to Elasticsearch recently: elastic/elasticsearch#132629

alexreal1314 · 2025-08-13T09:13:36Z

packages/m365_defender/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml

      copy_from: m365_defender.vulnerability.id
      ignore_empty_value: true
+  - set:
+      field: vulnerability.cve


@kcreddy it is supposed to be field: vulnerability.id, this it were we store cve data.

@alexreal1314, we also store CVE ids inside vulnerability.id field right above this processor.

integrations/packages/m365_defender/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml

Lines 689 to 693 in 2f27420

- set:

field: vulnerability.id

tag: set_vulnerability_id_from_vulnerability_id

copy_from: m365_defender.vulnerability.id

ignore_empty_value: true

In here, we are populating vulnerability.cve whenever ctx.vulnerability.id.toUpperCase().contains('CVE'). This is discussed in the spreadsheet

@kcreddy Thanks, I didn't see it as it was not part of the changes.

Why are we adding vulnerability.cve it only for some of the 3p integrations? I see it in aws inspectr, google scc, microsoft defender and here.

We should definitely be adding for all 3p integrations where applicable. It may have been missed because it wasn't Must Have. But will take a look for other integrations and add accordingly. Its also possible that for those other integrations, the vulnerbaility.id isn't given as a CVE.

alexreal1314 · 2025-08-13T09:17:59Z

packages/m365_defender/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml

          }
        }
+  - set:
+      field: vulnerability.title


@kcreddy I wonder what is the added value to the user seeing a title in this format:
'Vulnerability found in {{{package.name}}} {{{package.version}}}'

Since we already have package.name and package.version fields which appear in the flyout and data grid already and describe basically the same. This is kinda a duplication of data.
I understand that the title might be quite long - i'll check how we handle it today in the UI.

cc @maxcold

There is no field from the raw data to derive the title from. The integration previously added it from Summary: part of vulnerability.description field. But this alone is quite huge.

Sample value: The bzip2 utility, up to version 1.0.6, contains a vulnerability (CVE-2019-12900) in the BZ2_decompress function within decompress.c. This flaw involves an out-of-bounds write when processing files with numerous selectors, potentially leading to data integrity errors during decompression.

After discussing with @maxcold and @nick-alayil inside the spreadsheet, we updated it to avoid large multiline titles.

alexreal1314 · 2025-08-13T09:20:28Z

packages/m365_defender/data_stream/vulnerability/sample_event.json

    ],
    "vulnerability": {
        "classification": "CVSS",
+        "cve": "CVE-2025-3074",


should be "id": "CVE-2025-3074",

Same explanation as #14809 (comment).

alexreal1314 · 2025-08-13T09:21:40Z

packages/m365_defender/docs/README.md

    ],
    "vulnerability": {
        "classification": "CVSS",
+        "cve": "CVE-2025-3074",


should be "id": "CVE-2025-3074",

Same explanation as #14809 (comment)

alexreal1314 · 2025-08-13T09:22:45Z

...es/m365_defender/elasticsearch/transform/latest_cdr_vulnerabilities/fields/vulnerability.yml

+- name: vulnerability
+  type: group
+  fields:
+    - name: cve


- name: id

Same explanation as #14809 (comment)

alexreal1314 · 2025-08-13T09:32:11Z

packages/microsoft_defender_endpoint/elasticsearch/transform/latest_cdr_vuln/transform.yml

+      move_on_creation: true
+latest:
+  unique_key:
+    - vulnerability.id


@kcreddy seems like event.id as described here already includes vulnerability.id, package.name and package.version in that case and as we did in other integrations (rapid7, qualys, tenable) lets use this instead:

unique_key: - event.id - resource.id - data_stream.namespace

I was going for consistency with CDR guide on this. Let me know if you want me to change to event.id.

I think yes. Would like to get @maxcold opinion regarding it also.

I think yes. Would like to get @maxcold opinion regarding it also.

@maxcold, let me know if you like the unique combination changed to using event.id or the current combination (as per CDR guide) is okay.

My thinking: as in our native integrations, we rely on this combination of fields, we went with this set of fields for 3p integrations as well. Then we discovered that for some integrations where some of these fields can be multivalue, we looked for a workaround and decided to use the event.id, but before we made sure it doesn't break our flows, eg. it's not a generated value that changes for the same vulnerability.id, package.name and package.version as we would see the duplication in that case. But in my opinion, in integrations where these fields are not multivalue, it's better to use the combination instead of event.id even if event.id covers exactly these fields for the following reasons:

It's much clearer from looking at the current transform what the unique logic is. Because event.id can be anything and differ from provider to provider. So one needs to look into the ingest pipeline or even in the provider docs to understand what this event.id is, if it's unique and so on

event.id implementation (if it's created via some logic in the ingest pipeline) can change and then it can stop providing the necessary qualities to be used in the unique setting. Again, with the combination of fields, it's much more transparent

@alexreal1314 do you have any specific reason that you think event.id would be a better fit here?

kcreddy · 2025-08-14T07:51:52Z

/test

efd6 · 2025-08-14T21:59:53Z

packages/microsoft_defender_endpoint/data_stream/log/agent/stream/httpjson.yml.hbs

 cursor:
  lastUpdateTime:
-    value: "[[.last_response.body.lastUpdateTime]]"
+    value: '[[with $f := (index .last_response.body "lastUpdateTime")]][[if (ne $f "")]][[.last_response.body.lastUpdateTime]][[end]][[end]]'


I think this can be simpler.

Suggested change

value: '[[with $f := (index .last_response.body "lastUpdateTime")]][[if (ne $f "")]][[.last_response.body.lastUpdateTime]][[end]][[end]]'

value: '[[with (index .last_response.body "lastUpdateTime")]][[.]][[end]]'

From "text/template"

{{with pipeline}} T1 {{end}} If the value of the pipeline is empty, no output is generated; otherwise, dot is set to the value of the pipeline and T1 is executed.

Probably (untested), it could actually be [[index .last_response.body "lastUpdateTime"]], but I don't think it's worth it since it relies on clear knowledge of the details of the template handler in the input; the with implementation is clearer.

Thanks, that makes sense. I updated as per your suggestion 0e47471

elasticmachine · 2025-08-16T16:55:21Z

💚 Build Succeeded

Buildkite Build
Commit: 0e47471

History

💚 Build #30020 succeeded e49b77a
💔 Build #30018 failed 86b94b2
💔 Build #30016 failed 2f27420
💔 Build #29917 failed 2f27420
💔 Build #29916 failed 90bf705
💔 Build #29896 failed e340286

cc @kcreddy

elastic-sonarqube · 2025-08-16T16:55:31Z

Quality Gate failed

Failed conditions
30.2% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

elastic-vault-github-plugin-prod · 2025-08-19T17:11:33Z

Package m365_defender - 4.0.0 containing this change is available at https://epr.elastic.co/package/m365_defender/4.0.0/

elastic-vault-github-plugin-prod · 2025-08-19T17:11:34Z

Package microsoft_defender_endpoint - 3.0.0 containing this change is available at https://epr.elastic.co/package/microsoft_defender_endpoint/3.0.0/

kcreddy added 2 commits August 5, 2025 11:41

Add missing CDR mapping for m365_defender

99b4b49

Add missing CDR mapping for microsoft_defender_endpoint

9067166

Add more cloud ECS fields and changelog

4d117b6

andrewkroh added Integration:m365_defender Microsoft Defender XDR Integration:microsoft_defender_endpoint Microsoft Defender for Endpoint documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. labels Aug 5, 2025

kcreddy mentioned this pull request Aug 5, 2025

Add {m365_defender,microsoft_defender_endpoint}.vulnerability indices to kibana_system role permissions elastic/elasticsearch#132445

Merged

kcreddy added 5 commits August 5, 2025 18:48

Add transform into m365_defender

ff15d2f

Add transform into microsoft_defender_endpoint

729c82e

Merge remote-tracking branch 'upstream/main' into defender_xdr_endpoi…

d5f886d

…nt-cdr

update readme

3c4a85e

Run system tests

51d2e3c

kcreddy marked this pull request as ready for review August 5, 2025 14:27

kcreddy requested a review from a team as a code owner August 5, 2025 14:27

kcreddy added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Aug 5, 2025

kcreddy self-assigned this Aug 5, 2025

kcreddy commented Aug 5, 2025

View reviewed changes

update min kibana stack versions to match permissions PR

50b5fd1

kcreddy requested a review from a team August 5, 2025 14:44

kcreddy added the Category: CDR label Aug 5, 2025

revert stack version

4c60986

kcreddy commented Aug 5, 2025

View reviewed changes

Fix retention intervals

eb9d41e

andrewkroh added the breaking change label Aug 7, 2025

chrisberkhout reviewed Aug 8, 2025

View reviewed changes

kcreddy added 3 commits August 11, 2025 16:49

Update kibana version constraints

4473e4f

Merge remote-tracking branch 'upstream/main' into defender_xdr_endpoi…

aee7a7f

…nt-cdr

run pipeline tests

e340286

Add new lines inside transform fields

2f27420

kcreddy requested a review from chrisberkhout August 13, 2025 08:00

alexreal1314 reviewed Aug 13, 2025

View reviewed changes

kcreddy requested a review from alexreal1314 August 13, 2025 15:51

update readme

86b94b2

alexreal1314 approved these changes Aug 14, 2025

View reviewed changes

Fix log data stream httpjson system tests

e49b77a

kcreddy requested a review from efd6 August 14, 2025 10:23

kcreddy mentioned this pull request Aug 14, 2025

sonarqube: Update exclusions paths #14880

Closed

efd6 reviewed Aug 14, 2025

View reviewed changes

Address PR comment

0e47471

kcreddy requested a review from efd6 August 16, 2025 16:29

efd6 approved these changes Aug 17, 2025

View reviewed changes

chrisberkhout mentioned this pull request Aug 18, 2025

Fix link inclusion during package build elastic/elastic-package#2830

Open

chrisberkhout approved these changes Aug 18, 2025

View reviewed changes

kcreddy merged commit b40b443 into elastic:main Aug 19, 2025
8 of 9 checks passed

This was referenced Sep 26, 2025

[Internal]: Add docs for MS Defender for Endpoint Integration elastic/docs-content#3183

Open

[Internal]: Add docs for MS Defender XDR Integration elastic/docs-content#3184

Open

	- set:
	field: vulnerability.id
	tag: set_vulnerability_id_from_vulnerability_id
	copy_from: m365_defender.vulnerability.id
	ignore_empty_value: true

	value: '[[with $f := (index .last_response.body "lastUpdateTime")]][[if (ne $f "")]][[.last_response.body.lastUpdateTime]][[end]][[end]]'
	value: '[[with (index .last_response.body "lastUpdateTime")]][[.]][[end]]'

{m365_defender,microsoft_defender_endpoint}: Add mapping and transform for CDR workflows #14809

{m365_defender,microsoft_defender_endpoint}: Add mapping and transform for CDR workflows #14809

Conversation

kcreddy commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elastic-vault-github-plugin-prod bot commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Uh oh!

elasticmachine commented Aug 5, 2025

Uh oh!

kcreddy Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chrisberkhout left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 commented Aug 12, 2025

Uh oh!

kcreddy commented Aug 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexreal1314 Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy commented Aug 5, 2025 •

edited

Loading

elastic-vault-github-plugin-prod bot commented Aug 5, 2025 •

edited

Loading

kcreddy Aug 5, 2025 •

edited

Loading

kcreddy Aug 5, 2025 •

edited

Loading

alexreal1314 Aug 13, 2025 •

edited

Loading

kcreddy Aug 19, 2025 •

edited

Loading