[Fleet] Install only OTel component template mappings for otelcol inputs #237165

jsoriano · 2025-10-01T17:01:30Z

Summary

Fix https://github.com/elastic/ingest-dev/issues/6173.

Use OTel mappings more strictly in OTel integrations. Main index template is still managed by Fleet, but only otel component templates are used for mappings. Other templates are still used for Fleet-managed settings.
Packages are not expected to define fields with this approach, but they still can.

Fields, and subobjects: false are removed from test package. This needs to be adjusted in a follow up in package spec as packages are required to contain fields now.

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
Flaky Test Runner was used on any tests changed
The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
Review the backport guidelines and apply applicable backport:* labels.

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

See some risk examples
...

…no-cache --fix'

github-actions · 2025-10-06T16:41:23Z

🔍 Preview links for changed docs

docs/extend/plugin-list.md

…otel

…o install-no-templates-otel

…otel

elasticmachine · 2025-10-07T09:58:06Z

Pinging @elastic/fleet (Team:Fleet)

jsoriano · 2025-10-07T09:59:45Z

x-pack/platform/plugins/shared/fleet/server/services/epm/elasticsearch/template/install.ts

-
-  // Override `subobjects: false` to avoid conflicts with traces-otel@mappings
-  const shouldOverrideSettingsForOtelInputsTraces =
-    shouldOverrideSettingsForOtelInputs && type === 'traces' && !indexTemplateMappings.runtime;


These overrides can be avoided now that non-otel mappings are removed, and packages are not required to define fields.

jsoriano · 2025-10-07T10:03:04Z

x-pack/platform/plugins/shared/fleet/cypress/e2e/input_packages_real.cy.ts

  const agentPolicyName = 'Test input package policy';
  const packagePolicyName = 'input-package-policy';
-  const datasetName = 'check'; // Default from the package.
+  const datasetName = 'httpcheckreceiver'; // Default from the package.


About this change, see #237145 and elastic/elastic-package#2974.

criamico · 2025-10-07T12:40:46Z

@jsoriano to test this change locally, can I use the httpcheck test package in https://github.com/elastic/elastic-package/tree/main/test/packages/parallel/httpcheck? Is there a new version to use or it remains unchanged?

jsoriano · 2025-10-07T14:11:34Z

@jsoriano to test this change locally, can I use the httpcheck test package in https://github.com/elastic/elastic-package/tree/main/test/packages/parallel/httpcheck? Is there a new version to use or it remains unchanged?

This package needs changes, I am doing them in elastic/elastic-package#2971. You can use the package from that PR, or the zip included in this one (that is a renamed version of the same package).

criamico · 2025-10-08T09:03:37Z

@jsoriano I built the package from elastic/elastic-package#2971 and tried installing, but I'm seeing again that conflict error when installing the assets:

with metrics dataset:

installAssetsForInputPackagePolicy error: ResponseError: illegal_argument_exception
	Caused by:
		illegal_argument_exception: invalid composite mappings for [metrics-check]
	Root causes:
		illegal_argument_exception: composable template [metrics-check] template after composition with component templates [otel@mappings, otel@settings, metrics-otel@mappings, semconv-resource-to-ecs@mappings, metrics-check@package, metrics@custom, metrics-otel@custom, httpcheck@custom, metrics-check@custom, .fleet_globals-1, .fleet_agent_id_verification-1] is invalid
[2025-10-08T11:01:42.916+02:00][WARN ][plugins.fleet.PackagePolicyClient.create] installAssetsForInputPackagePolicy error: ResponseError: illegal_argument_exception
	Caused by:
		illegal_argument_exception: invalid composite mappings for [metrics-check]
	Root causes:
		illegal_argument_exception: composable template [metrics-check] template after composition with component templates [otel@mappings, otel@settings, metrics-otel@mappings, semconv-resource-to-ecs@mappings, metrics-check@package, metrics@custom, metrics-otel@custom, httpcheck@custom, metrics-check@custom, .fleet_globals-1, .fleet_agent_id_verification-1] is invalid

with logs dataset:

 installAssetsForInputPackagePolicy error: ResponseError: illegal_argument_exception
	Caused by:
		illegal_argument_exception: invalid composite mappings for [logs-check]
	Root causes:
		illegal_argument_exception: composable template [logs-check] template after composition with component templates [otel@mappings, otel@settings, logs-otel@mappings, semconv-resource-to-ecs@mappings, logs-check@package, logs@custom, logs-otel@custom, httpcheck@custom, logs-check@custom, .fleet_globals-1, .fleet_agent_id_verification-1] is invalid
[2025-10-08T11:04:20.849+02:00][WARN ][plugins.fleet.PackagePolicyClient.create] installAssetsForInputPackagePolicy error: ResponseError: illegal_argument_exception
	Caused by:
		illegal_argument_exception: invalid composite mappings for [logs-check]
	Root causes:
		illegal_argument_exception: composable template [logs-check] template after composition with component templates [otel@mappings, otel@settings, logs-otel@mappings, semconv-resource-to-ecs@mappings, logs-check@package, logs@custom, logs-otel@custom, httpcheck@custom, logs-check@custom, .fleet_globals-1, .fleet_agent_id_verification-1] is invalid

in both cases no assets are actually installed at the end.

Maybe something should be updated in the package itself? Otherwise we'll need to keep some of the overrides, this seems the same issue I had in my PR.

jsoriano · 2025-10-08T09:11:35Z

illegal_argument_exception: invalid composite mappings for [metrics-check]

Did you configure check as dataset? It should be using httpcheckreceiver now when using the default values. This is changed in https://github.com/elastic/elastic-package/pull/2971/files#diff-c919485fe09a8e515c05a6cdf0d8cbb60343346504e9477c83d19a9fe1393e1eR29.

A renamed version of this package is being installed by the cypress tests (using the zip in this changeset), and they are passing.

criamico · 2025-10-08T13:50:03Z

A renamed version of this package is being installed by the cypress tests (using the zip in this changeset), and they are passing.

I tested with this zip file and it worked. The previous version didn't work, even with the updated dataset name, so it must be missing something else.

criamico

LGTM 🚀 Thanks for the changes!

jsoriano · 2025-10-09T14:31:25Z

@elasticmachine merge upstream

jsoriano · 2025-10-09T14:55:08Z

The previous version didn't work, even with the updated dataset name, so it must be missing something else.

Interesting. I think elastic-package may be including leftovers from previous builds, so in this case the package built from source might still contain the removed fields files. I have created an issue about this: elastic/elastic-package#2985

kibanamachine · 2025-10-09T16:16:39Z

Starting backport for target branches: 9.2

https://github.com/elastic/kibana/actions/runs/18382119475

elasticmachine · 2025-10-09T16:16:48Z

💚 Build Succeeded

Buildkite Build
Commit: c7dc034

Metrics [docs]

✅ unchanged

History

💛 Build #348169 was flaky ae56246
💔 Build #348000 failed e1c2779
💔 Build #347893 failed a25948e
💔 Build #347024 failed adc88be
💔 Build #346712 failed 274bb27
💔 Build #346426 failed a8077a8

cc @jsoriano

…uts (elastic#237165) Use OTel mappings more strictly in OTel integrations. Main index template is still managed by Fleet, but only OTel component templates are used for mappings. Other templates are still used for Fleet-managed settings. Packages are not expected to define fields with this approach, but they still can. Fields, and `subobjects: false` are removed from test package. This needs to be adjusted in a follow up in package spec as packages are required to contain fields now. (cherry picked from commit 2a79659)

kibanamachine · 2025-10-09T16:23:38Z

💚 All backports created successfully

Status	Branch	Result
✅	9.2

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…ol inputs (#237165) (#238298) # Backport This will backport the following commits from `main` to `9.2`: - [[Fleet] Install only OTel component template mappings for otelcol inputs (#237165)](#237165)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

Install only OTel component template mappings for otelcol inputs

a8077a8

jsoriano self-assigned this Oct 1, 2025

kibanamachine and others added 5 commits October 1, 2025 17:30

[CI] Auto-commit changed files from 'node scripts/eslint_all_files --…

274bb27

…no-cache --fix'

Linting

adc88be

Merge branch 'main' into install-no-templates-otel

2802501

Linting

3f6eed3

[CI] Auto-commit changed files from 'node scripts/lint_packages --fix'

a25948e

jsoriano added 6 commits October 6, 2025 23:54

Fix tests

5626cb8

Remove override of subobjects

829dc1a

Merge remote-tracking branch 'origin/main' into install-no-templates-…

2794e0a

…otel

Merge remote-tracking branch 'jsoriano/install-no-templates-otel' int…

e1c2779

…o install-no-templates-otel

Fix cypress test

21a42ae

Merge remote-tracking branch 'origin/main' into install-no-templates-…

ae56246

…otel

jsoriano added release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team backport:version Backport to applied version labels v9.2.0 labels Oct 7, 2025

jsoriano marked this pull request as ready for review October 7, 2025 09:58

jsoriano requested a review from a team as a code owner October 7, 2025 09:58

jsoriano commented Oct 7, 2025

View reviewed changes

This was referenced Oct 7, 2025

Input packages don't require to define fields elastic/package-spec#994

Merged

Update otelcol test package elastic/elastic-package#2971

Draft

criamico approved these changes Oct 8, 2025

View reviewed changes

Merge branch 'main' into install-no-templates-otel

c7dc034

jsoriano enabled auto-merge (squash) October 9, 2025 14:32

auto-merge was automatically disabled October 9, 2025 15:31
Pull Request is not mergeable

jsoriano enabled auto-merge (squash) October 9, 2025 15:46

jsoriano merged commit 2a79659 into elastic:main Oct 9, 2025
13 checks passed

kibanamachine added the v9.3.0 label Oct 9, 2025

kibanamachine mentioned this pull request Oct 9, 2025

[9.2] [Fleet] Install only OTel component template mappings for otelcol inputs (#237165) #238298

Merged

[Fleet] Install only OTel component template mappings for otelcol inputs #237165

[Fleet] Install only OTel component template mappings for otelcol inputs #237165

Uh oh!

Conversation

jsoriano commented Oct 1, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Identify risks

Uh oh!

github-actions bot commented Oct 6, 2025

🔍 Preview links for changed docs

Uh oh!

elasticmachine commented Oct 7, 2025

Uh oh!

jsoriano Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

jsoriano Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

criamico commented Oct 7, 2025

Uh oh!

jsoriano commented Oct 7, 2025

Uh oh!

criamico commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jsoriano commented Oct 8, 2025

Uh oh!

criamico commented Oct 8, 2025

Uh oh!

criamico left a comment

Choose a reason for hiding this comment

Uh oh!

jsoriano commented Oct 9, 2025

Uh oh!

jsoriano commented Oct 9, 2025

Uh oh!

Uh oh!

kibanamachine commented Oct 9, 2025

Uh oh!

elasticmachine commented Oct 9, 2025

💚 Build Succeeded

Metrics [docs]

History

Uh oh!

kibanamachine commented Oct 9, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jsoriano commented Oct 1, 2025 •

edited by kibanamachine

Loading

criamico commented Oct 8, 2025 •

edited

Loading