Fix otel extension status reporting

[swiatekm]

What does this PR do?

Fix a bug where Elastic Agent would enter a failed state if components were running as beats receivers and the otel collector also had an extension defined via hybrid mode.

The result would be the following agent status:

┌─ fleet
│  └─ status: (STOPPED) Not enrolled into Fleet
└─ elastic-agent
   ├─ status: (FAILED) OTel manager failed: pipeline status id extensions is not a pipeline
   ├─ extensions
   │  ├─ status: StatusOK
   │  └─ extension:health_check/v2
   │     └─ status: StatusOK

Why is it important?

We should report status for otel extensions correctly.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in ./changelog/fragments using the changelog tool
• [ ] I have added an integration test or an E2E test

How to test this PR locally

Build agent locally and run it with the following configuration:

outputs:
  default:
    type: elasticsearch
    hosts: [127.0.0.1:9200]
    username: "elastic"
    password: ""

agent:
  monitoring:
    enabled: true
    _runtime_experimental: otel

inputs: []

# Embedded Otel configuration
receivers:
  nop:
exporters:
  nop:
extensions:
  health_check/v2:
service:
  extensions: [health_check/v2]
  pipelines:
    logs:
      receivers: [nop]
      exporters: [nop]

Then run elastic-agent status. You should see the extension status and the statuses of monitoring components:

┌─ fleet
│  └─ status: (STOPPED) Not enrolled into Fleet
└─ elastic-agent
   ├─ status: (DEGRADED) 1 or more components/units in a degraded state
   ├─ beat/metrics-monitoring
   │  ├─ status: (DEGRADED) DEGRADED
   │  └─ beat/metrics-monitoring
   │     └─ status: (DEGRADED) Elasticsearch request failed: dial tcp 127.0.0.1:9200: connect: connection refused
   ├─ filestream-monitoring
   │  ├─ status: (DEGRADED) DEGRADED
   │  └─ filestream-monitoring
   │     └─ status: (DEGRADED) Elasticsearch request failed: dial tcp 127.0.0.1:9200: connect: connection refused
   ├─ http/metrics-monitoring
   │  ├─ status: (DEGRADED) DEGRADED
   │  └─ http/metrics-monitoring
   │     └─ status: (DEGRADED) Elasticsearch request failed: dial tcp 127.0.0.1:9200: connect: connection refused
   ├─ extensions
   │  ├─ status: StatusOK
   │  └─ extension:health_check/v2
   │     └─ status: StatusOK
   └─ pipeline:logs
      ├─ status: StatusOK
      ├─ exporter:nop
      │  └─ status: StatusOK
      └─ receiver:nop
         └─ status: StatusOK

Related issues

• Closes https://github.com/elastic/ingest-dev/issues/5751

[mergify]

This pull request does not have a backport label. Could you fix it @swiatekm? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: b5a7833

Failed CI Steps

• Unit tests - Ubuntu 22.04
• Unit tests - Ubuntu 22.04
• Unit tests - Ubuntu 22.04 with requirefips build tag
• Unit tests - Ubuntu 22.04 with requirefips build tag
• Unit tests - fips140=only Ubuntu 22.04
• Unit tests - fips140=only Ubuntu 22.04
• Unit tests - Ubuntu 22.04 ARM64
• Unit tests - Ubuntu 22.04 ARM64
• Unit tests - Windows 2022
• Unit tests - Windows 2022
• Unit tests - Windows 2016
• Unit tests - Windows 2016
• Unit tests - macOS 15 ARM
• Unit tests - macOS 15 ARM
• Unit tests - Windows 10
• Unit tests - Windows 10
• Unit tests - Windows 11
• Unit tests - Windows 11

History

cc @swiatekm

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
94.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[pkoutsovasilis]

I did merge main and run this locally with subprocess mode and I still see this behaviour everytime I "kill" the collector subprocess

┌─ fleet
│  └─ status: (STOPPED) Not enrolled into Fleet
└─ elastic-agent
   ├─ status: (HEALTHY) Running
   └─ extensions
      ├─ status: StatusOK
      ├─ extension:healthcheckv2/0018a7f6-cbea-4a2f-b314-9e75eb9aa55f
      │  └─ status: StatusOK
      ├─ extension:healthcheckv2/5c5f54ad-54f4-4ed9-bf9d-a7e8a8b3a161
      │  └─ status: StatusOK
      └─ extension:healthcheckv2/6c2bbefd-f0cf-4a76-aba5-5df3412d0185
         └─ status: StatusOK

Is the above something that this PR should handle? I am still not quite sure why I even see extensions in the elastic-agent status 😄

[swiatekm]

I did merge main and run this locally with subprocess mode and I still see this behaviour everytime I "kill" the collector subprocess

┌─ fleet
│  └─ status: (STOPPED) Not enrolled into Fleet
└─ elastic-agent
   ├─ status: (HEALTHY) Running
   └─ extensions
      ├─ status: StatusOK
      ├─ extension:healthcheckv2/0018a7f6-cbea-4a2f-b314-9e75eb9aa55f
      │  └─ status: StatusOK
      ├─ extension:healthcheckv2/5c5f54ad-54f4-4ed9-bf9d-a7e8a8b3a161
      │  └─ status: StatusOK
      └─ extension:healthcheckv2/6c2bbefd-f0cf-4a76-aba5-5df3412d0185
         └─ status: StatusOK

Is the above something that this PR should handle? I am still not quite sure why I even see extensions in the elastic-agent status 😄

And this only happens in subprocess mode?

[pkoutsovasilis]

And this only happens in subprocess mode?

Is there any other case where we actively have an extension loaded at the moment?! IIUC, as this regards only hybrid elastic agent, for the embedded mode no extensions are loaded at the moment, right?

[swiatekm]

@pkoutsovasilis I found the root cause of that bug. It has nothing to do with status reporting, we're actually running multiple extensions in the subprocess collector. The reason is that we don't make a copy of the collector config at any point, so we continue adding new healthcheckv2 extensions every time the collector is restarted.

#8529 fixes this by accident because it always creates its own config file. Up to you if we want to fix it separately first.

In any case, this PR fixes a different unrelated issue, and we shouldn't block it on that one.

[pkoutsovasilis]

@pkoutsovasilis I found the root cause of that bug. It has nothing to do with status reporting, we're actually running multiple extensions in the subprocess collector. The reason is that we don't make a copy of the collector config at any point, so we continue adding new healthcheckv2 extensions every time the collector is restarted.

#8529 fixes this by accident because it always creates its own config file. Up to you if we want to fix it separately first.

In any case, this PR fixes a different unrelated issue, and we shouldn't block it on that one.

okkk now I see, under the subprocess mode we mutate the cfg but the same cfg is always mutated. If we didn't generate a random UUID for the healthcheck extension that would be fine as well I guess. Agreed let's deal with that separately