Add support for running EDOT inside of running Elastic Agent

[blakerouse]

What does this PR do?

Adds the ability to run the EDOT along side the running Elastic Agent.

This connects the EDOT into the coordinator of the Elastic Agent. At any point if any of these top-level keys (receivers, processors, exporters, extensions, service) exist in the configuration or policy for the elastic-agent the EDOT is started. If all of those keys are removed from the configuration or policy then the EDOT is automatically stopped. If any configuration change occurs the updated configuration is passed along to the EDOT to handle.

Why is it important?

This allows EDOT configuration to exist inside of the configuration or policy and allow it to work as expected.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

This is an addition and doesn't affect the way the current Elastic Agent runs at all.

How to test this PR locally

Place OTel configuration into the elastic-agent.yml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
processors:
  batch:

exporters:
  otlp:
    endpoint: otelcol:4317

extensions:
  health_check:
  pprof:

service:
  extensions: [health_check, pprof]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

Run elastic-agent run -e.

Related issues

Closes #5796

[mergify]

This pull request does not have a backport label. Could you fix it @blakerouse? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-v8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b otel-hybrid upstream/otel-hybrid
git merge upstream/main
git push upstream otel-hybrid

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b otel-hybrid upstream/otel-hybrid
git merge upstream/main
git push upstream otel-hybrid

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b otel-hybrid upstream/otel-hybrid
git merge upstream/main
git push upstream otel-hybrid

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b otel-hybrid upstream/otel-hybrid
git merge upstream/main
git push upstream otel-hybrid

[blakerouse]

This is ready for a review. Don't be scare too much by the size of this change, most of that comes from NOTICE.txt, the generate control protocol and the rename of Unpack to UnpackTo.

To answer the incoming question of why rename Unpack to UnpackTo is that the Unpacker interface used by go-ucfg has the function named Unpack so that collides. This required the rename for the configuration to work correctly.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elastic-sonarqube]

Quality Gate passed

Issues
11 New issues
2 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
61.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube