[Fleet] POC - Add support for OTEL policies and integrations

[criamico]

Summary

🚧 🚧 POC - DO NOT MERGE! 🚧 🚧

Experimental API support of OTEL policies and integrations in Fleet UI

Create new OTEL integration

This endpoint allows to load a yml file containing the configs. The content is parsed to json and saved in a new SO. THis is just a way to "mock" the installation of an OTEL integration.

Usage:

curl -XPOST -H 'content-type: text/x-yaml' -H 'kbn-xsrf: true' http://localhost:5601/cri/api/fleet/otel/integrations/prometheus -u elastic:changeme --data-binary @prometheus.yml

Verify that it's saved with GET .kibana_ingest/_search?q=type:fleet-otel-integrations

Create new OTEL policy

This endpoint allows to create a new OTEL policy.

POST kbn:api/fleet/otel/policies
{
  "name": "otel-policy-2",
  "description": "Otel policy",
  "policy_ids": ["6b323736-5916-445e-a599-5c5dc5152795"],
  "integration": {
    "name": "prometheus"
  },
  "vars": {
    "access_paths": ["/var/log/nginx/access*.log"],
    "error_paths": ["/var/log/nginx/error*.log"],
    "endpoint": "http://localhost:80/status"
  },
  "pipelines": [
    "status"
  ]
}

Verify that it's saved with GET .kibana_ingest/_search?q=type:fleet-otel-policies

Retrieve full agent policy

The generated config is nested under inputs and merged with the regular inputs. It also retrieves the "installed" template from the fleet/otel/integrations and adds it under extensions. This is something that can be changed easily if needed.

Considerations

This is by no means production ready code. It needs some additional things:

• exporters that take the ES outputs and parses it to the correct OTEL format
• some metadata for the agent, which can easily be added to the Otel policy

Also the code organization needs to be improved, I hacked my way trough the code to make the POC work :)

[elasticmachine]

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

• Click to trigger kibana-pull-request for this PR!

[obltmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[jsoriano]

Interesting, does flattened work for complex data types? Or we do some additional processing for these when reading the saved object?

[criamico]

Yes it works well for complex data types and it doesn't need additional processing. We're already using it for existing mappings.

[criamico]

Closing as the POC is considered completed