Add OTLP metrics endpoint

[felixbarny]

Adds an OTLP endpoint for metric ingestion into TSDB.

This simplifies getting data into TSDB as configuration of mappings and dimensions isn't required. It also improves ingestion performance. Partly due to the binary encoding but also because it can re-use partial TsidBuilders for common resource attributes.

My aim was to retain exactly the same semantics and behavior as the OTel collector's Elasticsearch exporter, including data stream routing and mapping hints.

Depends on

• Optimize IP field parsing #132463
• TSDB ingest performance: combine routing and tsdb hashing #132566
• Add metrics OTLP endpoint skeleton #133296
• Generate Java classes from OTLP proto files #133451
• OTLP: add support for counters and gauges #133702
• OTLP: create metric documents #133898
• OTLP: ignore duplicate metric data points #133899
• OTLP: add support for histograms #133902
• OTLP: determine target data stream based on attributes and scope name #133903
• OTLP: add mapping hints #133905
• OTLP: add support for summaries #133906
• OTLP: rest and transport action #133907
• OTLP: reuse hasher for creating _metric_names_hash #133941
• Set storage engine as codeowner for the OTLP endpoint #133957
• OTLP: add test asserting _tsid is the same as _bulk #134283

To review, select from the 3rd commit (9cea250) onwards
Note that I might force-push to the branch once the PRs this depends on get merged.

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[ruflin]

Did a quick test, and so far seems to be working as expected. For everyone that needs a config to do a quick test run with the otel collector:

receivers:
  hostmetrics:
    collection_interval: 10s
    scrapers:
      cpu:
      disk:
      load:
      filesystem:
      memory:
      network:
      #process:
      paging:

processors:
  batch:


exporters:
  otlphttp/elasticsearch:
    compression: gzip
    encoding: proto
    endpoint: http://localhost:9200/_otlp
    sending_queue:
      enabled: true
      block_on_overflow: true
      queue_size: 10
      num_consumers: 10

service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [batch]
      exporters: [otlphttp/elasticsearch] # only use with batch processor

[JonasKunz]

Had a look at the histogram stuff only

[felixbarny]

Functionality-wise, everything has been merged. There are some outstanding performance improvements as part of #132566 but these are fully decoupled from the functionality.

Currently, the endpoint is behind a feature flag, meaning that it's enabled by default on snapshot builds.

[ezimuel]

@felixbarny the _otlp endpoint will be private or public?

[felixbarny]

The idea is that on serverless, we hide the endpoint in ES so that the only OTLP endpoint is the one based on the managed OTel collector. For on-prem deployments, this endpoint is publicly available.

At this instant, the endpoint is behind a feature flag. This means it's only available in snapshot builds. The plan is to release this as tech preview in 9.2.

[ezimuel]

@felixbarny thanks, I was wondering if this endpoint will be added to the elasticsearch-specification. In that case, I suggest to mark it as "visibility": "private" since, if I understood right, the _otlp endpoint is more for (internal) optimization rather than an alternative of the EDOT collector.

[felixbarny]

What does visibility: private exactly do?

I was planning to add this to elasticsearch-specification but only as a means to document the endpoint, not for language client code generation.

I think it makes sense to document the endpoint and mention in the docs that in serverless, the managed OTLP endpoint should be used.

if I understood right, the _otlp endpoint is more for (internal) optimization rather than an alternative of the EDOT collector.

Completely agree. This is not an alternative to EDOT or an OTel collector. But OTel collectors can use this endpoint with the otlphttp receiver.

[ezimuel]

The private visibility means the endpoint will not be available in the clients.

[felixbarny]

Does this have any impact on the documentation? IIUC, the API docs also get generated from the es spec. Would setting this to private omit the endpoint from the docs?

[ezimuel]

The endpoints marked as private will not been considered for the docs.

[felixbarny]

Then this is probably not the right flag for us to use. We want to have this endpoint documented.

[mlunadia]

@felixbarny what would be the effect of someone sending a considerable amount of metrics to this endpoint on a considerably large ECH cluster?

[felixbarny]

The effect would be similar to sending the same metrics to the _bulk endpoint. In both cases we prefer larger batches rather than many small ones.