[Feature Request] Make pull-based ingestion work with OTel collector out of the box

[andrross]

Is your feature request related to a problem? Please describe

The OpenTelemetry collector can output to Kafka. I would like to be able to consume this data directly using pull-based ingestion without using a second application as I do not need to transform or enrich the data before indexing into OpenSearch.

Describe the solution you'd like

Make pull-based ingestion able to understand the OpenTelemetry format that is written to Kafka.

Related component

Indexing

Describe alternatives you've considered

It may be possible to do the transformation in the OpenTelemetry collector to make it output the message format that pull-based ingestion understands. This would also solve this problem.

Additional context

See #19374 for more discussion on this topic

[andrross]

FYI @varunbharadwaj @ps48

[ps48]

@andrross Adding some findings and updates from my PoC of using Kafka & OpenSearch pull-based ingestion for OTLP telemetry data. The transformation cannot happen inside the OTel collector afaik for top levele fields, hence used KsqlDB for in-flight streaming transformation between Kafka producer and consumer. Added some sample code here: https://github.com/ps48/otel-kafka-bridge. FYI, this transformation is still nascent doesn't completely de-serialize the data.

We cannot do this processing inside the OTel Collector for the below reasons:

1. Transform Processor Limitation: Works within the OTLP data model - it can modify spans, attributes, etc., but cannot restructure the entire message into the OpenSearch-required wrapper format.

2. Kafka Exporter Limitation: The encoding options (otlp_json, jaeger_json, etc.) produce telemetry-specific formats, not the OpenSearch ingestion format with _id, _version, _source, _op_type wrapper.

We can make this work in OTel collector with a new custom processor and enhancing the existing Kafka exporter (this not recommended).

[varunbharadwaj]

Thanks @ps48 for the detailed PoC. This is great! This is also a good example on transforming any given format (not just OTEL logs) into the current format expected by pull-based ingestion. I noticed the _id field containing unix timestamp is always unique. If this is true, it should be possible to set _op_type to create for better performance - just mentioning in case you wanted to try it out.

If you feel it is still worth exploring native support for this format in pull-based ingestion, we could try it out. It might make it a bit simpler to setup. But I believe your PoC can be generalized to support multiple formats not limited to OTEL which could be very powerful compared to native support.

[timojohlo]

Thank you @ps48 for your support! While this adds another component to the pipeline, its really cool to see that this works with the pull-based ingestion. I will look into it as well.

ksqlDB is a under Confluent Community License, so this would not work for us. Also adding another layer (same as the DataPrepper) is something we would like to avoid, so favoring for native support, as @varunbharadwaj described.

[timojohlo]

Hi,
just quickly wanted to catch up on the idea of native support. Is this something you plan to work on @ps48?

[varunbharadwaj]

@timojohlo I'll try out a quick PoC and get back, to see if this is possible.

[ps48]

@timojohlo I don't have this on my roadmap right now. We can wait for the PoC from @varunbharadwaj for the feasibility part with the current pull based ingestion framework.