Add elastic json encoding to kafka log exporter

[goto1134]

Is your feature request related to a problem? Please describe.
In my infrastructure we use kafka to collect logs in elastic json format.

Describe the solution you'd like
I would like to configure elastic marshaller for kafka log exporter and send logs in elastic format.

[jkowall]

@goto1134 Can you explain the flow? I assume you log to Kafka, and then you want the collector to take the logs from Kafka and write them to ElasticSearch?

[goto1134]

I would like to

1. send logs via opentelemetry javaagent to the opentelemetry-collector using otlp protocol
2. use custom processors to add attributes and other stuff
3. in the collector send these logs to a Kafka topic in elastic json format

[jkowall]

You can already do this with the Kafka exporter found here: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/kafkaexporter

As far as what "elastic json format" is I am not sure what you mean. The collector will work with JSON in OLTP format, but it doesn't know the "elastic json format" specifically.

[MovieStoreGuy]

A reference to expected format would help here @goto1134 and I happy to consider it, but I will warn you that adding json exported types to the collector requires a lot more resources compared to consuming protobuf; hence the otlp-json encoding is not recommended for production use.

[cyrille-leclerc]

Hello @goto1134 . Elastic v8.0 offers native support for OpenTelemetry logs on the Elastic OTLP intake.

Would this solve your problem?

The architecture would be:

                                                                                    ┌────────────────────────────────────────┐
                                                                                    │                ELASTIC                 │
                                                                                    │                                        │
    ┌────────────────────┐ OTLP  ┌──────────────┐  ┌─────┐  ┌──────────────┐ OTLP   │ ┌─────────────────┐    ┌─────────────┐ │
    │ Otel Logs Producer ├──────►│Otel Collector├─►│Kafka├─►│Otel Collector├────────┤►│Elstic APM Server│    │Elasticsearch│ │
    └────────────────────┘       └──────────────┘  └─────┘  └──────────────┘        │ └─────────────────┘    └─────────────┘ │
                                                                                    │                                        │
                                                                                    └────────────────────────────────────────┘

More details on the Elastic native support for OpenTelemetry on https://www.elastic.co/guide/en/apm/guide/current/open-telemetry.html

[goto1134]

Hello, @MovieStoreGuy !

The format I am looking for is a json

{
"@timestamp" : ""
"@version": ""
"message": ""
}

with the ability to add custom fields at any level of this object. We use logback-logstash-encoder to use this json log format.

[goto1134]

@cyrille-leclerc , as for the possible integration, we are using a third-party proprietary logging system, that supports only the mentioned logstash format, and the only way to send the logs there is via Kafka.

[cyrille-leclerc]

Thanks for the explanation @goto1134 . With the adoption of OpenTelemetry as a first class intake format, I guess the Elasticsearch & Logstash communities may not contribute much to adding to the OpenTelemetry Collector the capability to export with the Logstash's JSON format.

Did you consider working with the provider of your logging system to add support for OTELP logs?

[goto1134]

Yes, of course, but the provider does not plan to implement this.

[MovieStoreGuy]

Hey @goto1134 ,

Would a custom worker that just translates from otlp to the elastic json format from one topic to another?
(There is a risk that it would at least double the number of bytes stored within the kafka cluster).

[MovieStoreGuy]

Alright, here is my thoughts on what has been said so far.

A vendor specific encoding goes against the spirit of open telemetry trying to develop an ecosystem of adopting non vendor specific standard; any currently open components should only adopt open standards that multiple vendors / platforms can consume from.

Having said this, extending the open to be optimised for the vendor specific platform is acceptable (re: aws implementation of open source products) done in making the the non vendor specific component, as a vendor specific component.
Meaning, that this should be created as elastickafkaexporter as a community funded project that is heavily optimised to work with your exact use case.

My rationale of my thoughts (and my personal thoughts, not a reflection of those in the open telemetry project) is that the existing encodings are well established as open standards making it easier as a community to help make improvements, where vendor specific encodings (in this scenario) is harder for that community (specifically me as one of the code owners) to support. I unfortunately don't use any elastic products (personally or for work) so my ability to help personally is extremely limited since I can't go through my usual approach of validating changes, and I would be relying on someone like you @goto1134 to validate the changes for me. (which is a really shitty development loop, and I for one wouldn't wish that on anyone :\ )

To recap my thoughts here, yes I agree that there is a need for vendor specific encodings within open source exporters (like kafka, kinesis) that basically just byte forwarders, however, they should be bundled in a vendor specific exporter of the open source byte forwarded (for a lack of a better term).

[jkowall]

Yes, of course, but the provider does not plan to implement this.

@goto1134 Can we call out the provider so others can apply pressure for them to evolve their thinking? I am happy to speak to them and explain why this important.

[jpkrohling]

, yes I agree that there is a need for vendor specific encodings within open source exporters

Just throwing ideas around: how about having something similar to the auth extensions, where encoders are extensions and this exporter would then hold a reference to the encoder to use? This way, we can ship with open standards, while people can build their own distributions embedding their vendor-specific lock-in encoder if they want.