Exporter for Elasticsearch

[cyrille-leclerc]

Is your feature request related to a problem? Please describe.

The problem is to export OpenTelemetry data, primarily logs, to Elasticsearch the same way many observability collectors can send data to Elasticsearch.

Describe the solution you'd like

The solution would be an OpenTelemetry Collector Exporter for Elasticsearch.

This exporter should try to support 6.8+ versions of Elasticsearch as well as the recent ones (7.x) to help Elasticsearch users interested in adopting OpenTelemetry collectors.

Elasticsearch is a very popular Open Source storage for logs in a similar way Prometheus is very popular for metrics and Jaeger or Zipkin are popular for traces. This is the rationale to propose to add the Elasticsearch exporter directly in the opentelemetry-collector repository, next to the Prometheus, Zipkin and Jaeger exporters.

Examples of observability collectors supporting exports to Elasticsearch:

• Fluentd: fluentd-elasticsearch-plugin
• Fluent Bit: Fluent Bit Elasticsearch Output
• Telegraf: Elasticsearch Output Plugin
• RSyslog: Elasticsearch Output Module

Describe alternatives you've considered

An alternative would be to have Elasticsearch being capable of ingesting the OTLP protocol. It's a scenario we are also looking at in parallel but the adoption path of OpenTelemetry collectors for existing Elasticsearch users would be steeper.

Additional context

Elasticsearch is published under the Apache Software License 2.

[rakyll]

cc @alolita

[cyrille-leclerc]

We will be happy to soon propose a first small PR according to the Contributing guidelines.

[bogdandrutu]

Consider this to be in the contrib repo, I will move the issue as well.

[tigrannajaryan]

There is already an Elastic exporter for traces and metrics: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/master/exporter/elasticexporter
Logs likely should be added to it, I don't think we want a different exporter for logs.

[cyrille-leclerc]

Thanks for the guidance @tigrannajaryan and @bogdandrutu . I'll close this issue and we will ship logs to Elasticsearch through the OpenTelemetry Collector exporter for Elastic.