[exporter/elasticsearch] enhancement request to support specifying pipeline in event

[leehinman]

Component(s)

exporter/elasticsearch

Is your feature request related to a problem? Please describe.

Currently the elasticsearch exporter only allows you to set the pipeline in the configuration file for the exporter, so that all events go to the same pipeline. Beats allows for each event to specify which pipeline is run per event.

https://github.com/elastic/beats/blob/27adc68350051c939e403219c18a0bb3d6e585f8/libbeat/outputs/elasticsearch/client.go#L385-L390

Describe the solution you'd like

Much like #36882 we would like the ability to specify the pipeline in the event (probably an attribute) and have the exporter read the pipeline from there and use it to set the Elasticsearch Bulk API pipeline query parameter. This will also require changes to https://github.com/elastic/go-docappender to support passing in the pipeline per document.

Describe alternatives you've considered

No response

Additional context

This will be required to support use cases that rely on different pipelines within the same index.

[github-actions]

Pinging code owners:

• exporter/elasticsearch: @JaredTan95 @carsonip @lahsivjar

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[carsonip]

/label -needs-triage