Exporter/opensearch: logstash_format or something similar

[simoncrowe]

Component(s)

exporter/opensearch

Is your feature request related to a problem? Please describe.

I order to make use of OpenSearch index state management policies, it is useful to be able add a YYYY.MM.DD suffix to indexes.

Describe the solution you'd like

I would like the same logstash_format configuration as elasticsearchexporter.

Alternatively, if feature parity for logstash_format doesn't make sense, I'd like some other way of adding a date suffix to indexes.

Describe alternatives you've considered

My current workaround is to set the namespace of the opensearch log exporter to the current date whenever the OTel collector helm chart is installed/updated. Installs normally happen a few times daily so it's good enough to break up the index for state (lifecycle) management.

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/opensearch: @Aneurysm9 @MitchellGale @MaxKsyunz @YANG-DB

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

I don't have a lot of experience with OpenSearch, but I see it's a somewhat common use case to include the date in the name of indices for ISM. I'll remove needs triage.

I'll defer to code owners for implementation details, but you're welcome to submit a PR in the meantime if you'd like!

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/opensearch: @Aneurysm9 @MitchellGale @MaxKsyunz @YANG-DB

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[MaxKsyunz]

@simoncrowe have you considered using OpenSearch data streams? Logs is the main use case for that feature.

[PDCuong]

Hello @simoncrowe, how to config opensearch exporter to create new data streams instead of new index.
When i config

exporters:
   opensearch/log:
      logs_index: user-abcd

It will create new index name user-abcd. Can you help me?

[malayh]

have you considered using OpenSearch data streams? Logs is the main use case for that feature.

@MaxKsyunz Can you please give an example to how to use it?

Let's say I have created a data stream called logs, I want it to pick a backing index for every day.

• How do I configure this in OpenSearch?
• How do I configure the opensearch exporter to write to the data stream?

[malayh]

For anyone in the future who gets here, this is how I have configured a log data stream that rolls over every night.

• Create a data stream (eg. logs-stream)
• Configure Otel exporter as follows

      opensearch/logs:
        logs_index: "logs-stream"
        http:
          endpoint: "https://your-opensearch-endpoint:9200"
          auth:
            authenticator: basicauth/os
          tls:
            insecure: false
            ca_file: /tls/ca.crt

• Create a cronjob that runs every night to calls curl -XPOST https://your-opensearch-url:9200/logs-stream/_rollover -u username:password
• Optional: You can also write a simple script to delete indexes after X days of retention.

[owenchenxy]

For anyone in the future who gets here, this is how I have configured a log data stream that rolls over every night.

• Create a data stream (eg. logs-stream)
• Configure Otel exporter as follows

      opensearch/logs:
        logs_index: "logs-stream"
        http:
          endpoint: "https://your-opensearch-endpoint:9200"
          auth:
            authenticator: basicauth/os
          tls:
            insecure: false
            ca_file: /tls/ca.crt

• Create a cronjob that runs every night to calls curl -XPOST https://your-opensearch-url:9200/logs-stream/_rollover -u username:password
• Optional: You can also write a simple script to delete indexes after X days of retention.

I checked the document here: https://opensearch.org/docs/latest/im-plugin/ism/policies/#rollover

The index pattern should only be like logs-000001, logs-000002, how can it be a data formatted suffix?

[malayh]

I checked the document here: https://opensearch.org/docs/latest/im-plugin/ism/policies/#rollover
The index pattern should only be like logs-000001, logs-000002, how can it be a data formatted suffix?

@owenchenxy Why does this matter? I have written these helpers to deal with rollover and deleting old indices. You'd never have to look into the underlying indices of the data stream.