Create OpenSearch output as fork diverges from ElasticSearch

[sandervandegeijn]

Relevant telegraf.conf

[[outputs.elasticsearch]]
  ## The full HTTP endpoint URL for your Elasticsearch instance
  ## Multiple urls can be specified as part of the same cluster,
  ## this means that only ONE of the urls will be written to each interval.
  urls = [ "${DIP_MONITORING_URL}" ] # required.
  ## Elasticsearch client timeout, defaults to "5s" if not set.
  timeout = "5s"
  ## Set to true to ask Elasticsearch a list of all cluster nodes,
  ## thus it is not necessary to list all nodes in the urls config option
  enable_sniffer = false
  ## Set the interval to check if the Elasticsearch nodes are available
  ## Setting to "0s" will disable the health check (not recommended in production)
  health_check_interval = "15s"

Logs from Telegraf

2022-06-21T19:04:50Z I! Starting Telegraf 1.20.4                                                                                                                                                                         
││ 2022-06-21T19:04:50Z I! Using config file: /etc/telegraf/telegraf.conf                                                                                                                                                   
││ 2022-06-21T19:04:50Z I! Loaded inputs: kube_inventory prometheus                                                                                                                                                         
││ 2022-06-21T19:04:50Z I! Loaded aggregators:                                                                                                                                                                              
││ 2022-06-21T19:04:50Z I! Loaded processors:                                                                                                                                                                               
││ 2022-06-21T19:04:50Z I! Loaded outputs: elasticsearch                                                                                                                                                                    
││ 2022-06-21T19:04:50Z I! Tags enabled: environment=dip-prd host=telegraf-namespace-7d69446766-fdm5k                                                                                                                       
││ 2022-06-21T19:04:50Z I! [agent] Config: Interval:5m0s, Quiet:false, Hostname:"telegraf-namespace-7d69446766-fdm5k", Flush Interval:5s                                                                                    
││ 2022-06-21T19:04:50Z D! [agent] Initializing plugins                                                                                                                                                                     
││ 2022-06-21T19:04:50Z D! [agent] Connecting outputs                                                                                                                                                                       
││ 2022-06-21T19:04:50Z D! [agent] Attempting connection to [outputs.elasticsearch]                                                                                                                                         
││ 2022-06-21T19:04:50Z E! [agent] Failed to connect to [outputs.elasticsearch], retrying in 15s, error was 'Elasticsearch version not supported: 2.0.0'                                                                    
││ 2022-06-21T19:05:05Z E! [telegraf] Error running agent: connecting output outputs.elasticsearch: Error connecting to output "outputs.elasticsearch": Elasticsearch version not supported: 2.0.0

System info

1.20.4

Docker

No response

Steps to reproduce

1. Use opensearch 2.0
2. Read telegraf docs
3. Find that the compatibility setting is removed as announced
4. Without this setting Telegraf crashes

Expected behavior

Support opensearch output

Actual behavior

Crash

Additional info

No response

[powersj]

Thanks for filing a new issue.

Some questions as my searching on Amazon docs seems to be failing me:

Find that the compatibility setting is removed as announced

1. Where was this announced? Do you have a link?
2. What version of Elasticsearch is 2.0 based on?
3. Is there any documented way to identify that we are connected to an OpenSearch 2.0 instance?

Thanks!

[sandervandegeijn]

1. It was announced for Opensearch 2.0, can't find the link, but proof on a 2.0.1 cluster:

2. They are developing independently, OS was based on the last OSS version of Elastic and are now maintaining their own code because of the licensing change of Elastic.

3. Yes:

[powersj]

Hi,

Thanks for the answers.

I took a look at their repo on GitHub and found opensearch-project/OpenSearch#3023 which is where the removal of the setting was discussed. It does sound like the eventual plan is to have users move to a new client and not expect backwards compatibility. However, the last comment leads us to opensearch-project/OpenSearch#3530 which is a revert and adds the compatibility mode back in for now.

We already want/need to move to an elasticsearch-specific client due to the current library not getting further development, see olivere/elastic#1240 (comment) and olivere/elastic#1481. Ultimately, OpenSearch may need its own output plugin to itself as the fork diverges and/or the use of an OpenSearch specific client.

Version 2.0.1 had this revert in it, however from the above I see you are also running 2.0.1. Are you running a beta version or a development version?

[sandervandegeijn]

I'm running the 2.0.1 stable Docker container :) But ES and OS will become more or less independent although similar. A separate output plugin would be the best future proof option I guess.

[sandervandegeijn]

2.1.0 restores the compatibility setting for now, but this is not a long term viable option. There is still a need for the opensearch specific output plugin

[kclinden]

Any update on this?

[mannukalra]

Would really appreciate if someone can update about any future plans for this.

[matanbaruch]

Any update on this?

[yakir-shriker]

+1

[wooonderful]

Any update on this? I meet the same question.

[wooonderful]

hello

Whether version checking can be turned off， I think the Opensearch api is compatible with elasticsearch API.

just turn off the version check, the elasticsearch output plugin will be compatible with opensearch.

[wooonderful]

hello

Whether version checking can be turned off， I think the Opensearch api is compatible with elasticsearch API.

just turn off the version check, the elasticsearch output plugin will be compatible with opensearch.

To solve this problem, enable the compatibility mode of Opensearch, using the following command

curl -X PUT -H "Content-Type: application/json" -d '{"persistent" : {"compatibility.override_main_response_version" : true}}' http://opensearch-cluster-master.opensearch.svc:9200/_cluster/settings -u 'admin:admin'

[sandervandegeijn]

THis is a temporary workaround (that was initially already disabledi n the 2.x branch). Disabling the version check is valid, for now. Compatibility is not guaranteed for the future, architecturally I would recommend two plugins. For now, the open search variant could be a copy of the elastic one with different version numbers in the check. This would future proof it.

[powersj]

Instead of asking for updates, please look at the URL at the top of this issue that links to an open PR "May be fixed by #11958" A community member is actively working on this and we are working with him to resolve the lack of an official opensearch PR.

[sandervandegeijn]

Cool.thanks!