Conform to standardized parameters w/o underscores in ES7

[jsvd]

In elastic/elasticsearch#27414 many parameters will no longer be accepted with a prefixed underscore:

+* _parent
+* _retry_on_conflict
+* _routing
+* _version
+* _version_type

This is causing the tests against elasticsearch master to fail for this plugin. The following is an example the body of a request produced with this plugin:

{"index":{"_id":null,"_index":"9515404195","_type":"2536945077","_routing":"test"}}
{"@timestamp":"2017-11-22T22:10:19.060Z","@version":"1","message":"test","type":"2536945077"}

which ES master will respond with an error due to the PR mentioned above:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_routing]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_routing]"},"status":400}

[jordansissel]

Action/metadata line [1] contains an unknown parameter [_routing]"}],"

I really wish this were a more informative error message from Elasticsearch -- it's really unclear to the user what they should do with this response. We should open an issue with ES to see if they can improve the error message (example: It's not "unknown" -- it's been renamed)

[mjtrangoni]

@jsvd As of ES 6.3.1 I am still watching this in deprecation.log,

[2018-07-16T08:42:44,527][WARN ][o.e.d.c.ParseField       ] Deprecated field [_retry_on_conflict] used, expected [retry_on_conflict] instead

# /usr/share/logstash/bin/logstash-plugin list --verbose  | grep logstash-output-elasticsearch
logstash-output-elasticsearch (9.2.0)

Is there any ETA for getting this fixed? Or even a workaround?

Thanks!

[jsvd]

fixed in #812

[simmel]

This still isn't fixed:

# /local/logstash/bin/logstash-plugin list --verbose | grep logstash-output-elasticsearch
logstash-output-elasticsearch (9.2.4)

# /local/elasticsearch/bin/elasticsearch --version
Version: 6.3.0, Build: default/tar/424e937/2018-06-11T23:38:03.357887Z, JVM: 1.8.0_152

2018-11-28T09:26:38.645845+01:00 syslog-test-search03.it.su.se elasticsearch[610]: {"timeMillis":1543393598645,"thread":"elasticsearch[syslog-test-search03.it.su.se][http_server_worker][T#5]","level":"WARN","loggerName":"org.elasticsearch.deprecation.common.ParseField","message":"Deprecated field [_retry_on_conflict]
used, expected [retry_on_conflict] instead","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","threadId":67,"threadPriority":5}

[simmel]

This fixes it, for us, but I'm not sure which version the deprecation was introduced.

--- a/lib/logstash/outputs/elasticsearch/common.rb  2018-11-28 09:48:42.254592311 +0100
+++ b/lib/logstash/outputs/elasticsearch/common.rb  2018-11-28 09:48:36.186416948 +0100
@@ -120,7 +120,7 @@
     end

     def retry_on_conflict_action_name
-      maximum_seen_major_version >= 7 ? :retry_on_conflict : :_retry_on_conflict
+      maximum_seen_major_version >= 6 ? :retry_on_conflict : :_retry_on_conflict
     end

     def install_template