Add more visibility to "Encountered a retryable error."

[KIVagant]

Hello.

I would like to ask you to add more details into the next log:

[2018-08-23T13:15:54,028][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"https://mydomain.us-east-1.es.amazonaws.com:443/_bulk"}
[2018-08-23T13:15:54,029][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"https://mydomain.us-east-1.es.amazonaws.com:443/_bulk"}
[2018-08-23T13:15:54,030][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"https://mydomain.us-east-1.es.amazonaws.com:443/_bulk"}

The problem is that I see tons of logs like this and can't understand: are they all related to the one request or they all are results from different requests? Is Logstash trying to re-send one request again and again or just some of requests fail periodically and then they finally get to ES?

It would be nice to have a request Id of something like this in the log.

[lucabelluccini]

I agree we should also provide the reason, which would help a lot.
In DEBUG the message is complete but maybe too verbose.

[jeanfabrice]

+1
In my opinion, ERROR tagged messages should be consistent wether your log level is DEBUG or INFO since ERROR is a lower severity level.

According to https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/common.rb#L307, this is not the case at the moment

[colinsurprenant]

Thanks for the heads up @KIVagant - I agree this is hard to follow.

Thanks for your suggestions too @lucabelluccini and @jeanfabrice.

I will suggest something shortly for this and will sync with @jsvd who also worked on cleaning up some of the logs.