Open
Description
Logstash produces some garbage for "Elasticsearch Unreachable" events:
...
"loggerName": "logstash.outputs.elasticsearch",
"levelString": "warn",
"thread": "Ruby-0-Thread-34@[main]>worker1: :1",
"level": "40",
"timeMillis": 1521381519666,
"logEvent": {
"message": "Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://logs-sandbox:xxxxxx@<instance>.europe-west1.gcp.cloud.es.io:9243/][Manticore::SocketException] Connection reset",
"url": {
"metaClass": {
"metaClass": {
"metaClass": {
"error_message": "Elasticsearch Unreachable: [https://logs-sandbox:xxxxxx@<instance>.europe-west1.gcp.cloud.es.io:9243/][Manticore::SocketException] Connection reset",
"error_class": "LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError",
"url": "https://logs-sandbox:xxxxxx@<instance>.europe-west1.gcp.cloud.es.io:9243/"
}
}
}
}
}
...
Those extra metaClass layers are really unnecessary and, when Logstash logs are getting synced into Elastic, the field type for logEvent.url is getting violated as it is not a URL/string anymore but becomes an object.
For example "restored connection" event is getting logged like that:
...
"levelString": "warn",
"loggerName": "logstash.outputs.elasticsearch",
"logEvent": {
"message": "Restored connection to ES instance",
"url": "https://logs-sandbox:xxxxxx@<instance>.europe-west1.gcp.cloud.es.io:9243/"
},
"timeMillis": 1521063811404,
"thread": "Ruby-0-Thread-29: :1",
"level": "40"
...
- Version: Logstash 6.2.2, chart config is taken from https://github.com/kubernetes/charts/tree/f9826507a13abc719ca7240f9eb40ad9be984b17/incubator/logstash
- Operating System: Dockerized deployment with Kubernetes (v1.9.3-gke.0) Container-Optimized OS from Google (kernel version 4.4.111+)
- Config File (if you have sensitive info, please remove it):
o log4j2.properties
o logstash.yml - Sample Data: see description
- Steps to Reproduce: Let the logging pipeline run for a while so that connection to Elasticsearch gets interrupted (and then auto-restored)