Open
Description
openedon Apr 23, 2021
Build scan:
https://logstash-ci.elastic.co/job/elastic+logstash+pull-request+multijob-integration-1/3048/consoleFull
Repro line:
./gradlew integrationTests -PrubyIntegrationSpecs=specs/es_output_how_spec.rb
Reproduces locally?:
yes, just run the repro line command and check logs in build/logstash-8.0.0-SNAPSHOT/logs/logstash-plain.log
Applicable branches:
master, but all PR branches
Failure history:
Failure excerpt:
[2021-04-23T11:58:22,563][WARN ][logstash.outputs.elasticsearch][main][a8e3dc1ef9721fdf6415b66aaa0fbb9ab926b7010b3db5978d169cc43f2fc249] Could not index event to Elasticsearch. {:status=>400, :action=>["create", {:_id=>nil, :_index=>"logs-generic-default", :routing=>nil}, {"host"=>"kalispera", "auth"=>"-", "request"=>"/blog/geekery/disabling-battery-in-ubuntu-vms.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+semicomplete%2Fmain+%28semicomplete.com+-+Jordan+Sissel%29", "bytes"=>9316, "httpversion"=>"1.1", "geoip"=>{"country_code3"=>"US", "timezone"=>"America/Chicago", "country_code2"=>"US", "ip"=>"198.46.149.143", "latitude"=>37.751, "location"=>{"lat"=>37.751, "lon"=>-97.822}, "country_name"=>"United States", "longitude"=>-97.822, "continent_code"=>"NA"}, "@timestamp"=>2014-09-11T21:59:29.000Z, "@version"=>"1", "response"=>200, "referrer"=>"\"-\"", "ident"=>"-", "verb"=>"GET", "message"=>"198.46.149.143 - - [11/Sep/2014:21:59:29 +0000] \"GET /blog/geekery/disabling-battery-in-ubuntu-vms.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+semicomplete%2Fmain+%28semicomplete.com+-+Jordan+Sissel%29 HTTP/1.1\" 200 9316 \"-\" \"Tiny Tiny RSS/1.11 (http://tt-rss.org/)\"", "useragent"=>{"major"=>"1", "build"=>"", "name"=>"Tiny Tiny RSS", "minor"=>"11", "device"=>"Other", "os"=>"Other", "os_name"=>"Other"}, "clientip"=>"198.46.149.143", "agent"=>"\"Tiny Tiny RSS/1.11 (http://tt-rss.org/)\"", "timestamp"=>"11/Sep/2014:21:59:29 +0000", "data_stream"=>{"type"=>"logs", "dataset"=>"generic", "namespace"=>"default"}}], :response=>{"create"=>{"_index"=>".ds-logs-generic-default-2021.04.23-000001", "_id"=>"vUsq_ngBdswJt_MNgxba", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [host] tried to parse field [host] as object, but found a concrete value"}}}}
Description
The problem originates from the fact that Elasticsearch 8.0.0 has Datastream and ECS enabled. In the pipeline used by the test
logstash/qa/integration/fixtures/es_output_how_spec.yml
Lines 29 to 31 in 1e08341
index is specified so on Elasticsearch side a Datastream is created (.ds-logs-generic-default-2021.04.23-000001) and any indexing request got 404
object mapping for [host] tried to parse field [host] as object, but found a concrete value
The issue request to create an integration test like
that validate also the ingestion in an Elasticsearch 8.0.0Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment