Description
I think doc_as_upsert does not work properly in 1.0.7 (logstash 1.5.4, elastic 1.4.2 or 1.7.1)
I'm testing with this logstash.conf
input {
stdin {
add_field => {
"type" => "type_1"
"[@metadata][_id]" => "id_1"
"[@metadata][action]" => "update"
}
}
}
filter {
}
output {
elasticsearch {
action => "%{[@metadata][action]}"
document_id => "%{[@metadata][_id]}"
doc_as_upsert => "true"
host => "localhost"
protocol => "http"
index => "test_index"
}
}
elastic is running, logstash is running by logstash -f logstash.conf
Let's remove the index first
curl -XDELETE http://localhost:9200/test_index
Then pass any text to logstash console window:
Logstash startup completed
Hello
failed action with response of 404, dropping action: ["update", {:_id=>"id_1", :_index=>"test_index", :_type=>"type_1", :_routing=>nil}, #<LogStash::Event:0x3a0d6f83 @metadata_accessors=#<LogStash::Util::Accessors:0x66f2a7ef @store={"_id"=>"id_1", "action"=>"update", "retry_count"=>0}, @lut={"[_id]"=>[{"_id"=>"id_1", "action"=>"update", "retry_count"=>0}, "_id"], "[action]"=>[{"_id"=>"id_1", "action"=>"update", "retry_count"=>0}, "action"]}>, @cancelled=false, @data={"message"=>"Hello", "@version"=>"1", "@timestamp"=>"2015-08-31T12:06:08.273Z", "type"=>"type_1", "host"=>"minix.local"}, @metadata={"_id"=>"id_1", "action"=>"update", "retry_count"=>0}, @accessors=#<LogStash::Util::Accessors:0x6718b5a2 @store={"message"=>"Hello", "@version"=>"1", "@timestamp"=>"2015-08-31T12:06:08.273Z", "type"=>"type_1", "host"=>"minix.local"}, @lut={"type"=>[{"message"=>"Hello", "@version"=>"1", "@timestamp"=>"2015-08-31T12:06:08.273Z", "type"=>"type_1", "host"=>"minix.local"}, "type"], "host"=>[{"message"=>"Hello", "@version"=>"1", "@timestamp"=>"2015-08-31T12:06:08.273Z", "type"=>"type_1", "host"=>"minix.local"}, "host"]}>>] {:level=>:warn}
I think it is because 'doc_as_upsert' is not sent together with the doc. See part of the log if client logging is enabled:
POST http://localhost:9200//_bulk [status:200, request:0.059s, query:0.039s]
> {"update":{"_id":"id_1","_index":"test_index","_type":"type_1","_routing":null}}
> {"doc":{"message":"Hello","@version":"1","@timestamp":"2015-08-31T12:12:40.950Z","type":"type_1","host":"minix.local"}}
See https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch.rb#L406
The doc_as_upsert is taken into account only if action is 'update' but my action is set by [@metadata][action].
If I remove the 'if' on line 406, I get this request to elastic:
POST http://localhost:9200//_bulk [status:200, request:0.083s, query:0.060s]
> {"update":{"_id":"id_1","_index":"test_index","_type":"type_1","_routing":null}}
> {"doc":{"message":"Hello","@version":"1","@timestamp":"2015-08-31T12:13:23.367Z","type":"type_1","host":"minix.local"},"doc_as_upsert":true}
It is not proper solution, of course, but at least a hint what's wrong.
Thanks