Description
Elasticsearch version: 1.5.0
Logstash version: 2.2.0
OS: Ubuntu_14.04.3_LTS_HVM (uname -a = Linux my-test-logstash-1-us-west-2 3.13.0-63-generic #103-Ubuntu SMP Fri Aug 14 21:42:59 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux)
Cloud platform: AWS
Details
For the following two configs, Logstash gives different errors respectively. The 3rd config works well, though, but is not practical in a cloud environment like AWS.
Config 1 (failed)
output {
elasticsearch {
hosts => ["http://es_myapp.us-west-2.test.mydomain.net:9200"]
index => "logstash-%{+YYYYMMdd}"
document_type => "%{[@metadata][type]}"
}
}
Error message:
Attempted to send a bulk request to Elasticsearch configured at '["http://http://es_myapp.us-west-2.test.mydomain.net:9200/"]', but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided? {:client_config=>{:hosts=>["http://http://es_myapp.us-west-2.test.mydomain.net:9200/"], :ssl=>nil, :transport_options=>{:socket_timeout=>0, :request_timeout=>0, :proxy=>nil, :ssl=>{}}, :transport_class=>Elasticsearch::Transport::Transport::HTTP::Manticore, ...
Note the extra "http://" in the :hosts=> field in the error message. I believe this should definitely be a bug.
Config 2 (failed)
output {
elasticsearch {
hosts => ["es_myapp.us-west-2.test.mydomain.net:9200"]
index => "logstash-%{+YYYYMMdd}"
document_type => "%{[@metadata][type]}"
}
}
Error message:
The error reported is:
the scheme http does not accept registry part: es_myapp.us-west-2.test.mydomain.net:9200 (or bad hostname?)
/apps/logstash-2.2.0/vendor/jruby/lib/ruby/1.9/uri/generic.rb:214:in `initialize'
/apps/logstash-2.2.0/vendor/jruby/lib/ruby/1.9/uri/http.rb:84:in `initialize'
/apps/logstash-2.2.0/vendor/jruby/lib/ruby/1.9/uri/common.rb:214:in `parse'
/apps/logstash-2.2.0/vendor/jruby/lib/ruby/1.9/uri/common.rb:747:in `parse'
/apps/logstash-2.2.0/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/client.rb:155:in `__extract_hosts'
org/jruby/RubyArray.java:2414:in `map'
/apps/logstash-2.2.0/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/client.rb:151:in `__extract_hosts'
/apps/logstash-2.2.0/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/client.rb:115:in `initialize'
/apps/logstash-2.2.0/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport.rb:26:in `new'
/apps/logstash-2.2.0/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:136:in `build_client'
Config 3 (passed)
output {
elasticsearch {
hosts => ["100.10.10.10:9200"] # 100.10.10.10 is the IP address of es_myapp.us-west-2.test.mydomain.net
index => "logstash-%{+YYYYMMdd}"
document_type => "%{[@metadata][type]}"
}
}
This works, however, I'm in AWS environment and cannot rely on IP addresses.
The minimum requirement is that for Config 1 the parsing should work.
Please take a look at. Thanks in advance!
=================
Updated:
With Logstash 2.2.2 and logstash-output-elasticsearch 2.1, the issue still exists (see the 5th reply).
Tried to update the plugin to 2.5.5, however, the upgrade may have corrupted Logstash, and it now fails to start with error:
The error reported is:
you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command
no such file to load -- org/apache/httpcomponents/httpcore/4.4.1/httpcore-4.4.1 (LoadError)
================
Updated 11/1/16: put the above update under the original post to avoid confusion.