Fluentd can't work with Signature Version 4

[AndriiPopok]

Problem

Hi all,

I have a problem with the plugin

Fluentd can't work with Signature Version 4

2019-12-24 08:46:50 +0000 [error]: #0 unexpected error error_class=Elasticsearch::Transport::Transport::Errors::Forbidden error="[403] {\"message\":\"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\"}" 2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/elasticsearch-transport-6.8.0/lib/elasticsearch/transport/transport/base.rb:205:in __raise_transport_error'
2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/elasticsearch-transport-6.8.0/lib/elasticsearch/transport/transport/base.rb:323:in perform_request' 2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/elasticsearch-transport-6.8.0/lib/elasticsearch/transport/transport/http/faraday.rb:20:in perform_request'
2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/elasticsearch-transport-6.8.0/lib/elasticsearch/transport/client.rb:143:in perform_request' 2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/elasticsearch-api-6.8.0/lib/elasticsearch/api/actions/info.rb:15:in info'
2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-elasticsearch-3.5.5/lib/fluent/plugin/out_elasticsearch.rb:335:in detect_es_major_version' 2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-elasticsearch-3.5.5/lib/fluent/plugin/out_elasticsearch.rb:246:in block in configure'
2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-elasticsearch-3.5.5/lib/fluent/plugin/elasticsearch_index_template.rb:35:in retry_operate' 2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-elasticsearch-3.5.5/lib/fluent/plugin/out_elasticsearch.rb:245:in configure'
2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/plugin.rb:164:in configure' 2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/agent.rb:130:in add_match'
2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/agent.rb:72:in block in configure' 2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/agent.rb:64:in each'
2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/agent.rb:64:in configure' 2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/root_agent.rb:150:in configure'
2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/engine.rb:131:in configure' 2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/engine.rb:96:in run_configure'
2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/supervisor.rb:812:in run_configure' 2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/supervisor.rb:558:in block in run_worker'
2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/supervisor.rb:741:in main_process' 2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/supervisor.rb:554:in run_worker'
2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/command/fluentd.rb:330:in <top (required)>' 2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in require'
2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in require' 2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/bin/fluentd:8:in <top (required)>'
2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/bin/fluentd:23:in load' 2019-12-24 08:46:50 +0000 [error]: #0 /opt/td-agent/embedded/bin/fluentd:23:in

'
2019-12-24 08:46:50 +0000 [info]: Worker 0 finished unexpectedly with status 1
2019-12-24 08:46:50 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '2.3.0'
2019-12-24 08:46:50 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '3.5.5'
2019-12-24 08:46:50 +0000 [info]: gem 'fluent-plugin-kafka' version '0.12.1'
2019-12-24 08:46:50 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.7.0'
2019-12-24 08:46:50 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.0.1'
2019-12-24 08:46:50 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.2.0'
2019-12-24 08:46:50 +0000 [info]: gem 'fluent-plugin-s3' version '1.2.1'
2019-12-24 08:46:50 +0000 [info]: gem 'fluent-plugin-s3' version '1.2.0'
2019-12-24 08:46:50 +0000 [info]: gem 'fluent-plugin-td' version '1.0.0'
2019-12-24 08:46:50 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.4'
2019-12-24 08:46:50 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.2.4'
2019-12-24 08:46:50 +0000 [info]: gem 'fluentd' version '1.7.4'
2019-12-24 08:46:50 +0000 [info]: adding filter pattern="docker." type="parser"
2019-12-24 08:46:50 +0000 [info]: adding match pattern="docker." type="aws-elasticsearch-service"
2019-12-24 08:46:51 +0000 [info]: #0 'flush_interval' is configured at out side of . 'flush_mode' is set to 'interval' to keep existing behaviour
2019-12-24 08:46:51 +0000 [error]: #0 unexpected error error_class=Elasticsearch::Transport::Transport::Errors::Forbidden error="[403] {"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}"
`
td-agent 1.7.4

[braybaut]

some update about this?

[cosmo0920]

I couldn't get reproducing this issue with:

• Elasticsearch version 7.1

% bundle                                                                                                             (git)[master][Fail]
Fetching gem metadata from https://rubygems.org/.........
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...
Using rake 10.5.0
Using aws-eventstream 1.0.1
Using aws-partitions 1.141.0
Using aws-sigv4 1.0.3
Using jmespath 1.4.0
Using aws-sdk-core 3.46.2
Using bundler 1.17.2
Using cool.io 1.5.3
Using diff-lcs 1.3
Using dig_rb 1.0.1
Using multi_json 1.13.1
Using elasticsearch-api 6.1.0
Using multipart-post 2.0.0
Using faraday 0.15.4
Using elasticsearch-transport 6.1.0
Using elasticsearch 6.1.0
Using excon 0.62.0
Fetching faraday_middleware-aws-sigv4 0.3.0 (was 0.2.4)
Installing faraday_middleware-aws-sigv4 0.3.0 (was 0.2.4)
Using http_parser.rb 0.6.0
Using msgpack 1.2.7
Using sigdump 0.2.4
Using serverengine 2.1.0
Using strptime 0.2.3
Using thread_safe 0.3.6
Using tzinfo 1.2.5
Using tzinfo-data 1.2018.9
Using yajl-ruby 1.4.1
Using fluentd 1.4.0
Using fluent-plugin-elasticsearch 3.3.0
Using fluent-plugin-aws-elasticsearch-service 2.3.0 (was 2.0.1) from source at `.`
Using power_assert 1.1.3
Using rspec-support 3.8.0
Using rspec-core 3.8.0
Using rspec-expectations 3.8.2
Using rspec-mocks 3.8.0
Using rspec 3.8.0
Using test-unit 3.3.0
Bundle complete! 8 Gemfile dependencies, 37 gems now installed.
Bundled gems are installed into `./vendor/bundle`

  <source>
    @type forward
    port 24225
  </source>
  <filter **>
    @type stdout
  </filter>
  <match test.**>
    @type copy
    <store>
      @type "stdout"
    </store>
    <store>
      @type "aws-elasticsearch-service"
      @log_level "info"
      with_transporter_log true
      type_name "_doc"
      logstash_format true
      include_tag_key true
      <buffer>
        flush_interval 1s
      </buffer>
      <endpoint>
        url "https://search-your-domain-cvhaefuivhsduinhrciudgmvr.your-aws-region-1.es.amazonaws.com"
        region "us-east-1"
        assume_role_arn "arn:aws:es:your-aws-region-1:123456789000:domain/yourdomain"
        assume_role_session_name "fluentd"
        access_key_id "secretstring"
        secret_access_key xxxxxx
      </endpoint>
    </store>
  </match>

can bulk insert:

2020-01-28 16:12:51 +0900 [info]: #0 POST https://search-your-domain-cvhaefuivhsduinhrciudgmvr.your-aws-region-1.es.amazonaws.com:443/_bulk [status:200, request:0.716s, query:0.007s]

Access policy is:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789000:user/fluentd-testing"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:your-aws-region-1:123456789000:domain/yourdomain/*"
    }
  ]
}

[namishelex01]

I got the same error for different signature versions. It was a silly mistake where I put extra '/' at the end of URL 'https://search-your-domain-cvhaefuivhsduinhrciudgmvr.your-aws-region-1.es.amazonaws.com'.

[cosmo0920]

Hmm..., we should raise error or make warning log when specifying trailing slash....

[AndreLouisCaron]

I was also bit by this bug sometime ago. There is a comment about this in the "troubleshooting" section in the README. It would honestly be very cool if the plugin could detect and account for it automatically (e.g. by removing the trailing slash, if present). Semantically, http://es.example.org and http://es.example.org/ are the same, so users don't expect this to cause an error.

[cosmo0920]

Oh, thanks for the detailed information.
I'll consider to display warn or error for tailing slash("/").

[cosmo0920]

I'd registered a fix to handle this issue: #67

[AndreLouisCaron]

Thanks a lot for the fix!