configuration

[fragtom]

In the Readme the config-file is written in ruby.. I'm stuck with configuration of my fluentd env -

<source>
...
</source>

# Include config files in the ./config.d directory
@include /etc/fluentd/conf.d/*.conf

and

root@fluentd:/etc/fluentd/conf.d# vi aws-elasticsearch.conf 

<match "**">
      type "aws-elasticsearch-service"
      logstash_format true
      include_tag_key true
      tag_key "@log_name"
      log_level info

      flush_interval 5s
      buffer_chunk_limit 512K
...

      <endpoint>
        url "https://.....eu-central-1.es.amazonaws.com/"
        region "eu-central-1"
        access_key_id "ACCESS_KEY"
        secret_access_key "SECRET_KEY"
      </endpoint>
    </match>

that should be ok.. but starting agent ...

2015-10-23 09:02:26 +0000 [info]: adding match pattern="\"**\"" type="aws-elasticsearch-service"
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-aws-elasticsearch-service-0.1.1/lib/fluent/plugin/out_aws-elasticsearch-service.rb:76: warning: toplevel constant Elasticsearch referenced by Fluent::ElasticsearchOutput::Elasticsearch

[fragtom]

after a lot of debugging following error comes up by calling "/usr/sbin/td-agent" manually..

I'm sure, that I have correct IAM credentials configured in endpoint.

2015-10-23 15:38:51 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-10-23 15:38:52 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:1be2f34"
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/base.rb:135:in `__raise_transport_error'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/base.rb:227:in `perform_request'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/client.rb:119:in `perform_request'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.12/lib/elasticsearch/api/actions/ping.rb:20:in `block in ping'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.12/lib/elasticsearch/api/utils.rb:189:in `__rescue_from_not_found'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.12/lib/elasticsearch/api/actions/ping.rb:19:in `ping'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:69:in `client'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:176:in `rescue in send'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:174:in `send'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:168:in `write'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/buffer.rb:325:in `write_chunk'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/buffer.rb:304:in `pop'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/output.rb:321:in `try_flush'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/output.rb:140:in `run'

[fragtom]

I have debugged more .. and in connection string seems the the secret is missing? or could it be misunderstanding the authorization way..

{:host=>"search-......eu-central-1.es.amazonaws.com", 
:port=>80, :scheme=>"http", :path=>"/", :aws_elasticsearch_service=> {:credentials=>#<Aws::Credentials access_key_id="AKIAI....KQ">, :region=>"eu-central-1"}}

thank you for any help

[atomita]

hi @fragtom .

it was so as not to "warning: toplevel constant Elasticsearch referenced".

"[warn]: temporarily failed to flush the buffer." is I do not know yet well ...
But, response message from the AWS is I am sure the "The security token included in the request is expired".
So, I removed the memorize of credentials.

Please try using the "v0.1.2".

thanks!

[fragtom]

thank you for taking up the issue.. the warning isn't present now..

2015-10-24 20:22:19 +0000 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2015-10-24 20:22:19 +0000 [info]: starting fluentd-0.12.12
2015-10-24 20:22:19 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.0'
2015-10-24 20:22:19 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2015-10-24 20:22:19 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '0.1.2'
2015-10-24 20:22:19 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '0.1.1'
2015-10-24 20:22:19 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.1.0'
2015-10-24 20:22:19 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '0.9.0'
2015-10-24 20:22:19 +0000 [info]: gem 'fluent-plugin-json-in-json' version '0.1.4'
..
2015-10-24 20:22:19 +0000 [info]: adding match pattern="reform.**" type="record_reformer"
2015-10-24 20:22:19 +0000 [info]: adding match pattern="kubernetes.**" type="aws-elasticsearch-service"
2015-10-24 20:22:19 +0000 [info]: adding source type="tail"

but the 403 .. seems to wrong credentials...

2015-10-24 20:22:45 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-10-24 20:22:46 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:179e20c"
  2015-10-24 20:22:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/base.rb:135:in `__raise_transport_error'
  2015-10-24 20:22:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/base.rb:227:in `perform_request'
  2015-10-24 20:22:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request

what about the result of debugging statement ... could it be possible, that elasticsearch "connection" is not fully filled.. or is that an misunderstanding of mine? I cannot see "secret_access_key" ..

:aws_elasticsearch_service=> {:credentials=>#<Aws::Credentials access_key_id="AKIAI....KQ">, :region=>"eu-central-1"}}

[atomita]

There may be a problem with the "faraday_middleware-aws-signers-v4".
It is being check by applying the current patch.

I'm sorry, it is likely to take still time to fix...

[fragtom]

I'm so sorry, but on that issue I couldn't send signed request successfully.. Sadly I'm not the ruby expert to add debug statements on the right position to follow up the request .. it seems all ok with your plugin from my perspective .. If you have any things I could do for you, don't hesitate.

 {:host=>"search-....eu-central-1.es.amazonaws.com", :port=>443, :scheme=>"https", :path=>"/", :aws_elasticsearch_service=>{:credentials=>#<Proc:0x0000000075aaf8@/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-aws-elasticsearch-service-0.1.2/lib/fluent/plugin/out_aws-elasticsearch-service.rb:58 (lambda)>, :region=>"eu-central-1"}, :protocol=>"https"}

[atomita]

Please try using the "v0.1.3".
"[warn]: temporarily failed to flush the buffer." should have been fixed.

thanks!

[fragtom]

I've tested without success.. I'll post you debug output..

JFYI: thats our docker image for that.
https://hub.docker.com/r/organizeme/docker-fluentd-elasticsearch-aws/

2015-10-28 10:00:55 +0000 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2015-10-28 10:00:55 +0000 [info]: starting fluentd-0.12.12
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.0'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '0.1.3'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.1.0'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '0.9.0'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-json-in-json' version '0.1.4'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-mongo' version '0.7.10'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-record-reformer' version '0.7.0'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.4.1'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-s3' version '0.5.9'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-td' version '0.10.27'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.1'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1'
2015-10-28 10:00:55 +0000 [info]: gem 'fluentd' version '0.12.12'
2015-10-28 10:00:55 +0000 [info]: adding match pattern="reform." type="record_reformer"
2015-10-28 10:00:55 +0000 [info]: adding match pattern="kubernetes." type="aws-elasticsearch-service"
2015-10-28 10:00:55 +0000 [info]: adding source type="tail"
2015-10-28 10:00:55 +0000 [info]: adding source type="tail"
2015-10-28 10:00:55 +0000 [info]: adding source type="tail"
2015-10-28 10:00:55 +0000 [info]: adding source type="tail"
2015-10-28 10:00:55 +0000 [info]: adding source type="tail"
2015-10-28 10:00:55 +0000 [info]: adding source type="tail"
2015-10-28 10:00:55 +0000 [info]: adding source type="tail"
2015-10-28 10:00:55 +0000 [info]: adding source type="tail"
2015-10-28 10:00:55 +0000 [info]: adding source type="tail"
2015-10-28 10:00:55 +0000 [info]: adding source type="tail"
2015-10-28 10:00:55 +0000 [info]: using configuration file:

type tail format json_in_json time_key time path /varlog/containers/_.log pos_file /varlog/es-containers.log.pos time_format %Y-%m-%dT%H:%M:%S tag reform._ read_from_head true type record_reformer enable_ruby true tag kubernetes.${tag_suffix[3].split('-')[0..-2].join('-')} type tail format none path /varlog/monit.log pos_file /varlog/gcp-monit.log.pos tag kubernetes.monit type tail format none path /varlog/salt/minion pos_file /varlog/gcp-salt.pos tag kubernetes.salt type tail format none path /varlog/startupscript.log pos_file /varlog/es-startupscript.log.pos tag kubernetes.startupscript type tail format none path /varlog/docker.log pos_file /varlog/es-docker.log.pos tag kubernetes.docker type tail format none path /varlog/etcd.log pos_file /varlog/es-etcd.log.pos tag kubernetes.etcd type tail format none path /varlog/kubelet.log pos_file /varlog/es-kubelet.log.pos tag kubernetes.kubelet type tail format none path /varlog/kube-apiserver.log pos_file /varlog/es-kube-apiserver.log.pos tag kubernetes.kube-apiserver type tail format none path /varlog/kube-controller-manager.log pos_file /varlog/es-kube-controller-manager.log.pos tag kubernetes.kube-controller-manager type tail format none path /varlog/kube-scheduler.log pos_file /varlog/es-kube-scheduler.log.pos tag kubernetes.kube-scheduler type aws-elasticsearch-service logstash_format true include_tag_key true tag_key @log_name log_level info flush_interval 5s buffer_chunk_limit 512K buffer_queue_limit 128 max_retry_wait 300 url https://search------.eu-central-1.es.amazonaws.com/ region eu-central-1 access_key_id --- secret_access_key ---- 2015-10-28 10:00:55 +0000 [info]: following tail of /varlog/containers/fluentd-elasticsearch-172.31.5.40_kube-system_fluentd-elasticsearch-2f04135698fff595652f649ab2f840adfbabed408acefc190d762a062f228520.log 2015-10-28 10:00:55 +0000 [info]: following tail of /varlog/containers/fluentd-elasticsearch-172.31.5.40_kube-system_POD-73b4a8627ad690072517d4a5201b3cb6374b09847487a14598b2c1c4eb926849.log 2015-10-28 10:01:29 +0000 [info]: following tail of /varlog/monit.log

....

2015-10-28 10:02:32 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-10-28 10:02:33 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:1451748"
2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/base.rb:135:in __raise_transport_error' 2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/base.rb:227:inperform_request'
2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/http/faraday.rb:20:in perform_request' 2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/client.rb:119:inperform_request'
2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.12/lib/elasticsearch/api/actions/ping.rb:20:in block in ping' 2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.12/lib/elasticsearch/api/utils.rb:189:in__rescue_from_not_found'
2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.12/lib/elasticsearch/api/actions/ping.rb:19:in ping' 2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:69:inclient'
2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:176:in rescue in send' 2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:174:insend'
2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:168:in write' 2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/buffer.rb:325:inwrite_chunk'
2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/buffer.rb:304:in pop' 2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/output.rb:321:intry_flush'
2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/output.rb:140:in `run'

[atomita]

Error has immediately after starting the fluentd.
In this case, there is a possibility that has not been assigned a policy to IAM.
Please check the policies.

policy example:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1444210466000",
            "Effect": "Allow",
            "Action": [
                "es:ESHttpPost",
                "es:ESHttpGet",
                "es:ESHttpHead"
            ],
            "Resource": [
                "arn:aws:es:*:*:*"
            ]
        }
    ]
}

[fragtom]

I've written policy on es-service, there I thought, that is no explicit policy in IAM is needed...

but for testing, I've allowed my user in aim .. but no data comes in logstash..

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::MYACCOUNT:user/MY_ES_USER"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:eu-central-1:MYACCOUNT:searchdomain/*"
    },

[atomita]

How policy on user ?

[fragtom]

Ok, I would try ..

the user on IAM have the following policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::MYACCOUNT:user/MY_ES_USER"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:eu-central-1:MYACCOUNT:searchdomain/*"
    },

policy on elasticsearch service

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::MYACCOUNT:user/MY_ES_USER"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:eu-central-1:MYACCOUNT:domain/xtractorio/*"
    },

on a other environment the logstash-shipper collects data and stream it to es service via logstash-plugin, and there I have no IAM policy .. just the elasticsearch-policy..

just one difference between the environment is the region - eu-west-1 and eu-west-2..

could you give me a hint for debugging?

[fragtom]

I've listened via tcpdump on port 80 (just for debugging)

HEAD // HTTP/1.1
User-Agent: Faraday v0.9.1
X-Amz-Date: 20151029T091329Z
Host: search--sdfsfd7tsz3rqmhtvi.eu-central-1.es.amazonaws.com
X-Amz-Content-Sha256: e3b0...8fc1c149afbf4c899....4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256 Credential=AKdfskfjskldOA/20151029/eu-central-1/es/aws4_request, SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date, Signature=00ed1eda8ce90e6c27d4b3ee795f0cf8e90f0f17608554a71be58e5394c9f1e1
Content-Length: 0

HTTP/1.1 403 Forbidden
Content-Length: 788
Content-Type: application/json
Server: Jetty(8.1.12.v20130726)
x-amzn-RequestId: 51d800e3-7e1d-11e5-9215-29572a93ba20
Connection: keep-alive