We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fluentd started to crush for unknown reason.
2018-04-03 20:03:27 +0000 [warn]: #0 thread exited by unexpected error plugin=Fluent::Plugin::ForwardInput title=:event_loop error_class=OpenSSL::SSL::SSLError error="SSL_read: decryption failed or bad record ma$ " 2018-04-03 20:03:27 +0000 [error]: #0 unexpected error error_class=OpenSSL::SSL::SSLError error="SSL_read: decryption failed or bad record mac" 2018-04-03 20:03:27 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/openssl/buffering.rb:182:in `sysread_nonblock' 2018-04-03 20:03:27 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/openssl/buffering.rb:182:in `read_nonblock' 2018-04-03 20:03:27 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:123:in `on_readable' 2018-04-03 20:03:27 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/plugin_helper/server.rb:711:in `on_readable' 2018-04-03 20:03:27 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:186:in `on_readable' 2018-04-03 20:03:27 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once' 2018-04-03 20:03:27 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run' 2018-04-03 20:03:27 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start' 2018-04-03 20:03:27 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create' 2018-04-03 20:03:27 +0000 [error]: #0 unexpected error error_class=OpenSSL::SSL::SSLError error="SSL_read: decryption failed or bad record mac" 2018-04-03 20:03:27 +0000 [error]: #0 suppressed same stacktrace 2018-04-03 20:03:28 +0000 [info]: Worker 0 finished unexpectedly with status 1
Global config:
<source> @type multiprocess <process> cmdline -c /etc/td-agent/child_0.conf --log /var/log/td-agent/child_0.log sleep_before_start 1s sleep_before_shutdown 5s </process> <process> cmdline -c /etc/td-agent/child_1.conf --log /var/log/td-agent/child_1.log sleep_before_start 1s sleep_before_shutdown 5s </process> ... </source>
Child config:
<source> @type forward tag forwarded port 34222 bind 0.0.0.0 source_address_key forwarder_ip source_hostname_key forwarder_hostname <transport tls> ca_cert_path /etc/td-agent/server.pem ca_private_key_path /etc/td-agent/server.key </transport> <security> shared_key key self_hostname harvester-gce-eu-1.zz.zz </security> </source> <filter **> @type record_transformer enable_ruby true <record> pushed_by harvester-gce-eu-2.zz.zz:2 severity ${record["severity"].downcase if record["severity"]} </record> </filter> <match **> @type copy <store> @type elasticsearch host localhost port 9200 target_type_key @target_key type_name fluentd target_index_key target_index logstash_format true logstash_prefix fallback time_key @timestamp include_timestamp true reconnect_on_error true reload_on_failure true buffer_chunk_limit 32m buffer_queue_limit 4 reload_connections false request_timeout 120s <buffer> @type file flush_interval 10s retry_type periodic retry_forever true retry_wait 10s path /var/lib/td-agent/buffers/output_elasticsearch-2 </buffer> </store> </match>
2018-04-03 19:11:00 +0000 [info]: starting fluentd-1.0.2 pid=24923 ruby="2.4.2" 2018-04-03 19:11:00 +0000 [info]: spawn command to main: cmdline=["/opt/td-agent/embedded/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/sbin/td-agent", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"] 2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '2.4.0' 2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-flowcounter' version '1.3.0' 2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-kafka' version '0.6.5' 2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-multiprocess' version '0.2.2' 2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.0.1' 2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.0.1' 2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-s3' version '1.1.0' 2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-td' version '1.0.0' 2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.3' 2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.2.2' 2018-04-03 19:11:00 +0000 [info]: gem 'fluentd' version '1.0.2'
The text was updated successfully, but these errors were encountered:
Hmm... this is similar to https://stackoverflow.com/questions/21497591/urllib2-reading-https-url-failure openssl or something is wrong. Maybe, recover is hard so closing socket is better for next request.
Sorry, something went wrong.
Patch is here: #1942
ed9763f
Merge pull request #1942 from fluent/close-invalid-ssl-socket
410f95b
Close invalid socket when ssl error happens on reading. fix #1924
repeatedly
No branches or pull requests
Fluentd started to crush for unknown reason.
Global config:
Child config:
2018-04-03 19:11:00 +0000 [info]: starting fluentd-1.0.2 pid=24923 ruby="2.4.2"
2018-04-03 19:11:00 +0000 [info]: spawn command to main: cmdline=["/opt/td-agent/embedded/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/sbin/td-agent", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '2.4.0'
2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-flowcounter' version '1.3.0'
2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-kafka' version '0.6.5'
2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-multiprocess' version '0.2.2'
2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.0.1'
2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.0.1'
2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-s3' version '1.1.0'
2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-td' version '1.0.0'
2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.3'
2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.2.2'
2018-04-03 19:11:00 +0000 [info]: gem 'fluentd' version '1.0.2'
The text was updated successfully, but these errors were encountered: