fluentd crash on invalid packet

[dmnfortytwo]

Fluentd started to crush for unknown reason.

2018-04-03 20:03:27 +0000 [warn]: #0 thread exited by unexpected error plugin=Fluent::Plugin::ForwardInput title=:event_loop error_class=OpenSSL::SSL::SSLError error="SSL_read: decryption failed or bad record ma$
"
2018-04-03 20:03:27 +0000 [error]: #0 unexpected error error_class=OpenSSL::SSL::SSLError error="SSL_read: decryption failed or bad record mac"
  2018-04-03 20:03:27 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/openssl/buffering.rb:182:in `sysread_nonblock'
  2018-04-03 20:03:27 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/openssl/buffering.rb:182:in `read_nonblock'
  2018-04-03 20:03:27 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:123:in `on_readable'
  2018-04-03 20:03:27 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/plugin_helper/server.rb:711:in `on_readable'
  2018-04-03 20:03:27 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:186:in `on_readable'
  2018-04-03 20:03:27 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
  2018-04-03 20:03:27 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
  2018-04-03 20:03:27 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
  2018-04-03 20:03:27 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2018-04-03 20:03:27 +0000 [error]: #0 unexpected error error_class=OpenSSL::SSL::SSLError error="SSL_read: decryption failed or bad record mac"
  2018-04-03 20:03:27 +0000 [error]: #0 suppressed same stacktrace
2018-04-03 20:03:28 +0000 [info]: Worker 0 finished unexpectedly with status 1

Global config:

<source>
  @type multiprocess
  <process>
    cmdline -c /etc/td-agent/child_0.conf --log /var/log/td-agent/child_0.log
    sleep_before_start 1s
    sleep_before_shutdown 5s
  </process>
  <process>
    cmdline -c /etc/td-agent/child_1.conf --log /var/log/td-agent/child_1.log
    sleep_before_start 1s
    sleep_before_shutdown 5s
  </process>
...
</source>

Child config:

<source>
  @type forward
  tag forwarded
  port 34222
  bind 0.0.0.0
  source_address_key forwarder_ip
  source_hostname_key forwarder_hostname
    <transport tls>
      ca_cert_path /etc/td-agent/server.pem
      ca_private_key_path /etc/td-agent/server.key
    </transport>
  <security>
    shared_key key
    self_hostname harvester-gce-eu-1.zz.zz
  </security>
</source>
<filter **>
  @type record_transformer
  enable_ruby true
  <record>
    pushed_by harvester-gce-eu-2.zz.zz:2
    severity ${record["severity"].downcase if record["severity"]}
  </record>
</filter>
<match **>
  @type copy
  <store>
    @type elasticsearch
    host localhost
    port 9200
    target_type_key @target_key
    type_name fluentd
    target_index_key target_index
    logstash_format true
    logstash_prefix fallback
    time_key @timestamp
    include_timestamp true
    reconnect_on_error true
    reload_on_failure true
    buffer_chunk_limit 32m
    buffer_queue_limit 4
    reload_connections false
    request_timeout 120s
    <buffer>
      @type file
      flush_interval 10s
      retry_type periodic
      retry_forever true
      retry_wait 10s
      path /var/lib/td-agent/buffers/output_elasticsearch-2
    </buffer>
  </store>
</match>

2018-04-03 19:11:00 +0000 [info]: starting fluentd-1.0.2 pid=24923 ruby="2.4.2"
2018-04-03 19:11:00 +0000 [info]: spawn command to main: cmdline=["/opt/td-agent/embedded/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/sbin/td-agent", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '2.4.0'
2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-flowcounter' version '1.3.0'
2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-kafka' version '0.6.5'
2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-multiprocess' version '0.2.2'
2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.0.1'
2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.0.1'
2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-s3' version '1.1.0'
2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-td' version '1.0.0'
2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.3'
2018-04-03 19:11:00 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.2.2'
2018-04-03 19:11:00 +0000 [info]: gem 'fluentd' version '1.0.2'

[repeatedly]

Hmm... this is similar to https://stackoverflow.com/questions/21497591/urllib2-reading-https-url-failure
openssl or something is wrong.
Maybe, recover is hard so closing socket is better for next request.

[repeatedly]

Patch is here: #1942