emit transaction failed: error_class=NoMethodError error="undefined method `full_pack' #56
Description
Problem
Suddenly, in recent days, td-agent stops sending events via the ES plugin, showing the following for each event in its log file:
2019-08-27 06:57:24 +0000 [warn]: #0 emit transaction failed: error_class=NoMethodError error="undefined method `full_pack' for #<MessagePack::Packer:0x007f3de657dbb0>" location="/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-1.7.0/lib/fluent/event.rb:63:in `to_msgpack_stream'" tag="**************"
From what I could gather, this did not occur on machines where td-agent --version showed "1.6.3", but did start recently in machines with td-agent showing version 1.7.0. Also, looking at the logs, it showed fluentd 1.7.0 instead of 1.6.3 and fluentd elastic search plugin changed from 3.5.4 to 3.5.5. We do not install the plugin directly, but rather via fluent-plugin-aws-elasticsearch-service.
Steps to replicate
in ubuntu:
curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent2.sh | sh
td-agent-gem install fluent-plugin-aws-elasticsearch-service
The td-agent conf file is:
<source>
@type syslog
tag syslog
</source>
<source>
@type tail
format multiline
format_firstline /^\d{4}-\d{2}-\d{2}/
format1 /^(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+)\s+\[(?<pid>.+)\]\s+(?<level>[^ ]+)\s+(?<message>.*)/
time_format %Y-%m-%d %H:%M:%S,%N
time_key timestamp
path /var/log/xxxx/x*
tag xxxx
</source>
<filter **>
@type record_transformer
enable_ruby
remove_keys ["error"]
<record>
hostname "#{Socket.gethostname}"
environment dev
</record>
</filter>
<match fluent.**>
@type null
</match>
<match **>
@type "aws-elasticsearch-service"
type_name "access_log"
logstash_format true
include_tag_key true
tag_key "@log_name"
reload_connections false
flush_interval 1s
<endpoint>
url xxxxxxx
region xxxxx
</endpoint>
<secondary>
@type file
path /var/log/td-agent/failed_records
</secondary>
</match>
Expected Behavior or What you need to ask
error should not be issued, and the event should be emitted correctly to ES
Using Fluentd and ES plugin versions
- Ubuntu 18.04
- td-agent 1.7.0
boot log:
2019-08-27 08:56:08 +0000 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2019-08-27 08:56:08 +0000 [warn]: secondary type should be same with primary one primary="Fluent::TreasureDataLogOutput" secondary="Fluent::Plugin::FileOutput"
2019-08-27 08:56:08 +0000 [info]: using configuration file: <ROOT>
<source>
@type syslog
tag "syslog"
</source>
<source>
@type tail
format multiline
format_firstline /^\d{4}-\d{2}-\d{2}/
format1 /^(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+)\s+\[(?<pid>.+)\]\s+(?<level>[^ ]+)\s+(?<message>.*)/
time_format %Y-%m-%d %H:%M:%S,%N
time_key timestamp
path "/var/log/xxxx/x*"
tag "xxx.xxx"
<parse>
time_key timestamp
time_format %Y-%m-%d %H:%M:%S,%N
format_firstline /^\d{4}-\d{2}-\d{2}/
@type multiline
format1 /^(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+)\s+\[(?<pid>.+)\]\s+(?<level>[^ ]+)\s+(?<message>.*)/
</parse>
</source>
<filter **>
@type record_transformer
enable_ruby
remove_keys ["error"]
<record>
hostname 1fffcfd5bb61
environment staging
</record>
</filter>
<match fluent.**>
@type null
</match>
<match **>
@type aws-elasticsearch-service
type_name "access_log"
logstash_format true
include_tag_key true
tag_key "@log_name"
reload_connections false
flush_interval 1s
<endpoint>
url "xxxxx"
region "xxxxx"
</endpoint>
<secondary>
@type "file"
path "/var/log/td-agent/failed_records"
<buffer time>
path /var/log/td-agent/failed_records
</buffer>
</secondary>
<buffer>
flush_interval 1s
</buffer>
</match>
</ROOT>
2019-08-27 08:56:08 +0000 [info]: starting fluentd-1.7.0 pid=105 ruby="2.1.10"
2019-08-27 08:56:08 +0000 [info]: spawn command to main: cmdline=["/opt/td-agent/embedded/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/sbin/td-agent", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2019-08-27 08:56:08 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2019-08-27 08:56:08 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '2.1.0'
2019-08-27 08:56:08 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '3.5.5'
2019-08-27 08:56:08 +0000 [info]: gem 'fluent-plugin-kafka' version '0.6.1'
2019-08-27 08:56:08 +0000 [info]: gem 'fluent-plugin-mongo' version '0.8.1'
2019-08-27 08:56:08 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.6'
2019-08-27 08:56:08 +0000 [info]: gem 'fluent-plugin-s3' version '0.8.5'
2019-08-27 08:56:08 +0000 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2019-08-27 08:56:08 +0000 [info]: gem 'fluent-plugin-td' version '0.10.29'
2019-08-27 08:56:08 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.3'
2019-08-27 08:56:08 +0000 [info]: gem 'fluent-plugin-webhdfs' version '0.7.1'
2019-08-27 08:56:08 +0000 [info]: gem 'fluentd' version '1.7.0'
2019-08-27 08:56:08 +0000 [info]: gem 'fluentd' version '0.12.40'
2019-08-27 08:56:08 +0000 [info]: adding match pattern="td.*.*" type="tdlog"
2019-08-27 08:56:08 +0000 [warn]: #0 secondary type should be same with primary one primary="Fluent::TreasureDataLogOutput" secondary="Fluent::Plugin::FileOutput"
2019-08-27 08:56:08 +0000 [info]: adding match pattern="debug.**" type="stdout"
2019-08-27 08:56:08 +0000 [info]: adding source type="forward"
2019-08-27 08:56:08 +0000 [info]: adding source type="http"
2019-08-27 08:56:08 +0000 [info]: adding source type="debug_agent"
2019-08-27 08:56:08 +0000 [info]: #0 starting fluentd worker pid=115 ppid=110 worker=0
2019-08-27 08:56:08 +0000 [info]: #0 listening dRuby uri="druby://127.0.0.1:24230" object="Fluent::Engine" worker=0
2019-08-27 08:56:08 +0000 [info]: #0 listening port port=24224 bind="0.0.0.0"
2019-08-27 08:56:08 +0000 [info]: #0 fluentd worker is now running worker=0
*** LOCAL GEMS ***
actionmailer (4.2.8)
actionpack (4.2.8)
actionview (4.2.8)
activejob (4.2.8)
activemodel (4.2.8)
activerecord (4.2.8)
activesupport (4.2.8)
addressable (2.5.2, 2.5.1)
arel (6.0.4)
aws-eventstream (1.0.3)
aws-partitions (1.206.0)
aws-sdk (2.10.45)
aws-sdk-core (3.64.0, 2.10.45)
aws-sdk-resources (2.10.45)
aws-sigv4 (1.1.0, 1.0.2)
bigdecimal (default: 1.2.4)
bson (4.1.1)
builder (3.2.3)
bundler (1.14.5)
celluloid (0.15.2)
concurrent-ruby (1.1.5)
cool.io (1.5.1)
crass (1.0.2)
diff-lcs (1.3)
dig_rb (1.0.1)
draper (1.4.0)
elasticsearch (7.3.0)
elasticsearch-api (7.3.0)
elasticsearch-transport (7.3.0)
erubis (2.7.0)
excon (0.66.0)
faraday (0.15.4)
faraday_middleware-aws-sigv4 (0.2.4)
fluent-logger (0.7.1)
fluent-mixin-plaintextformatter (0.2.6)
fluent-plugin-aws-elasticsearch-service (2.1.0)
fluent-plugin-elasticsearch (3.5.5)
fluent-plugin-kafka (0.6.1)
fluent-plugin-mongo (0.8.1)
fluent-plugin-rewrite-tag-filter (1.5.6)
fluent-plugin-s3 (0.8.5)
fluent-plugin-scribe (0.10.14)
fluent-plugin-td (0.10.29)
fluent-plugin-td-monitoring (0.2.3)
fluent-plugin-webhdfs (0.7.1)
fluentd (1.7.0, 0.12.40)
fluentd-ui (0.4.4)
font-awesome-rails (4.7.0.1)
globalid (0.4.0)
haml (4.0.7)
haml-rails (0.5.3)
hike (1.2.3)
hirb (0.7.3)
http_parser.rb (0.6.0)
httpclient (2.8.2.4)
i18n (0.8.1)
io-console (default: 0.4.3)
ipaddress (0.8.3)
jbuilder (2.6.3)
jmespath (1.3.1)
jquery-rails (3.1.4)
json (default: 1.8.1)
kramdown (1.13.2)
kramdown-haml (0.0.3)
loofah (2.1.1, 2.0.3)
ltsv (0.1.0)
mail (2.6.4)
mime-types (3.1)
mime-types-data (3.2016.0521)
mini_portile2 (2.3.0, 2.1.0)
minitest (5.10.1, default: 4.7.5)
mixlib-cli (1.7.0)
mixlib-config (2.2.4)
mixlib-log (1.7.1)
mixlib-shellout (2.2.7)
mongo (2.2.7)
msgpack (1.1.0)
multi_json (1.12.1)
multipart-post (2.1.1)
nokogiri (1.8.1)
ohai (6.20.0)
oj (2.18.1)
parallel (1.8.0)
psych (default: 2.0.5)
public_suffix (3.0.0, 2.0.5)
puma (3.8.2)
rack (1.6.5)
rack-test (0.6.3)
rails (4.2.8)
rails-deprecated_sanitizer (1.0.3)
rails-dom-testing (1.0.8)
rails-html-sanitizer (1.0.3)
railties (4.2.8)
rake (default: 10.1.0)
rdoc (default: 4.1.0)
request_store (1.3.2)
ruby-kafka (0.4.2)
ruby-progressbar (1.8.3)
rubyzip (1.2.1, 1.1.7)
sass (3.2.19)
sass-rails (4.0.5)
serverengine (2.1.1)
settingslogic (2.0.9)
sigdump (0.2.4)
sprockets (2.12.4)
sprockets-rails (2.3.3)
string-scrub (0.0.5)
strptime (0.2.3)
sucker_punch (1.0.5)
systemu (2.5.2)
td (0.15.2)
td-client (0.8.85)
td-logger (0.3.27)
test-unit (default: 2.1.10.0)
thor (0.19.4)
thread_safe (0.3.6)
thrift (0.8.0)
tilt (1.4.1)
timers (1.1.0)
tzinfo (2.0.0, 1.2.3)
tzinfo-data (1.2017.2)
uuidtools (2.1.5)
webhdfs (0.8.0)
yajl-ruby (1.3.0)
zip-zip (0.3)
- ES version (6.x)