Telegraf fails stop sending metrics - took longer to collect than collection interval

[adityask2]

Bug report

Telegraf seems to be choking after certain amount of time. I'm not using aggregators. Seems like there other issues resported on the same lines - #3629

Relevant telegraf.conf:

telegraf.txt

System info:

Telegraf v1.6.0~6e240567 (git: master 6e24056)
Using it on Centos 7

Steps to reproduce:

1. service start telegraf
2. wait for some time (varies from system to system) for it reproduce.

Expected behavior:

Telegraf publishing metrics to mqtt.

Actual behavior:

Telegraf stops publishing metrics seemingly randomly, all input plugins start to fail with:
2018-01-18T16:10:00Z E! Error in plugin [inputs.mem]: took longer to collect than collection interval (10s)
2018-01-18T16:10:00Z E! Error in plugin [inputs.mysql]: took longer to collect than collection interval (10s)
2018-01-18T16:10:00Z E! Error in plugin [inputs.cpu]: took longer to collect than collection interval (10s)
2018-01-18T16:10:10Z E! Error in plugin [inputs.mysql]: took longer to collect than collection interval (10s)
2018-01-18T16:10:10Z E! Error in plugin [inputs.cpu]: took longer to collect than collection interval (10s)
2018-01-18T16:10:10Z E! Error in plugin [inputs.mem]: took longer to collect than collection interval (10s)
2018-01-18T16:10:20Z E! Error in plugin [inputs.mysql]: took longer to collect than collection interval (10s)
2018-01-18T16:10:20Z E! Error in plugin [inputs.cpu]: took longer to collect than collection interval (10s)
2018-01-18T16:10:20Z E! Error in plugin [inputs.mem]: took longer to collect than collection interval (10s)
2018-01-18T16:10:30Z E! Error in plugin [inputs.mysql]: took longer to collect than collection interval (10s)
2018-01-18T16:10:30Z E! Error in plugin [inputs.cpu]: took longer to collect than collection interval (10s)
2018-01-18T16:10:30Z E! Error in plugin [inputs.mem]: took longer to collect than collection interval (10s)
2018-01-18T16:10:40Z E! Error in plugin [inputs.mysql]: took longer to collect than collection interval (10s)
2018-01-18T16:10:40Z E! Error in plugin [inputs.cpu]: took longer to collect than collection interval (10s)
2018-01-18T16:10:40Z E! Error in plugin [inputs.mem]: took longer to collect than collection interval

Additional info:

telegraf-dump.txt

[danielnelson]

This was caused by the mqtt output, I believe this issue I opened upstream is the cause: eclipse-paho/paho.mqtt.golang#185

I should be able to improve the use of timeouts in the output to avoid the hang.

[adityask2]

Hi,
I tested with timeout fix. Telegraf was able to send data successfully for 8 days. However, after 8 days 44 of machines choked. Attaching the telegraf conf.
telegraf.conf.txt

[danielnelson]

Do you think you can try to get another stack dump by killing the process with SIGQUIT?

[adityask2]

I'm running the process in background and had closed the session. Used kill -3 command. However not getting the stack dump. Is there any other way to get it?

[danielnelson]

I don't know of any other way, any log messages when it started or just the took longer to collect than collection interval?

[adityask2]

Attaching telegraf log
telegraf.log

Will see if I can get telegraf dump.

[adityask2]

Does taking core dump through gcore be useful?

[danielnelson]

I haven't tried it, but according to this blog it should work.

[adityask2]

Sharing the dump.
telegraf_coredump_131.txt

[adityask2]

Was able to catch the dump using
kill -SIGABRT
This dumps the stack Trace to journal