telegraf just stops working

[RainerW]

i'm testing telegraf on different systems and it seems to stop working after some time.
I guess tehere is a kind of network hickup, because it stops working on multipe servers at around the same time. But some Servers just continued to work. So I'm sure neither influx nor grafana had a problem.

The Systems in question are most Ubuntu 11,14 or 16. But one of the ubuntu 16 servers continue to work.
In all cases the logfiles just stopped containing anything, but the process continues to run. My guess is that there is no safeguard around the metrics, so when they start hanging for whatever reason telegraf stops working .?

Last Log entries are:

2016/05/13 11:15:30 Wrote 21 metrics to output influxdb in 46.968928ms
2016/05/13 11:15:40 Gathered metrics, (10s interval), from 11 inputs in 33.493802ms

Which is the point in time where telegraf stops reporting.
It seems to be still running:

> ps aux | grep tel
telegraf  7095  0.0  0.0 129524  3372 ?        Sl   May13   0:25 /usr/bin/telegraf -pidfile /var/run/telegraf/telegraf.pid -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d

After an service restart all is working fine again. But this happend before, so i guess it will also happen again.

I know this ticket is very broad, but i have nothing to pin it down to. Only there should be safeguards in place to prevent telegraf from stopping completely.

[RainerW]

Oh forgot, this happens with 0.12.1 and 0.13.0

[sparrc]

can you provide your configuration?

[sparrc]

recently in 0.13 there were some safeguards added to prevent lockups when running exec commands, but these have not all been patched up as some of Telegraf's dependencies could still be running commands without timeouts.

[RainerW]

This is basically the default config with :

• NFS excluded
• and apache monitoring enabled
• telegraf 0.12.1 in that case

https://gist.github.com/RainerW/94c11ac7f4ce42ea17a12ee7e40257eb

[RainerW]

On the Server with 0.13.0

• NFS and glusterfs excluded
• apache monitoring added via telegraf.d folder

https://gist.github.com/RainerW/586508bc90bca51e9d8cab0dfe98d97a

[RainerW]

hint: Apache status check is not active on all instances. on one server i had default 0.12.1 config (Onle nfs and glusterfs excludes) https://gist.github.com/RainerW/d3a5f38c1b69d13f69c75e1b1778ccee with the same effect

[sparrc]

If you could SIGQUIT (ctrl-\) the process while it's hung and provide the stack trace that would help a lot

[RainerW]

Not totally sure how to do that with a service. "kill -3 6413" just quits the process but the log does not contain a stracktrace (At least a restarted instance i tried, which was not hung but i would like to see a stracktrace first before using it on a hung instance ) :
2016/05/19 16:22:40 Gathered metrics, (10s interval), from 8 inputs in 54.154783ms
Verlassen --> "german for exited"

[sparrc]

that's odd, when I send a SIGQUIT I get a stack trace in the logs.... I'm not sure then what's going on with yours, is the disk full or locked to where it can't even write? (or maybe a permissions problem?)

[sparrc]

since it's happening with the default config, it's likely related to #1215. It could even be the exact same problem.

[sparrc]

You could get hung child processes while the telegraf process is hung with something like:

ps aux -o ppid | grep `pgrep telegraf`

[sparrc]

been kicking this around in the back of my head for a bit, and this seems like as good a reason as any to get it implemented: #1235

[RainerW]

lol, got a stracktrace after fixing an crashed nfs server/mount ... which simultaniously fixed the remaining not restarted servers. So now all telegraf insances are running again.

Sadly the one server which had the problem, but did not have that nfs mount had beend restarted earlier. So I cannot provide a stacktrace of a hung instance. But it seems at least to be a problem while accessing the disk statistics, even i had beend excluding "nfs" (or because?)