node_exporter diskstats collector failed

[lastsky]

have you any ideas how to troubleshoot this?

symptoms:

• curl http://127.0.0.1:9100/metrics -> connection refused
• node_exporter eats all cpu (100%)
• no metrics

frequency: 24-72h on any server (any of 4 servers with unique detail have the same symptom)

unique server's detail:

*-storage
description: RAID bus controller
product: MegaRAID SAS-3 3108 [Invader]

workaround: service node_exporter restart

start string:
/opt/node_exporter --collector.textfile.directory=/var/lib/node_exporter -collectors.enabled=conntrack,diskstats,entropy,filefd,filesystem,hwmon,loadavg,mdadm,meminfo,netdev,netstat,sockstat,stat,textfile,time,uname,vmstat,bonding,megacli,runit -collector.megacli.command=/opt/sudo_megacli.sh

versions: all previous and
node_exporter_build_info{branch="*******",env="********",goversion="go1.7.3",instance="*************",job="node",revision="42d1658d5632fe0bfb281761d5e3620cfc37990b",version="0.13.0-rc.2"}

logs:
# cat /var/log/node_exporter/current 2016-12-28_22:43:37.76655 time="2016-12-28T22:43:37Z" level=error msg="ERROR: diskstats collector failed after 0.002182s: couldn't get diskstats: invalid line in /proc/diskstats: \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00" source="node_exporter.go:91"

sudo cat '/var/log/node_exporter/current' | grep "ERROR" 2016-12-26_08:00:22.41718 time="2016-12-26T08:00:22Z" level=error msg="ERROR: diskstats collector failed after 0.003493s: couldn't get diskstats: invalid line in /proc/diskstats: \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00" source="node_exporter.go:91"

diskstats example:

   1       0 ram0 0 0 0 0 0 0 0 0 0 0 0
   1       1 ram1 0 0 0 0 0 0 0 0 0 0 0
   1       2 ram2 0 0 0 0 0 0 0 0 0 0 0
   1       3 ram3 0 0 0 0 0 0 0 0 0 0 0
   1       4 ram4 0 0 0 0 0 0 0 0 0 0 0
   1       5 ram5 0 0 0 0 0 0 0 0 0 0 0
   1       6 ram6 0 0 0 0 0 0 0 0 0 0 0
   1       7 ram7 0 0 0 0 0 0 0 0 0 0 0
   1       8 ram8 0 0 0 0 0 0 0 0 0 0 0
   1       9 ram9 0 0 0 0 0 0 0 0 0 0 0
   1      10 ram10 0 0 0 0 0 0 0 0 0 0 0
   1      11 ram11 0 0 0 0 0 0 0 0 0 0 0
   1      12 ram12 0 0 0 0 0 0 0 0 0 0 0
   1      13 ram13 0 0 0 0 0 0 0 0 0 0 0
   1      14 ram14 0 0 0 0 0 0 0 0 0 0 0
   1      15 ram15 0 0 0 0 0 0 0 0 0 0 0
   7       0 loop0 0 0 0 0 0 0 0 0 0 0 0
   7       1 loop1 0 0 0 0 0 0 0 0 0 0 0
   7       2 loop2 0 0 0 0 0 0 0 0 0 0 0
   7       3 loop3 0 0 0 0 0 0 0 0 0 0 0
   7       4 loop4 0 0 0 0 0 0 0 0 0 0 0
   7       5 loop5 0 0 0 0 0 0 0 0 0 0 0
   7       6 loop6 0 0 0 0 0 0 0 0 0 0 0
   7       7 loop7 0 0 0 0 0 0 0 0 0 0 0
   8       0 sda 363780 60548 20615176 76348 153711378 69466958 5358538074 9181108 0 4049156 9163092
   8       1 sda1 3159 0 10374 28 19 8 66 0 0 28 28
   8       2 sda2 4 0 8 0 0 0 0 0 0 0 0
   8       5 sda5 359940 60548 20599354 76304 153711359 69466950 5358538008 9181108 0 4049164 9163360
   8      16 sdb 321985188 0 356189572 13448624 21872485 920561279 4586388586 129760020 0 8985648 143020888
   8      17 sdb1 321982277 0 356161374 13448520 21850656 920561279 4575212138 129753492 0 8984180 143014676
 252       0 dm-0 82810 0 662728 18160 1414122 0 11312976 9832164 0 15748 9850424
 252       1 dm-1 338156 0 19865506 71972 221765537 0 5347225032 20749572 0 4291752 21030628

[SuperQ]

It looks like the node_exporter is having trouble reading your diskstats file. What kernel version are you running?

[lastsky]

my kernel is:
Linux 4.2.0-35-generic #40~14.04.1-Ubuntu SMP Fri Mar 18 16:37:35 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

there are several sensitive to disk io apps on servers, no problems found. disk metrics are normal and disk i/o load is low.

Problem for reading diskstats file repeats on 4 (four) different physical servers. But all of them are running with the same kernel, on the same server platform and model, and on the same ssd type and model.

distribution is:

# cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.4 LTS"
NAME="Ubuntu"
VERSION="14.04.4 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.4 LTS"
VERSION_ID="14.04"

[lastsky]

should I disable some devices? should I test it with changing node_exporter start string?

-collector.diskstats.ignored-devices="^(ram|loop|fd|(h|s|v|xv)d[a-z])\\d+$": Regexp of devices to ignore for diskstats.

[juliusv]

Ignoring devices won't help if the entire diskstats file cannot be parsed. The log invalid line in /proc/diskstats: \x00\x00\x00... suggests that Node Exporter sees a bunch of binary zeros in there for some reason. Would you be able to attach a binary copy of your /proc/diskstats here?

[lastsky]

i use runit as supervisor and added to all startup files string: -collector.diskstats.ignored-devices="^(ram|loop|fd|(h|s|v|xv)d[a-z])\\d+$"

no changes, node_exporter has crashed again.

[lastsky]

and binary /proc/diskstats is the same as I show it with cat command

[lastsky]

latest log:
2017-01-07_01:03:56.63254 fatal error: stack growth after fork
2017-01-07_01:03:56.63259 fatal error: unexpected signal during runtime execution

looks like something other. i dunno how to continue debug.

[lastsky]

-log.level="debug" added.

[lastsky]

#228 - this is the same bug on the same server.

also the same records:

2017-01-03_20:44:57.51836 time="2017-01-03T20:44:57Z" level=info msg="Listening on :9100" source="node_exporter.go:176"`
2017-01-04_07:06:57.17592 fatal error: systemstack called from unexpected goroutine
2017-01-04_07:06:57.17595 fatal error: runtime·unlock: lock count
2017-01-04_07:06:57.17596 fatal error: runtime·unlock: lock count
2017-01-04_07:06:57.17596 fatal error: runtime·unlock: lock count
2017-01-04_07:06:57.17597 fatal error: runtime·unlock: lock count
2017-01-04_07:06:57.17597 fatal error: runtime·unlock: lock count
2017-01-04_07:06:57.17598 fatal error: runtime·unlock: lock count
2017-01-04_07:06:57.17598 fatal error: runtime·unlock: lock count
2017-01-04_07:06:57.17599 fatal error: runtime·unlock: lock count
2017-01-04_07:06:57.17599 fatal error: runtime·unlock: lock count
2017-01-04_07:06:57.17600 fatal error: runtime·unlock: lock count
2017-01-04_07:06:57.17601 fatal error: runtime·unlock: lock count
2017-01-04_07:06:57.17601 fatal error: runtime·unlock: lock count
2017-01-04_07:06:57.17602 fatal error: runtime·unlock: lock count
2017-01-04_07:06:57.17602 fatal error: runtime·unlock: lock count
2017-01-04_07:06:57.17603 fatal error: runtime·unlock: lock count
2017-01-04_07:06:57.17603 fatal error: runtime·unlock: lock count
2017-01-04_07:06:57.17604 runtime: newstack called from g=0xc420589380
2017-01-04_07:06:57.17605 	m=0xc420296000 m->curg=0x0 m->g0=0xc4202a21a0 m->gsignal=0xc4202a2000

[SuperQ]

@juliusv I wonder if this would be better with the current master where we converted to Const metrics.

[lastsky]

2017-01-09_17:32:02.30524 time="2017-01-09T17:32:02Z" level=debug msg="OK: textfile collector succeeded after 0.000000s." source="node_exporter.go:94"
2017-01-09_17:32:02.30527 time="2017-01-09T17:32:02Z" level=debug msg="Set time: 1483983122.000000" source="time.go:47"
2017-01-09_17:32:02.30528 time="2017-01-09T17:32:02Z" level=debug msg="OK: time collector succeeded after 0.000141s." source="node_exporter.go:94"
2017-01-09_17:32:02.30537 time="2017-01-09T17:32:02Z" level=debug msg="OK: sockstat collector succeeded after 0.000218s." source="node_exporter.go:94"
2017-01-09_17:32:02.30606 fatal error: schedule: holding locks
2017-01-09_17:32:02.30935
2017-01-09_17:32:02.30936 runtime stack:
2017-01-09_17:32:02.30937 runtime.throw(0x8f0a50, 0x17)
2017-01-09_17:32:02.30937 	/usr/local/go/src/runtime/panic.go:566 +0x95
2017-01-09_17:32:02.30938 runtime.schedule()
2017-01-09_17:32:02.30940 	/usr/local/go/src/runtime/proc.go:2074 +0x33b
2017-01-09_17:32:02.30941 runtime.goexit0(0xc4203d3d40)
2017-01-09_17:32:02.30941 	/usr/local/go/src/runtime/proc.go:2257 +0x180
2017-01-09_17:32:02.30942 runtime.mcall(0x0)
2017-01-09_17:32:02.30943 	/usr/local/go/src/runtime/asm_amd64.s:240 +0x5b
2017-01-09_17:32:02.30943
2017-01-09_17:32:02.30944 goroutine 1 [IO wait]:
2017-01-09_17:32:02.30944 net.runtime_pollWait(0x7fb0ea764228, 0x72, 0x0)
2017-01-09_17:32:02.30945 	/usr/local/go/src/runtime/netpoll.go:160 +0x59
2017-01-09_17:32:02.30946 net.(*pollDesc).wait(0xc4201db170, 0x72, 0xc420047a48, 0xc4200820a0)
2017-01-09_17:32:02.30946 	/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
2017-01-09_17:32:02.30947 net.(*pollDesc).waitRead(0xc4201db170, 0xafd1c0, 0xc4200820a0)
2017-01-09_17:32:02.30948 	/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
2017-01-09_17:32:02.30948 net.(*netFD).accept(0xc4201db110, 0x0, 0xafb900, 0xc420766340)
2017-01-09_17:32:02.30949 	/usr/local/go/src/net/fd_unix.go:419 +0x238
2017-01-09_17:32:02.30950 net.(*TCPListener).accept(0xc420088130, 0x29e8d60800, 0x0, 0x0)
2017-01-09_17:32:02.30950 	/usr/local/go/src/net/tcpsock_posix.go:132 +0x2e
2017-01-09_17:32:02.30951 net.(*TCPListener).AcceptTCP(0xc420088130, 0xc420047b70, 0xc420047b78, 0xc420047b68)
2017-01-09_17:32:02.30953 	/usr/local/go/src/net/tcpsock.go:209 +0x49
2017-01-09_17:32:02.30954 net/http.tcpKeepAliveListener.Accept(0xc420088130, 0x92b9a8, 0xc42052e000, 0xb00d40, 0xc4201deb10)
2017-01-09_17:32:02.30954 	/usr/local/go/src/net/http/server.go:2608 +0x2f
2017-01-09_17:32:02.30955 net/http.(*Server).Serve(0xc42008c900, 0xb00500, 0xc420088130, 0x0, 0x0)
2017-01-09_17:32:02.30955 	/usr/local/go/src/net/http/server.go:2273 +0x1ce
2017-01-09_17:32:02.30956 net/http.(*Server).ListenAndServe(0xc42008c900, 0xc42008c900, 0x2)
2017-01-09_17:32:02.30956 	/usr/local/go/src/net/http/server.go:2219 +0xb4
2017-01-09_17:32:02.30957 net/http.ListenAndServe(0x8e3d54, 0x5, 0x0, 0x0, 0xc4201cde30, 0xc420083a58)
2017-01-09_17:32:02.30957 	/usr/local/go/src/net/http/server.go:2351 +0xa0
2017-01-09_17:32:02.30958 main.main()
2017-01-09_17:32:02.30958 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:177 +0xa5d
2017-01-09_17:32:02.30959
2017-01-09_17:32:02.30960 goroutine 17 [syscall, 2727 minutes, locked to thread]:
2017-01-09_17:32:02.30960 runtime.goexit()
2017-01-09_17:32:02.30961 	/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1
2017-01-09_17:32:02.30962
2017-01-09_17:32:02.30962 goroutine 705189 [runnable]:
2017-01-09_17:32:02.30962 main.NodeCollector.Collect.func1(0xc42071c1e0, 0xc4202660b0, 0x7ffcddd00e64, 0x7, 0xafa300, 0xc4200880b8)
2017-01-09_17:32:02.30963 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:64
2017-01-09_17:32:02.30964 created by main.NodeCollector.Collect
2017-01-09_17:32:02.30964 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x120
2017-01-09_17:32:02.30965
2017-01-09_17:32:02.30965 goroutine 705197 [runnable]:
2017-01-09_17:32:02.30966 main.NodeCollector.Collect.func1(0xc42071c1e0, 0xc4202660b0, 0x7ffcddd00ee2, 0x5, 0xafa740, 0xc4201d4d20)
2017-01-09_17:32:02.30967 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:64
2017-01-09_17:32:02.30967 created by main.NodeCollector.Collect
2017-01-09_17:32:02.30968 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x120
2017-01-09_17:32:02.30968
2017-01-09_17:32:02.30969 goroutine 705198 [runnable]:
2017-01-09_17:32:02.30970 syscall.Lstat(0xc420392f60, 0x23, 0xc420288038, 0x0, 0xc4206056a0)
2017-01-09_17:32:02.30970 	/usr/local/go/src/syscall/zsyscall_linux_amd64.go:1349
2017-01-09_17:32:02.30971 os.Lstat(0xc420392f60, 0x23, 0x17, 0x8e2b3d, 0x1, 0xc4204ca2e0)
2017-01-09_17:32:02.30971 	/usr/local/go/src/os/file_unix.go:183 +0x7d
2017-01-09_17:32:02.30972 os.(*File).readdir(0xc420036030, 0xffffffffffffffff, 0x0, 0x0, 0xc420036030, 0x0, 0x0)
2017-01-09_17:32:02.30973 	/usr/local/go/src/os/file_unix.go:199 +0x184
2017-01-09_17:32:02.30973 os.(*File).Readdir(0xc420036030, 0xffffffffffffffff, 0xc420036030, 0x0, 0x0, 0x0, 0xc420605830)
2017-01-09_17:32:02.30974 	/usr/local/go/src/os/doc.go:119 +0x3e
2017-01-09_17:32:02.30975 io/ioutil.ReadDir(0xc4200ea820, 0x17, 0x1, 0xc4204ca130, 0xa, 0xc4200eabc0, 0x13)
2017-01-09_17:32:02.30976 	/usr/local/go/src/io/ioutil/ioutil.go:105 +0x7c
2017-01-09_17:32:02.30977 github.com/prometheus/node_exporter/collector.collectSensorData(0xc4200ea820, 0x17, 0xc420605b78, 0xc420605c68, 0xc420605b78)
2017-01-09_17:32:02.30979 	/go/src/github.com/prometheus/node_exporter/collector/hwmon_linux.go:107 +0x60
2017-01-09_17:32:02.30980 github.com/prometheus/node_exporter/collector.(*hwMonCollector).updateHwmon(0xb405b0, 0xc42071c1e0, 0xc4200ea820, 0x17, 0x0, 0x0)
2017-01-09_17:32:02.30980 	/go/src/github.com/prometheus/node_exporter/collector/hwmon_linux.go:135 +0x141
2017-01-09_17:32:02.30981 github.com/prometheus/node_exporter/collector.(*hwMonCollector).Update(0xb405b0, 0xc42071c1e0, 0xb23d60, 0x800000000000002)
2017-01-09_17:32:02.30982 	/go/src/github.com/prometheus/node_exporter/collector/hwmon_linux.go:379 +0x298
2017-01-09_17:32:02.30983 main.execute(0x7ffcddd00e7e, 0x5, 0xafa400, 0xb405b0, 0xc42071c1e0)
2017-01-09_17:32:02.30984 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:86 +0x97
2017-01-09_17:32:02.30985 main.NodeCollector.Collect.func1(0xc42071c1e0, 0xc4202660b0, 0x7ffcddd00e7e, 0x5, 0xafa400, 0xb405b0)
2017-01-09_17:32:02.30986 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:65 +0x53
2017-01-09_17:32:02.30987 created by main.NodeCollector.Collect
2017-01-09_17:32:02.30988 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x120
2017-01-09_17:32:02.30988
2017-01-09_17:32:02.30989 goroutine 705190 [runnable]:
2017-01-09_17:32:02.30990 strings.Join(0xc4207eb978, 0x2, 0x2, 0x8e2b3d, 0x1, 0x0, 0x0)
2017-01-09_17:32:02.30991 	/usr/local/go/src/strings/strings.go:362 +0x286
2017-01-09_17:32:02.30991 path.Join(0xc4207eb978, 0x2, 0x2, 0x0, 0x0)
2017-01-09_17:32:02.30992 	/usr/local/go/src/path/path.go:153 +0xa7
2017-01-09_17:32:02.30993 github.com/prometheus/node_exporter/collector.procFilePath(0x8e5568, 0x6, 0x0, 0x0)
2017-01-09_17:32:02.30993 	/go/src/github.com/prometheus/node_exporter/collector/paths.go:30 +0x91
2017-01-09_17:32:02.30994 github.com/prometheus/node_exporter/collector.mountPointDetails(0x0, 0x0, 0x0, 0x0, 0x0)
2017-01-09_17:32:02.30995 	/go/src/github.com/prometheus/node_exporter/collector/filesystem_linux.go:83 +0x87
2017-01-09_17:32:02.30996 github.com/prometheus/node_exporter/collector.(*filesystemCollector).GetStats(0xc420081300, 0x2a, 0x41099c, 0xc420034540, 0x413395, 0xc4206520a0)
2017-01-09_17:32:02.30997 	/go/src/github.com/prometheus/node_exporter/collector/filesystem_linux.go:41 +0x5a
2017-01-09_17:32:02.30998 github.com/prometheus/node_exporter/collector.(*filesystemCollector).Update(0xc420081300, 0xc42071c1e0, 0xb23d60, 0x800000000000005)
2017-01-09_17:32:02.30999 	/go/src/github.com/prometheus/node_exporter/collector/filesystem_common.go:118 +0x43
2017-01-09_17:32:02.31000 main.execute(0x7ffcddd00e73, 0xa, 0xafa380, 0xc420081300, 0xc42071c1e0)
2017-01-09_17:32:02.31000 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:86 +0x97
2017-01-09_17:32:02.31001 main.NodeCollector.Collect.func1(0xc42071c1e0, 0xc4202660b0, 0x7ffcddd00e73, 0xa, 0xafa380, 0xc420081300)
2017-01-09_17:32:02.31002 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:65 +0x53
2017-01-09_17:32:02.31003 created by main.NodeCollector.Collect
2017-01-09_17:32:02.31003 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x120
2017-01-09_17:32:02.31004
2017-01-09_17:32:02.31005 goroutine 705199 [runnable]:
2017-01-09_17:32:02.31005 strings.Join(0xc4203a7c90, 0x2, 0x2, 0x8e2b3d, 0x1, 0x80000, 0x8e3d40)
2017-01-09_17:32:02.31006 	/usr/local/go/src/strings/strings.go:362 +0x286
2017-01-09_17:32:02.31007 path.Join(0xc4203a7c90, 0x2, 0x2, 0x8e5538, 0x6)
2017-01-09_17:32:02.31008 	/usr/local/go/src/path/path.go:153 +0xa7
2017-01-09_17:32:02.31008 github.com/prometheus/node_exporter/collector.procFilePath(0x8e677e, 0x7, 0xc4206bcd28, 0x4e9ec6)
2017-01-09_17:32:02.31009 	/go/src/github.com/prometheus/node_exporter/collector/paths.go:30 +0x91
2017-01-09_17:32:02.31010 github.com/prometheus/node_exporter/collector.getMemInfo(0x0, 0x0, 0x0)
2017-01-09_17:32:02.31011 	/go/src/github.com/prometheus/node_exporter/collector/meminfo_linux.go:67 +0x55
2017-01-09_17:32:02.31012 github.com/prometheus/node_exporter/collector.(*meminfoCollector).Update(0xb405b0, 0xc42071c1e0, 0xb23d60, 0x1)
2017-01-09_17:32:02.31013 	/go/src/github.com/prometheus/node_exporter/collector/meminfo_linux.go:48 +0x3f
2017-01-09_17:32:02.31013 main.execute(0x7ffcddd00e92, 0x7, 0xafa600, 0xb405b0, 0xc42071c1e0)
2017-01-09_17:32:02.31014 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:86 +0x97
2017-01-09_17:32:02.31015 main.NodeCollector.Collect.func1(0xc42071c1e0, 0xc4202660b0, 0x7ffcddd00e92, 0x7, 0xafa600, 0xb405b0)
2017-01-09_17:32:02.31016 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:65 +0x53
2017-01-09_17:32:02.31017 created by main.NodeCollector.Collect
2017-01-09_17:32:02.31017 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x120
2017-01-09_17:32:02.31018
2017-01-09_17:32:02.31018 goroutine 705192 [runnable]:
2017-01-09_17:32:02.31019 bufio.(*Scanner).Scan(0xc420613ab8, 0x0)
2017-01-09_17:32:02.31020 	/usr/local/go/src/bufio/scan.go:128
2017-01-09_17:32:02.31021 github.com/prometheus/node_exporter/collector.parseNetStats(0xafb840, 0xc4201b4030, 0xc4208a8010, 0xe, 0x0, 0x5, 0x8e7689)
2017-01-09_17:32:02.31022 	/go/src/github.com/prometheus/node_exporter/collector/netstat_linux.go:95 +0xd1
2017-01-09_17:32:02.31023 github.com/prometheus/node_exporter/collector.getNetStats(0xc4208a8010, 0xe, 0x0, 0x0, 0x0)
2017-01-09_17:32:02.31023 	/go/src/github.com/prometheus/node_exporter/collector/netstat_linux.go:86 +0xdf
2017-01-09_17:32:02.31024 github.com/prometheus/node_exporter/collector.(*netStatCollector).Update(0xb405b0, 0xc42071c1e0, 0xb23d60, 0x80000000051216f)
2017-01-09_17:32:02.31025 	/go/src/github.com/prometheus/node_exporter/collector/netstat_linux.go:50 +0xc7
2017-01-09_17:32:02.31026 main.execute(0x7ffcddd00ea1, 0x7, 0xafa6c0, 0xb405b0, 0xc42071c1e0)
2017-01-09_17:32:02.31027 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:86 +0x97
2017-01-09_17:32:02.31027 main.NodeCollector.Collect.func1(0xc42071c1e0, 0xc4202660b0, 0x7ffcddd00ea1, 0x7, 0xafa6c0, 0xb405b0)
2017-01-09_17:32:02.31028 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:65 +0x53
2017-01-09_17:32:02.31029 created by main.NodeCollector.Collect
2017-01-09_17:32:02.31030 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x120
2017-01-09_17:32:02.31031
2017-01-09_17:32:02.31031 goroutine 705187 [runnable]:
2017-01-09_17:32:02.31032 sync.runtime_Semacquire(0xc420062b70)
2017-01-09_17:32:02.31032 	/usr/local/go/src/runtime/sema.go:47 +0x30
2017-01-09_17:32:02.31033 sync.(*Mutex).Lock(0xc420062b6c)
2017-01-09_17:32:02.31034 	/usr/local/go/src/sync/mutex.go:85 +0xd0
2017-01-09_17:32:02.31034 github.com/prometheus/node_exporter/vendor/github.com/Sirupsen/logrus.Entry.log(0xc420062b40, 0xc42034c510, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x83e505, 0xc4205b03c0, ...)
2017-01-09_17:32:02.31035 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/Sirupsen/logrus/entry.go:101 +0x306
2017-01-09_17:32:02.31036 github.com/prometheus/node_exporter/vendor/github.com/Sirupsen/logrus.(*Entry).Debug(0xc4205b0040, 0xc4207eadc8, 0x1, 0x1)
2017-01-09_17:32:02.31037 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/Sirupsen/logrus/entry.go:119 +0x9a
2017-01-09_17:32:02.31038 github.com/prometheus/node_exporter/vendor/github.com/Sirupsen/logrus.(*Entry).Debugf(0xc4205b0040, 0x8f875c, 0x25, 0xc4207eaee8, 0x2, 0x2)
2017-01-09_17:32:02.31039 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/Sirupsen/logrus/entry.go:167 +0x102
2017-01-09_17:32:02.31040 github.com/prometheus/node_exporter/vendor/github.com/prometheus/common/log.Debugf(0x8f875c, 0x25, 0xc4207eaee8, 0x2, 0x2)
2017-01-09_17:32:02.31041 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/common/log/log.go:243 +0x6d
2017-01-09_17:32:02.31042 main.execute(0x7ffcddd00e50, 0x9, 0xafa280, 0xc420083cd0, 0xc42071c1e0)
2017-01-09_17:32:02.31043 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:94 +0x4b1
2017-01-09_17:32:02.31044 main.NodeCollector.Collect.func1(0xc42071c1e0, 0xc4202660b0, 0x7ffcddd00e50, 0x9, 0xafa280, 0xc420083cd0)
2017-01-09_17:32:02.31045 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:65 +0x53
2017-01-09_17:32:02.31046 created by main.NodeCollector.Collect
2017-01-09_17:32:02.31046 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x120
2017-01-09_17:32:02.31047
2017-01-09_17:32:02.31047 goroutine 705125 [semacquire]:
2017-01-09_17:32:02.31048 sync.runtime_Semacquire(0xc4207777ac)
2017-01-09_17:32:02.31049 	/usr/local/go/src/runtime/sema.go:47 +0x30
2017-01-09_17:32:02.31049 sync.(*WaitGroup).Wait(0xc4207777a0)
2017-01-09_17:32:02.31050 	/usr/local/go/src/sync/waitgroup.go:131 +0x97
2017-01-09_17:32:02.31051 github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB.func1(0xc4207777a0, 0xc42071c1e0)
2017-01-09_17:32:02.31051 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:409 +0x2d
2017-01-09_17:32:02.31052 created by github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB
2017-01-09_17:32:02.31053 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:411 +0x1db
2017-01-09_17:32:02.31054
2017-01-09_17:32:02.31054 goroutine 705194 [runnable]:
2017-01-09_17:32:02.31055 os.(*File).Close(0xc420794000, 0x0, 0x0)
2017-01-09_17:32:02.31055 	/usr/local/go/src/os/file_unix.go:128
2017-01-09_17:32:02.31056 io/ioutil.ReadFile(0xc4206f8020, 0x1e, 0xc42068c400, 0x10, 0x1200, 0x0, 0x0)
2017-01-09_17:32:02.31056 	/usr/local/go/src/io/ioutil/ioutil.go:70 +0x167
2017-01-09_17:32:02.31057 github.com/prometheus/node_exporter/collector.readBondingStats(0xc4203ea010, 0xe, 0xc4203ea010, 0xe, 0x5402d1)
2017-01-09_17:32:02.31058 	/go/src/github.com/prometheus/node_exporter/collector/bonding_linux.go:76 +0x13a
2017-01-09_17:32:02.31058 github.com/prometheus/node_exporter/collector.(*bondingCollector).Update(0xc4201cc8e0, 0xc42071c1e0, 0xb23d60, 0x5)
2017-01-09_17:32:02.31059 	/go/src/github.com/prometheus/node_exporter/collector/bonding_linux.go:61 +0x6d
2017-01-09_17:32:02.31060 main.execute(0x7ffcddd00ed2, 0x7, 0xafa240, 0xc4201cc8e0, 0xc42071c1e0)
2017-01-09_17:32:02.31060 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:86 +0x97
2017-01-09_17:32:02.31061 main.NodeCollector.Collect.func1(0xc42071c1e0, 0xc4202660b0, 0x7ffcddd00ed2, 0x7, 0xafa240, 0xc4201cc8e0)
2017-01-09_17:32:02.31062 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:65 +0x53
2017-01-09_17:32:02.31063 created by main.NodeCollector.Collect
2017-01-09_17:32:02.31063 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x120
2017-01-09_17:32:02.31064
2017-01-09_17:32:02.31064 goroutine 705123 [runnable]:
2017-01-09_17:32:02.31069 github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB(0xc42000c4e0, 0xafc500, 0xc420766820, 0x0, 0x0)
2017-01-09_17:32:02.31070 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:427 +0x321
2017-01-09_17:32:02.31072 github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).ServeHTTP(0xc42000c4e0, 0xaffc00, 0xc420088598, 0xc4204c8000)
2017-01-09_17:32:02.31073 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:375 +0x13b
2017-01-09_17:32:02.31074 net/http.(Handler).ServeHTTP-fm(0xaffc00, 0xc420088598, 0xc4204c8000)
2017-01-09_17:32:02.31075 	/usr/local/go/src/net/http/h2_bundle.go:4084 +0x4d
2017-01-09_17:32:02.31075 github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0xb00140, 0xc42078a410, 0xc4204c8000)
2017-01-09_17:32:02.31076 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/http.go:158 +0x2ab
2017-01-09_17:32:02.31077 net/http.HandlerFunc.ServeHTTP(0xc42007d0e0, 0xb00140, 0xc42078a410, 0xc4204c8000)
2017-01-09_17:32:02.31078 	/usr/local/go/src/net/http/server.go:1726 +0x44
2017-01-09_17:32:02.31079 net/http.(*ServeMux).ServeHTTP(0xb23ac0, 0xb00140, 0xc42078a410, 0xc4204c8000)
2017-01-09_17:32:02.31079 	/usr/local/go/src/net/http/server.go:2022 +0x7f
2017-01-09_17:32:02.31080 net/http.serverHandler.ServeHTTP(0xc42008c900, 0xb00140, 0xc42078a410, 0xc4204c8000)
2017-01-09_17:32:02.31081 	/usr/local/go/src/net/http/server.go:2202 +0x7d
2017-01-09_17:32:02.31082 net/http.(*conn).serve(0xc42052e000, 0xb00c80, 0xc4207fdec0)
2017-01-09_17:32:02.31082 	/usr/local/go/src/net/http/server.go:1579 +0x4b7
2017-01-09_17:32:02.31083 created by net/http.(*Server).Serve
2017-01-09_17:32:02.31084 	/usr/local/go/src/net/http/server.go:2293 +0x44d
2017-01-09_17:32:02.31084
2017-01-09_17:32:02.31085 goroutine 705128 [runnable]:
2017-01-09_17:32:02.31086 regexp.(*Regexp).Split(0xc42014c320, 0xc420a6a370, 0x4e, 0x3, 0xc420a6a370, 0x4e, 0xc4201d5ce0)
2017-01-09_17:32:02.31087 	/usr/local/go/src/regexp/regexp.go:1108

<manually restarted by hands>


2017-01-09_17:52:53.11052 time="2017-01-09T17:52:53Z" level=info msg="Starting node_exporter (version=0.13.0-rc.2, branch=master, revision=42d1658d5632fe0bfb281761d5e3620cfc37990b)" source="node_exporter.go:135"
2017-01-09_17:52:53.11054 time="2017-01-09T17:52:53Z" level=info msg="Build context (go=go1.7.3, user=root@bc02144873a8, date=20161119-07:40:56)" source="node_exporter.go:136"
2017-01-09_17:52:53.11108 time="2017-01-09T17:52:53Z" level=info msg="Enabled collectors:" source="node_exporter.go:155"
2017-01-09_17:52:53.11110 time="2017-01-09T17:52:53Z" level=info msg=" - netstat" source="node_exporter.go:157"
2017-01-09_17:52:53.11111 time="2017-01-09T17:52:53Z" level=info msg=" - textfile" source="node_exporter.go:157"
2017-01-09_17:52:53.11112 time="2017-01-09T17:52:53Z" level=info msg=" - meminfo" source="node_exporter.go:157"
2017-01-09_17:52:53.11115 time="2017-01-09T17:52:53Z" level=info msg=" - netdev" source="node_exporter.go:157"
2017-01-09_17:52:53.11117 time="2017-01-09T17:52:53Z" level=info msg=" - sockstat" source="node_exporter.go:157"
2017-01-09_17:52:53.11120 time="2017-01-09T17:52:53Z" level=info msg=" - bonding" source="node_exporter.go:157"
2017-01-09_17:52:53.11122 time="2017-01-09T17:52:53Z" level=info msg=" - uname" source="node_exporter.go:157"
2017-01-09_17:52:53.11124 time="2017-01-09T17:52:53Z" level=info msg=" - vmstat" source="node_exporter.go:157"
2017-01-09_17:52:53.11126 time="2017-01-09T17:52:53Z" level=info msg=" - conntrack" source="node_exporter.go:157"
2017-01-09_17:52:53.11128 time="2017-01-09T17:52:53Z" level=info msg=" - filefd" source="node_exporter.go:157"
2017-01-09_17:52:53.11130 time="2017-01-09T17:52:53Z" level=info msg=" - filesystem" source="node_exporter.go:157"
2017-01-09_17:52:53.11132 time="2017-01-09T17:52:53Z" level=info msg=" - hwmon" source="node_exporter.go:157"
2017-01-09_17:52:53.11134 time="2017-01-09T17:52:53Z" level=info msg=" - mdadm" source="node_exporter.go:157"
2017-01-09_17:52:53.11136 time="2017-01-09T17:52:53Z" level=info msg=" - time" source="node_exporter.go:157"
2017-01-09_17:52:53.11138 time="2017-01-09T17:52:53Z" level=info msg=" - runit" source="node_exporter.go:157"
2017-01-09_17:52:53.11141 time="2017-01-09T17:52:53Z" level=info msg=" - diskstats" source="node_exporter.go:157"
2017-01-09_17:52:53.11143 time="2017-01-09T17:52:53Z" level=info msg=" - entropy" source="node_exporter.go:157"
2017-01-09_17:52:53.11145 time="2017-01-09T17:52:53Z" level=info msg=" - loadavg" source="node_exporter.go:157"
2017-01-09_17:52:53.11147 time="2017-01-09T17:52:53Z" level=info msg=" - stat" source="node_exporter.go:157"
2017-01-09_17:52:53.11149 time="2017-01-09T17:52:53Z" level=info msg=" - megacli" source="node_exporter.go:157"
2017-01-09_17:52:53.11168 time="2017-01-09T17:52:53Z" level=info msg="Listening on :9100" source="node_exporter.go:176"

[lastsky]

-log.level="debug" removed.

[lastsky]

crash repeated today on another server
- same versions for kernel and exporter
- same logs and outputs (debug already attached)
- same resolution (service restart)

[lastsky]

yet another server, all data is the same:

2017-01-13_20:52:22.41462 panic: runtime error: slice bounds out of range
2017-01-13_20:52:22.41464
2017-01-13_20:52:22.41465 goroutine 30969 [running]:
2017-01-13_20:52:22.41465 panic(0x866440, 0xc4200120a0)
2017-01-13_20:52:22.41466 	/usr/local/go/src/runtime/panic.go:500 +0x1a1
2017-01-13_20:52:22.41467 regexp.(*Regexp).Split(0xc4201aad20, 0xc4201be7e4, 0x83, 0xffffffffffffffff, 0xc4201be7e4, 0x83, 0x12)
2017-01-13_20:52:22.41468 	/usr/local/go/src/regexp/regexp.go:1130 +0x2ba
2017-01-13_20:52:22.41468 github.com/prometheus/node_exporter/collector.parseNetDevStats(0xafb840, 0xc420280018, 0xc4201ab220, 0x0, 0x0, 0x74)
2017-01-13_20:52:22.41469 	/go/src/github.com/prometheus/node_exporter/collector/netdev_linux.go:57 +0x23b
2017-01-13_20:52:22.41470 github.com/prometheus/node_exporter/collector.getNetDevStats(0xc4201ab220, 0x0, 0x0, 0x0)
2017-01-13_20:52:22.41470 	/go/src/github.com/prometheus/node_exporter/collector/netdev_linux.go:40 +0xee
2017-01-13_20:52:22.41471 github.com/prometheus/node_exporter/collector.(*netDevCollector).Update(0xc42019dc80, 0xc420644960, 0xb23d60, 0xc4207881a0)
2017-01-13_20:52:22.41472 	/go/src/github.com/prometheus/node_exporter/collector/netdev_common.go:55 +0x52
2017-01-13_20:52:22.41473 main.execute(0x7ffea85ebeab, 0x6, 0xafa680, 0xc42019dc80, 0xc420644960)
2017-01-13_20:52:22.41474 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:86 +0x97
2017-01-13_20:52:22.41475 main.NodeCollector.Collect.func1(0xc420644960, 0xc420246160, 0x7ffea85ebeab, 0x6, 0xafa680, 0xc42019dc80)
2017-01-13_20:52:22.41476 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:65 +0x53
2017-01-13_20:52:22.41477 created by main.NodeCollector.Collect
2017-01-13_20:52:22.41477 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x120
2017-01-13_20:52:22.42239
2017-01-13_20:52:22.42240 runtime stack:
2017-01-13_20:52:22.42241 runtime.throw(0x8fc483, 0x2d)
2017-01-13_20:52:22.42241 	/usr/local/go/src/runtime/panic.go:566 +0x95
2017-01-13_20:52:22.42242 runtime.exitsyscall.func1()
2017-01-13_20:52:22.42242 	/usr/local/go/src/runtime/proc.go:2470 +0x36
2017-01-13_20:52:22.42243 runtime.systemstack(0xc4203c4520)
2017-01-13_20:52:22.42244 	/usr/local/go/src/runtime/asm_amd64.s:298 +0x79
2017-01-13_20:52:22.42244 runtime.mstart()
2017-01-13_20:52:22.42245 	/usr/local/go/src/runtime/proc.go:1079
2017-01-13_20:52:22.42245
2017-01-13_20:52:22.42245 goroutine 30988 [running]:
2017-01-13_20:52:22.42246 runtime.systemstack_switch()
2017-01-13_20:52:22.42246 	/usr/local/go/src/runtime/asm_amd64.s:252 fp=0xc420058750 sp=0xc420058748
2017-01-13_20:52:22.42247 runtime.exitsyscall(0x648b35)
2017-01-13_20:52:22.42248 	/usr/local/go/src/runtime/proc.go:2471 +0x275 fp=0xc420058780 sp=0xc420058750
2017-01-13_20:52:22.42248 syscall.Syscall(0x0, 0xf, 0xc420550000, 0x1000, 0x5ab, 0x1000, 0x0)
2017-01-13_20:52:22.42249 	/usr/local/go/src/syscall/asm_linux_amd64.s:39 +0x64 fp=0xc420058788 sp=0xc420058780
2017-01-13_20:52:22.42249 syscall.read(0xf, 0xc420550000, 0x1000, 0x1000, 0x1, 0x10100c4200588a8, 0x0)
2017-01-13_20:52:22.42250 	/usr/local/go/src/syscall/zsyscall_linux_amd64.go:783 +0x55 fp=0xc4200587e0 sp=0xc420058788
2017-01-13_20:52:22.42250 syscall.Read(0xf, 0xc420550000, 0x1000, 0x1000, 0x0, 0xf, 0x32)
2017-01-13_20:52:22.42251 	/usr/local/go/src/syscall/syscall_unix.go:161 +0x49 fp=0xc420058828 sp=0xc4200587e0
2017-01-13_20:52:22.42252 os.(*File).read(0xc420374000, 0xc420550000, 0x1000, 0x1000, 0x1, 0xc420550000, 0xc41ffcd9f6)
2017-01-13_20:52:22.42253 	/usr/local/go/src/os/file_unix.go:228 +0x4f fp=0xc420058870 sp=0xc420058828
2017-01-13_20:52:22.42254 os.(*File).Read(0xc420374000, 0xc420550000, 0x1000, 0x1000, 0x1000, 0x1000, 0xc4203b4400)
2017-01-13_20:52:22.42255 	/usr/local/go/src/os/file.go:101 +0x59 fp=0xc4200588d8 sp=0xc420058870
2017-01-13_20:52:22.42256 bufio.(*Scanner).Scan(0xc420058ab8, 0x0)
2017-01-13_20:52:22.42257 	/usr/local/go/src/bufio/scan.go:208 +0x50f fp=0xc4200589e8 sp=0xc4200588d8
2017-01-13_20:52:22.42258 github.com/prometheus/node_exporter/collector.parseNetStats(0xafb840, 0xc420374000, 0xc42047a130, 0xe, 0x0, 0x5, 0x8e7689)
2017-01-13_20:52:22.42260 	/go/src/github.com/prometheus/node_exporter/collector/netstat_linux.go:95 +0xd1 fp=0xc420058b48 sp=0xc4200589e8
2017-01-13_20:52:22.42261 github.com/prometheus/node_exporter/collector.getNetStats(0xc42047a130, 0xe, 0x0, 0x0, 0x0)
2017-01-13_20:52:22.42262 	/go/src/github.com/prometheus/node_exporter/collector/netstat_linux.go:86 +0xdf fp=0xc420058b98 sp=0xc420058b48
2017-01-13_20:52:22.42263 github.com/prometheus/node_exporter/collector.(*netStatCollector).Update(0xb405b0, 0xc420644960, 0xb23d60, 0x1)
2017-01-13_20:52:22.42264 	/go/src/github.com/prometheus/node_exporter/collector/netstat_linux.go:50 +0xc7 fp=0xc420058e38 sp=0xc420058b98
2017-01-13_20:52:22.42265 main.execute(0x7ffea85ebeb2, 0x7, 0xafa6c0, 0xb405b0, 0xc420644960)
2017-01-13_20:52:22.42266 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:86 +0x97 fp=0xc420058f48 sp=0xc420058e38
2017-01-13_20:52:22.42268 main.NodeCollector.Collect.func1(0xc420644960, 0xc420246160, 0x7ffea85ebeb2, 0x7, 0xafa6c0, 0xb405b0)
2017-01-13_20:52:22.42269 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:65 +0x53 fp=0xc420058f80 sp=0xc420058f48
2017-01-13_20:52:22.42270 runtime.goexit()
2017-01-13_20:52:22.42270 	/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420058f88 sp=0xc420058f80
2017-01-13_20:52:22.42271 created by main.NodeCollector.Collect
2017-01-13_20:52:22.42273 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x120
2017-01-13_20:52:22.42274
2017-01-13_20:52:22.42274 goroutine 1 [IO wait]:
2017-01-13_20:52:22.42275 net.runtime_pollWait(0x7fa7fe12e2a8, 0x72, 0x0)
2017-01-13_20:52:22.42275 	/usr/local/go/src/runtime/netpoll.go:160 +0x59
2017-01-13_20:52:22.42276 net.(*pollDesc).wait(0xc4201e1100, 0x72, 0xc42006ba38, 0xc420012028)
2017-01-13_20:52:22.42276 	/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
2017-01-13_20:52:22.42277 net.(*pollDesc).waitRead(0xc4201e1100, 0xafd1c0, 0xc420012028)
2017-01-13_20:52:22.42278 	/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
2017-01-13_20:52:22.42278 net.(*netFD).accept(0xc4201e10a0, 0x0, 0xafb900, 0xc420b7a000)
2017-01-13_20:52:22.42279 	/usr/local/go/src/net/fd_unix.go:419 +0x238
2017-01-13_20:52:22.42279 net.(*TCPListener).accept(0xc42019e0f8, 0x29e8d60800, 0x0, 0x0)
2017-01-13_20:52:22.42280 	/usr/local/go/src/net/tcpsock_posix.go:132 +0x2e
2017-01-13_20:52:22.42281 net.(*TCPListener).AcceptTCP(0xc42019e0f8, 0xc42006bb60, 0xc42006bb68, 0xc42006bb58)
2017-01-13_20:52:22.42281 	/usr/local/go/src/net/tcpsock.go:209 +0x49
2017-01-13_20:52:22.42282 net/http.tcpKeepAliveListener.Accept(0xc42019e0f8, 0x92b9a8, 0xc4203faa00, 0xb00d40, 0xc4201df680)
2017-01-13_20:52:22.42283 	/usr/local/go/src/net/http/server.go:2608 +0x2f
2017-01-13_20:52:22.42285 net/http.(*Server).Serve(0xc4201a2600, 0xb00500, 0xc42019e0f8, 0x0, 0x0)
2017-01-13_20:52:22.42286 	/usr/local/go/src/net/http/server.go:2273 +0x1ce
2017-01-13_20:52:22.42287 net/http.(*Server).ListenAndServe(0xc4201a2600, 0xc4201a2600, 0x2)
2017-01-13_20:52:22.42288 	/usr/local/go/src/net/http/server.go:2219 +0xb4
2017-01-13_20:52:22.42288 net/http.ListenAndServe(0x8e3d54, 0x5, 0x0, 0x0, 0xc4201e2db0, 0xc4201989d8)
2017-01-13_20:52:22.42289 	/usr/local/go/src/net/http/server.go:2351 +0xa0
2017-01-13_20:52:22.42290 main.main()
2017-01-13_20:52:22.42290 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:177 +0xa5d
2017-01-13_20:52:22.42291
2017-01-13_20:52:22.42292 goroutine 17 [syscall, 121 minutes, locked to thread]:
2017-01-13_20:52:22.42293 runtime.goexit()
2017-01-13_20:52:22.42293 	/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1
2017-01-13_20:52:22.42294
2017-01-13_20:52:22.42294 goroutine 30983 [runnable]:
2017-01-13_20:52:22.42295 runtime.SetFinalizer(0x85d9c0, 0xc420a00020, 0x84ef00, 0x92bbc0)
2017-01-13_20:52:22.42296 	/usr/local/go/src/runtime/mfinal.go:283
2017-01-13_20:52:22.42297 os.NewFile(0x11, 0xc420344010, 0xc, 0xc400000000)
2017-01-13_20:52:22.42298 	/usr/local/go/src/os/file_unix.go:57 +0xee
2017-01-13_20:52:22.42298 os.OpenFile(0xc420344010, 0xc, 0x0, 0x0, 0x5, 0x8e5b02, 0x6)
2017-01-13_20:52:22.42299 	/usr/local/go/src/os/file_unix.go:123 +0x18f
2017-01-13_20:52:22.42300 os.Open(0xc420344010, 0xc, 0xc420344010, 0xc, 0x53a862)
2017-01-13_20:52:22.42301 	/usr/local/go/src/os/file.go:250 +0x46
2017-01-13_20:52:22.42301 github.com/prometheus/node_exporter/collector.(*vmStatCollector).Update(0xb405b0, 0xc420644960, 0x0, 0x0)
2017-01-13_20:52:22.42302 	/go/src/github.com/prometheus/node_exporter/collector/vmstat_linux.go:45 +0x92
2017-01-13_20:52:22.42303 main.execute(0x7ffea85ebedc, 0x6, 0xafa980, 0xb405b0, 0xc420644960)
2017-01-13_20:52:22.42304 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:86 +0x97
2017-01-13_20:52:22.42304 main.NodeCollector.Collect.func1(0xc420644960, 0xc420246160, 0x7ffea85ebedc, 0x6, 0xafa980, 0xb405b0)
2017-01-13_20:52:22.42305 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:65 +0x53
2017-01-13_20:52:22.42306 created by main.NodeCollector.Collect
2017-01-13_20:52:22.42307 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x120
2017-01-13_20:52:22.42307
2017-01-13_20:52:22.42308 goroutine 30896 [semacquire]:
2017-01-13_20:52:22.42308 sync.runtime_Semacquire(0xc4202460ec)
2017-01-13_20:52:22.42309 	/usr/local/go/src/runtime/sema.go:47 +0x30
2017-01-13_20:52:22.42309 sync.(*WaitGroup).Wait(0xc4202460e0)
2017-01-13_20:52:22.42310 	/usr/local/go/src/sync/waitgroup.go:131 +0x97
2017-01-13_20:52:22.42311 github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB.func1(0xc4202460e0, 0xc420644960)
2017-01-13_20:52:22.42312 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:409 +0x2d
2017-01-13_20:52:22.42313 created by github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB
2017-01-13_20:52:22.42313 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:411 +0x1db
2017-01-13_20:52:22.42314
2017-01-13_20:52:22.42314 goroutine 30973 [runnable]:
2017-01-13_20:52:22.42315 syscall.Read(0x6, 0xc420590000, 0x200, 0x200, 0x41209e, 0x7fa7fe1c9000, 0x19)
2017-01-13_20:52:22.42316 	/usr/local/go/src/syscall/syscall_unix.go:160
2017-01-13_20:52:22.42316 os.(*File).read(0xc420276140, 0xc420590000, 0x200, 0x200, 0xc420590000, 0x0, 0xc420418400)
2017-01-13_20:52:22.42317 	/usr/local/go/src/os/file_unix.go:228 +0x4f
2017-01-13_20:52:22.42317 os.(*File).Read(0xc420276140, 0xc420590000, 0x200, 0x200, 0xc4206b0c28, 0xc4206b0c38, 0x456c50)
2017-01-13_20:52:22.42318 	/usr/local/go/src/os/file.go:101 +0x59
2017-01-13_20:52:22.42319 bytes.(*Buffer).ReadFrom(0xc4206b0c58, 0xafb840, 0xc420276140, 0xc420590000, 0x0, 0x200)
2017-01-13_20:52:22.42320 	/usr/local/go/src/bytes/buffer.go:176 +0x155
2017-01-13_20:52:22.42320 io/ioutil.readAll(0xafb840, 0xc420276140, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
2017-01-13_20:52:22.42321 	/usr/local/go/src/io/ioutil/ioutil.go:33 +0x147
2017-01-13_20:52:22.42322 io/ioutil.ReadFile(0xc4208ccb90, 0xd, 0x0, 0x0, 0x0, 0x0, 0x0)
2017-01-13_20:52:22.42323 	/usr/local/go/src/io/ioutil/ioutil.go:70 +0x12e
2017-01-13_20:52:22.42324 github.com/prometheus/node_exporter/collector.getLoad(0x0, 0x0, 0x0, 0x0, 0x99bca0)
2017-01-13_20:52:22.42325 	/go/src/github.com/prometheus/node_exporter/collector/loadavg_linux.go:27 +0x52
2017-01-13_20:52:22.42326 github.com/prometheus/node_exporter/collector.(*loadavgCollector).Update(0xc42019dc60, 0xc420644960, 0xb23d60, 0x2)
2017-01-13_20:52:22.42326 	/go/src/github.com/prometheus/node_exporter/collector/loadavg.go:58 +0x34
2017-01-13_20:52:22.42327 main.execute(0x7ffea85ebe95, 0x7, 0xafa500, 0xc42019dc60, 0xc420644960)
2017-01-13_20:52:22.42328 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:86 +0x97
2017-01-13_20:52:22.42329 main.NodeCollector.Collect.func1(0xc420644960, 0xc420246160, 0x7ffea85ebe95, 0x7, 0xafa500, 0xc42019dc60)
2017-01-13_20:52:22.42330 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:65 +0x53
2017-01-13_20:52:22.42330 created by main.NodeCollector.Collect
2017-01-13_20:52:22.42331 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x120
2017-01-13_20:52:22.42335
2017-01-13_20:52:22.42335 goroutine 30962 [runnable]:
2017-01-13_20:52:22.42336 bufio.(*Scanner).Scan(0xc4200678f0, 0x5)
2017-01-13_20:52:22.42336 	/usr/local/go/src/bufio/scan.go:128
2017-01-13_20:52:22.42337 github.com/prometheus/node_exporter/vendor/github.com/prometheus/procfs.FS.NewStat(0x8e3d40, 0x5, 0x0, 0x0, 0x0)
2017-01-13_20:52:22.42338 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/procfs/stat.go:36 +0x162
2017-01-13_20:52:22.42339 github.com/prometheus/node_exporter/vendor/github.com/prometheus/procfs.ProcStat.StartTime(0x3d8b, 0xc4207b0020, 0xd, 0xc4207b0030, 0x1, 0x619, 0x619, 0x619, 0x0, 0xffffffffffffffff, ...)
2017-01-13_20:52:22.42340 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/procfs/proc_stat.go:165 +0x3f
2017-01-13_20:52:22.42341 github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*processCollector).processCollect(0xc420092680, 0xc420644960)
2017-01-13_20:52:22.42342 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/process_collector.go:127 +0x581
2017-01-13_20:52:22.42343 github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*processCollector).(github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.processCollect)-fm(0xc420644960)
2017-01-13_20:52:22.42344 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/process_collector.go:85 +0x34
2017-01-13_20:52:22.42345 github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*processCollector).Collect(0xc420092680, 0xc420644960)
2017-01-13_20:52:22.42346 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/process_collector.go:103 +0x34
2017-01-13_20:52:22.42346 github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB.func2(0xc4202460e0, 0xc420644960, 0xafe040, 0xc420092680)
2017-01-13_20:52:22.42347 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:415 +0x63
2017-01-13_20:52:22.42348 created by github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB
2017-01-13_20:52:22.42348 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:416 +0x283
2017-01-13_20:52:22.42349
2017-01-13_20:52:22.42350 goroutine 30965 [semacquire]:
2017-01-13_20:52:22.42350 sync.runtime_Semacquire(0xc42024616c)
2017-01-13_20:52:22.42351 	/usr/local/go/src/runtime/sema.go:47 +0x30
2017-01-13_20:52:22.42351 sync.(*WaitGroup).Wait(0xc420246160)
2017-01-13_20:52:22.42352 	/usr/local/go/src/sync/waitgroup.go:131 +0x97
2017-01-13_20:52:22.42353 main.NodeCollector.Collect(0xc420196bd0, 0xc420644960)
2017-01-13_20:52:22.42353 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:69 +0x146
2017-01-13_20:52:22.42354 github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB.func2(0xc4202460e0, 0xc420644960, 0xaff080, 0xc420196bd0)
2017-01-13_20:52:22.42355 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:415 +0x63
2017-01-13_20:52:22.42355 created by github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB
2017-01-13_20:52:22.42356 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:416 +0x283
2017-01-13_20:52:22.42357
2017-01-13_20:52:22.42357 goroutine 30976 [syscall]:
2017-01-13_20:52:22.42358 syscall.Syscall(0x0, 0xa, 0xc42013e000, 0x1000, 0x2b, 0x7fa7fe1d05c0, 0x4552a0)
2017-01-13_20:52:22.42359 	/usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5
2017-01-13_20:52:22.42359 syscall.read(0xa, 0xc42013e000, 0x1000, 0x1000, 0x1, 0x101000000000010, 0x0)
2017-01-13_20:52:22.42360 	/usr/local/go/src/syscall/zsyscall_linux_amd64.go:783 +0x55
2017-01-13_20:52:22.42361 syscall.Read(0xa, 0xc42013e000, 0x1000, 0x1000, 0x0, 0x6, 0x0)
2017-01-13_20:52:22.42361 	/usr/local/go/src/syscall/syscall_unix.go:161 +0x49
2017-01-13_20:52:22.42362 os.(*File).read(0xc42019e008, 0xc42013e000, 0x1000, 0x1000, 0x1, 0xc42013e000, 0xc41ffbf3fb)
2017-01-13_20:52:22.42363 	/usr/local/go/src/os/file_unix.go:228 +0x4f
2017-01-13_20:52:22.42364 os.(*File).Read(0xc42019e008, 0xc42013e000, 0x1000, 0x1000, 0x1000, 0x1000, 0xc4204a5000)
2017-01-13_20:52:22.42365 	/usr/local/go/src/os/file.go:101 +0x59
2017-01-13_20:52:22.42365 bufio.(*Scanner).Scan(0xc420840c68, 0x0)
2017-01-13_20:52:22.42366 	/usr/local/go/src/bufio/scan.go:208 +0x50f
2017-01-13_20:52:22.42367 github.com/prometheus/node_exporter/collector.parseMegaCliAdapter(0xafb840, 0xc42019e008, 0xc42019e008, 0xafb840, 0xc42019e008)
2017-01-13_20:52:22.42368 	/go/src/github.com/prometheus/node_exporter/collector/megacli.go:135 +0xf3
2017-01-13_20:52:22.42369 github.com/prometheus/node_exporter/collector.(*megaCliCollector).updateAdapter(0xc420197f80, 0xc420626638, 0x5050d2)
2017-01-13_20:52:22.42369 	/go/src/github.com/prometheus/node_exporter/collector/megacli.go:171 +0x126
2017-01-13_20:52:22.42370 github.com/prometheus/node_exporter/collector.(*megaCliCollector).Update(0xc420197f80, 0xc420644960, 0xb23d60, 0x0)
2017-01-13_20:52:22.42371 	/go/src/github.com/prometheus/node_exporter/collector/megacli.go:74 +0x2f
2017-01-13_20:52:22.42372 main.execute(0x7ffea85ebeeb, 0x7, 0xafa5c0, 0xc420197f80, 0xc420644960)
2017-01-13_20:52:22.42372 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:86 +0x97
2017-01-13_20:52:22.42373 main.NodeCollector.Collect.func1(0xc420644960, 0xc420246160, 0x7ffea85ebeeb, 0x7, 0xafa5c0, 0xc420197f80)
2017-01-13_20:52:22.42374 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:65 +0x53
2017-01-13_20:52:22.42374 created by main.NodeCollector.Collect
2017-01-13_20:52:22.42375 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x120
2017-01-13_20:52:22.42376
2017-01-13_20:52:22.42376 goroutine 30969 [running]:
2017-01-13_20:52:22.42377 	goroutine running on other thread; stack unavailable
2017-01-13_20:52:22.42377 created by main.NodeCollector.Collect
2017-01-13_20:52:22.42378 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x120
2017-01-13_20:52:22.42379
2017-01-13_20:52:22.42379 goroutine 30985 [runnable]:
2017-01-13_20:52:22.42379 fmt.(*pp).doPrintf(0xc4207a4fc0, 0x8e7629, 0x8, 0xc4207c8c18, 0x1, 0x1)
2017-01-13_20:52:22.42380 	/usr/local/go/src/fmt/print.go:933
2017-01-13_20:52:22.42381 fmt.Sprintf(0x8e7629, 0x8, 0xc4207c8c18, 0x1, 0x1, 0x3, 0x0)
2017-01-13_20:52:22.42381 	/usr/local/go/src/fmt/print.go:196 +0x6a
2017-01-13_20:52:22.42382 github.com/prometheus/node_exporter/collector.readBondingStats(0xc4207e9300, 0xe, 0xc4207e9300, 0xe, 0x0)
2017-01-13_20:52:22.42383 	/go/src/github.com/prometheus/node_exporter/collector/bonding_linux.go:87 +0x3c3
2017-01-13_20:52:22.42383 github.com/prometheus/node_exporter/collector.(*bondingCollector).Update(0xc420199860, 0xc420644960, 0xb23d60, 0x80000c420888ea0)
2017-01-13_20:52:22.42384 	/go/src/github.com/prometheus/node_exporter/collector/bonding_linux.go:61 +0x6d
2017-01-13_20:52:22.42385 main.execute(0x7ffea85ebee3, 0x7, 0xafa240, 0xc420199860, 0xc420644960)
2017-01-13_20:52:22.42386 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:86 +0x97
2017-01-13_20:52:22.42386 main.NodeCollector.Collect.func1(0xc420644960, 0xc420246160, 0x7ffea85ebee3, 0x7, 0xafa240, 0xc420199860)
2017-01-13_20:52:22.42387 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:65 +0x53
2017-01-13_20:52:22.42388 created by main.NodeCollector.Collect
2017-01-13_20:52:22.42388 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x120
2017-01-13_20:52:22.42389
2017-01-13_20:52:22.42389 goroutine 30982 [runnable]:
2017-01-13_20:52:22.42390 github.com/prometheus/node_exporter/vendor/github.com/prometheus/common/log.logger.sourced(0xc420041300, 0xc420915e38)
2017-01-13_20:52:22.42391 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/common/log/log.go:217 +0x115
2017-01-13_20:52:22.42391 github.com/prometheus/node_exporter/vendor/github.com/prometheus/common/log.Debugf(0x8f875c, 0x25, 0xc42076cee8, 0x2, 0x2)
2017-01-13_20:52:22.42392 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/common/log/log.go:243 +0x2d
2017-01-13_20:52:22.42393 main.execute(0x7ffea85ebec8, 0x8, 0xafa8c0, 0xc420199770, 0xc420644960)
2017-01-13_20:52:22.42394 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:94 +0x4b1
2017-01-13_20:52:22.42394 main.NodeCollector.Collect.func1(0xc420644960, 0xc420246160, 0x7ffea85ebec8, 0x8, 0xafa8c0, 0xc420199770)
2017-01-13_20:52:22.42395 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:65 +0x53
2017-01-13_20:52:22.42396 created by main.NodeCollector.Collect
2017-01-13_20:52:22.42397 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x120
2017-01-13_20:52:22.42397
2017-01-13_20:52:22.42398 goroutine 30979 [runnable]:
2017-01-13_20:52:22.42399 main.NodeCollector.Collect.func1(0xc420644960, 0xc420246160, 0x7ffea85ebe8f, 0x5, 0xafa400, 0xb405b0)
2017-01-13_20:52:22.42400 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:64
2017-01-13_20:52:22.42401 created by main.NodeCollector.Collect
2017-01-13_20:52:22.42401 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x120
2017-01-13_20:52:22.42402
2017-01-13_20:52:22.42403 goroutine 30894 [runnable]:
2017-01-13_20:52:22.42404 github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).writePB(0xc42008e6c0, 0xafc500, 0xc420b7a460, 0x0, 0x0)
2017-01-13_20:52:22.42405 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:432 +0x3ab
2017-01-13_20:52:22.42406 github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*registry).ServeHTTP(0xc42008e6c0, 0xaffc00, 0xc42029c008, 0xc4200f20f0)
2017-01-13_20:52:22.42407 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:375 +0x13b
2017-01-13_20:52:22.42408 net/http.(Handler).ServeHTTP-fm(0xaffc00, 0xc42029c008, 0xc4200f20f0)
2017-01-13_20:52:22.42409 	/usr/local/go/src/net/http/h2_bundle.go:4084 +0x4d
2017-01-13_20:52:22.42410 github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0xb00140, 0xc420642000, 0xc4200f20f0)
2017-01-13_20:52:22.42410 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/http.go:158 +0x2ab
2017-01-13_20:52:22.42411 net/http.HandlerFunc.ServeHTTP(0xc420194a50, 0xb00140, 0xc420642000, 0xc4200f20f0)
2017-01-13_20:52:22.42412 	/usr/local/go/src/net/http/server.go:1726 +0x44
2017-01-13_20:52:22.42412 net/http.(*ServeMux).ServeHTTP(0xb23ac0, 0xb00140, 0xc420642000, 0xc4200f20f0)
2017-01-13_20:52:22.42413 	/usr/local/go/src/net/http/server.go:2022 +0x7f
2017-01-13_20:52:22.42414 net/http.serverHandler.ServeHTTP(0xc4201a2600, 0xb00140, 0xc420642000, 0xc4200f20f0)
2017-01-13_20:52:22.42415 	/usr/local/go/src/net/http/server.go:2202 +0x7d
2017-01-13_20:52:22.42415 net/http.(*conn).serve(0xc4203faa00, 0xb00c80, 0xc420284500)
2017-01-13_20:52:22.42416 	/usr/local/go/src/net/http/server.go:1579 +0x4b7
2017-01-13_20:52:22.42416 created by net/http.(*Server).Serve
2017-01-13_20:52:22.42417 	/usr/local/go/src/net/http/server.go:2293 +0x44d
2017-01-13_20:52:22.42418
2017-01-13_20:52:22.42419 goroutine 30970 [runnable]:
2017-01-13_20:52:22.42420 strings.Fields(0xc4206d8000, 0x1740, 0x1740, 0x2000, 0xc4206d8000)
2017-01-13_20:52:22.42421 	/usr/local/go/src/strings/strings.go:300
2017-01-13_20:52:22.42421 github.com/prometheus/node_exporter/collector.(*statCollector).Update(0xc42019b140, 0xc420644960, 0x0, 0x0)
2017-01-13_20:52:22.42422 	/go/src/github.com/prometheus/node_exporter/collector/stat_linux.go:97 +0x1bc
2017-01-13_20:52:22.42423 main.execute(0x7ffea85ebec3, 0x4, 0xafa7c0, 0xc42019b140, 0xc420644960)
2017-01-13_20:52:22.42424 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:86 +0x97
2017-01-13_20:52:22.42424 main.NodeCollector.Collect.func1(0xc420644960, 0xc420246160, 0x7ffea85ebec3, 0x4, 0xafa7c0, 0xc42019b140)
2017-01-13_20:52:22.42425 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:65 +0x53
2017-01-13_20:52:22.42426 created by main.NodeCollector.Collect
2017-01-13_20:52:22.42426 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x120
2017-01-13_20:52:22.42427
2017-01-13_20:52:22.42428 goroutine 30980 [runnable]:
2017-01-13_20:52:22.42428 os.Open(0xc42079e780, 0x24, 0x6, 0xc42079e780, 0x24)
2017-01-13_20:52:22.42429 	/usr/local/go/src/os/file.go:249
2017-01-13_20:52:22.42429 github.com/prometheus/node_exporter/vendor/github.com/soundcloud/go-runit/runit.(*service).status(0xc42068c4e0, 0x0, 0x0, 0x0, 0x0, 0x0)
2017-01-13_20:52:22.42430 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/soundcloud/go-runit/runit/runit.go:96 +0x92
2017-01-13_20:52:22.42431 github.com/prometheus/node_exporter/vendor/github.com/soundcloud/go-runit/runit.(*service).Status(0xc42068c4e0, 0xc, 0xc420539480, 0xb)
2017-01-13_20:52:22.42432 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/soundcloud/go-runit/runit/runit.go:112 +0x40
2017-01-13_20:52:22.42433 github.com/prometheus/node_exporter/collector.(*runitCollector).Update(0xc4201da6c0, 0xc420644960, 0xb23d60, 0x80000000041323d)
2017-01-13_20:52:22.42434 	/go/src/github.com/prometheus/node_exporter/collector/runit.go:90 +0xae
2017-01-13_20:52:22.42434 main.execute(0x7ffea85ebef3, 0x5, 0xafa740, 0xc4201da6c0, 0xc420644960)
2017-01-13_20:52:22.42435 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:86 +0x97
2017-01-13_20:52:22.42436 main.NodeCollector.Collect.func1(0xc420644960, 0xc420246160, 0x7ffea85ebef3, 0x5, 0xafa740, 0xc4201da6c0)
2017-01-13_20:52:22.42437 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:65 +0x53
2017-01-13_20:52:22.42437 created by main.NodeCollector.Collect
2017-01-13_20:52:22.42438 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x120
2017-01-13_20:52:22.42439
2017-01-13_20:52:22.42439 goroutine 30895 [chan send]:
2017-01-13_20:52:22.42440 github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.computeApproximateRequestSize(0xc4200f20f0, 0xc420644900, 0xe0)
2017-01-13_20:52:22.42441 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/http.go:187 +0x151
2017-01-13_20:52:22.42441 created by github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1
2017-01-13_20:52:22.42442 	/go/src/github.com/prometheus/node_exporter/vendor/github.com/prometheus/client_golang/prometheus/http.go:146 +0x16a
2017-01-13_20:52:22.42442
2017-01-13_20:52:22.42443 goroutine 30986 [runnable]:
2017-01-13_20:52:22.42443 syscall.Fstat(0xc, 0xc4201d8038, 0x456c50, 0xc4208d6a30)
2017-01-13_20:52:22.42444 	/usr/local/go/src/syscall/zsyscall_linux_amd64.go:1220
2017-01-13_20:52:22.42445 os.(*File).Stat(0xc4202c0000, 0x92bbb0, 0xc4202c0000, 0x0, 0x0)
2017-01-13_20:52:22.42445 	/usr/local/go/src/os/file_unix.go:157 +0x87
2017-01-13_20:52:22.42446 io/ioutil.ReadFile(0xc420272010, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0)
2017-01-13_20:52:22.42446 	/usr/local/go/src/io/ioutil/ioutil.go:59 +0xf3
2017-01-13_20:52:22.42447 github.com/prometheus/node_exporter/collector.parseMdstat(0xc420272010, 0xc, 0xb02a00, 0xc4203b2000, 0x0, 0x0, 0x25)
2017-01-13_20:52:22.42448 	/go/src/github.com/prometheus/node_exporter/collector/mdadm_linux.go:136 +0x5b
2017-01-13_20:52:22.42448 github.com/prometheus/node_exporter/collector.(*mdadmCollector).Update(0xb405b0, 0xc420644960, 0xb23d60, 0x1)
2017-01-13_20:52:22.42449 	/go/src/github.com/prometheus/node_exporter/collector/mdadm_linux.go:294 +0xcb
2017-01-13_20:52:22.42449 main.execute(0x7ffea85ebe9d, 0x5, 0xafa580, 0xb405b0, 0xc420644960)
2017-01-13_20:52:22.42450 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:86 +0x97
2017-01-13_20:52:22.42451 main.NodeCollector.Collect.func1(0xc420644960, 0xc420246160, 0x7ffea85ebe9d, 0x5, 0xafa580, 0xb405b0)
2017-01-13_20:52:22.42452 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:65 +0x53
2017-01-13_20:52:22.42452 created by main.NodeCollector.Collect
2017-01-13_20:52:22.42453 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x120
2017-01-13_20:52:22.42453
2017-01-13_20:52:22.42454 goroutine 30987 [runnable]:
2017-01-13_20:52:22.42455 main.NodeCollector.Collect.func1(0xc420644960, 0xc420246160, 0x7ffea85ebea3, 0x7, 0xafa600, 0xb405b0)
2017-01-13_20:52:22.42455 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:64
2017-01-13_20:52:22.42456 created by main.NodeCollector.Collect
2017-01-13_20:52:22.42456 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x120
2017-01-13_20:52:22.42457
2017-01-13_20:52:22.42457 goroutine 30981 [runnable]:
2017-01-13_20:52:22.42458 bytes.(*Buffer).ReadFrom(0xc42083cca8, 0xafb840, 0xc4208c0000, 0xc420248000, 0x0, 0x200)
2017-01-13_20:52:22.42458 	/usr/local/go/src/bytes/buffer.go:157
2017-01-13_20:52:22.42459 io/ioutil.readAll(0xafb840, 0xc4208c0000, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
2017-01-13_20:52:22.42460 	/usr/local/go/src/io/ioutil/ioutil.go:33 +0x147
2017-01-13_20:52:22.42461 io/ioutil.ReadFile(0xc420b6c060, 0x2a, 0x0, 0x0, 0x0, 0x0, 0x0)
2017-01-13_20:52:22.42462 	/usr/local/go/src/io/ioutil/ioutil.go:70 +0x12e
2017-01-13_20:52:22.42462 github.com/prometheus/node_exporter/collector.readUintFromFile(0xc420b6c060, 0x2a, 0xc420b6c060, 0x2a, 0xc420913e18)
2017-01-13_20:52:22.42463 	/go/src/github.com/prometheus/node_exporter/collector/helper.go:35 +0x39
2017-01-13_20:52:22.42464 github.com/prometheus/node_exporter/collector.(*conntrackCollector).Update(0xc420198c50, 0xc420644960, 0xb23d60, 0x2)
2017-01-13_20:52:22.42464 	/go/src/github.com/prometheus/node_exporter/collector/conntrack_linux.go:49 +0x52
2017-01-13_20:52:22.42465 main.execute(0x7ffea85ebe61, 0x9, 0xafa280, 0xc420198c50, 0xc420644960)
2017-01-13_20:52:22.42466 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:86 +0x97
2017-01-13_20:52:22.42466 main.NodeCollector.Collect.func1(0xc420644960, 0xc420246160, 0x7ffea85ebe61, 0x9, 0xafa280, 0xc420198c50)
2017-01-13_20:52:22.42467 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:65 +0x53
2017-01-13_20:52:22.42467 created by main.NodeCollector.Collect
2017-01-13_20:52:22.42468 	/go/src/github.com/prometheus/node_exporter/node_exporter.go:67 +0x120
2017-01-13_20:52:22.44389 time="2017-01-13T20:52:22Z" level=info msg="Starting node_exporter (version=0.13.0-rc.2, branch=master, revision=42d1658d5632fe0bfb281761d5e3620cfc37990b)" source="node_exporter.go:135"
2017-01-13_20:52:22.44390 time="2017-01-13T20:52:22Z" level=info msg="Build context (go=go1.7.3, user=root@bc02144873a8, date=20161119-07:40:56)" source="node_exporter.go:136"

[SuperQ]

I will be cutting a new release soon, there are a few changes including migrating all of these metrics to const metrics that should help.

[SuperQ]

The release is out. Please let me know if it helps.

[discordianfish]

Going to assume this is fixed. If not, please reopen!

[stef97]

Hi,
I've added NVME devices to my OSDs and this seems to break node_exporter
" ...msg="ERROR: diskstats collector failed "

Any suggestions / ideas will be greatly appreciated

/proc/diskstats is as follows
cat /proc/diskstats
259 0 nvme0n1 55381 0 4027200 2463 121 2631 22016 61 0 17535 19241 0 0 0 0
259 1 nvme0n1p1 608 0 46368 21 0 0 0 0 0 21 21 0 0 0 0
259 2 nvme0n1p2 661 0 48672 22 0 0 0 0 0 24 26 0 0 0 0
259 3 nvme0n1p3 560 0 42176 19 0 0 0 0 0 13 14 0 0 0 0
259 4 nvme0n1p4 631 0 48432 23 0 0 0 0 0 26 27 0 0 0 0
259 6 nvme0n1p5 631 0 48432 24 0 0 0 0 0 22 24 0 0 0 0
259 7 nvme0n1p6 560 0 42176 20 0 0 0 0 0 16 17 0 0 0 0
259 10 nvme0n1p7 631 0 48432 22 0 0 0 0 0 18 19 0 0 0 0
259 11 nvme0n1p8 631 0 48432 23 0 0 0 0 0 22 25 0 0 0 0
259 17 nvme0n1p9 560 0 42176 20 0 0 0 0 0 17 24 0 0 0 0
259 20 nvme0n1p10 631 0 48432 22 0 0 0 0 0 26 26 0 0 0 0
259 21 nvme0n1p11 631 0 48432 22 0 0 0 0 0 14 16 0 0 0 0
259 24 nvme0n1p12 631 0 48432 23 0 0 0 0 0 17 18 0 0 0 0
259 25 nvme0n1p13 583 0 44240 22 0 0 0 0 0 23 26 0 0 0 0
259 28 nvme0n1p14 631 0 48432 23 0 0 0 0 0 20 24 0 0 0 0
259 29 nvme0n1p15 535 0 40048 20 0 0 0 0 0 22 26 0 0 0 0
259 31 nvme0n1p16 606 0 46304 22 0 0 0 0 0 26 31 0 0 0 0
8 0 sda 10823138 0 94504280 1571212 26005951 19406 208223048 2236550 0 365787 3808349 0 0 0 0
8 16 sdb 17940 0 923160 1227 40 2520 20480 10 0 720 1158 0 0 0 0
8 32 sdc 17940 0 923160 1047 40 2520 20480 10 0 705 1189 0 0 0 0
8 48 sdd 17940 0 923160 1316 40 2520 20480 9 0 730 1205 0 0 0 0
8 64 sde 17940 0 923160 1292 40 2520 20480 10 0 787 1385 0 0 0 0
8 80 sdf 40684 282 1113465 140696 74404 7192 3272067 43079689 0 3633695 43197380 0 0 0 0
8 81 sdf1 322 0 18927 2832 24 0 4136 21254 0 17769 24084 0 0 0 0
8 82 sdf2 14901 267 687400 76779 27742 4760 975175 27664431 0 2085479 27734760 0 0 0 0
8 83 sdf3 259 0 11512 1156 0 0 0 0 0 773 1159 0 0 0 0
8 84 sdf4 8 0 52 26 0 0 0 0 0 26 26 0 0 0 0
8 85 sdf5 25078 14 391214 59265 46638 2432 2292756 15394003 0 1824345 15436722 0 0 0 0
11 0 sr0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
253 0 dm-0 670 0 63808 150 419 0 3352 82 0 84 232 0 0 0 0

[stef97]

the full error message is ( and it is the same for all devices i.e sdb, sdf ...etc)
msg="ERROR: diskstats collector failed after 0.001248s: invalid line for /proc/diskstats for sdb" source="collector.go:132"

[discordianfish]

@stef97 That looks like a different problem. Can you open a new issue?

[pgier]

@stef97 Looks like the problem you are seeing could be the same as #1049 which is fixed in latest master. Can you try with latest master? And then as @discordianfish said if you still see the error please open a new issue.

[SuperQ]

@pgier @stef97 Version >= 0.17.0-rc.0 should have this fix.

[stef97]

yes ver 0.17 works as expected Thanks

…

On Tue, 30 Oct 2018 at 10:32, Ben Kochie ***@***.***> wrote: @pgier <https://github.com/pgier> @stef97 <https://github.com/stef97> Version >= 0.17.0-rc.0 should have this fix. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#392 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJn1v4smLmppuTjnNTbaZRhRq4CW_kIeks5uqGLjgaJpZM4LXnbJ> .

[zeleniy]

I have same problem on Ubuntu 14.04.2 LTS. The solution is to run node_exporter with --no-collector.diskstats option.

[dss010101]

seeing this same er

I have same problem on Ubuntu 14.04.2 LTS. The solution is to run node_exporter with --no-collector.diskstats option.

dosent seeem like a solution...just avoiding the issue. i have same issue on lastest

nodeexporter | ts=2023-05-13T20:24:00.721Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data