Index out of range error in parseMDStat #3198
Description
Host operating system: output of uname -a
Linux dl01harvester 5.14.21-150400.24.128-default #1 SMP PREEMPT_DYNAMIC Wed Aug 7 10:28:44 UTC 2024 (a6f23d4) x86_64 x86_64 x86_64 GNU/Linux
node_exporter version: output of node_exporter --version
node_exporter, version 1.3.1 (branch: HEAD, revision: a2321e7)
build user: root@243aafa5525c
build date: 20211205-11:09:49
go version: go1.17.3
platform: linux/amd64
node_exporter command line flags
--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/host/root --web.listen-address=[$(HOST_IP)]:9796 --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/) --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
node_exporter log output
ts=2024-12-04T08:40:33.142Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)"
2024-12-04T08:40:33.142899675Z ts=2024-12-04T08:40:33.142Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)"
ts=2024-12-04T08:40:33.143Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
ts=2024-12-04T08:40:33.143Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:108 level=info msg="Enabled collectors"
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=arp
2024-12-04T08:40:33.167589481Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=bcache
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=bonding
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=btrfs
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=conntrack
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=cpu
2024-12-04T08:40:33.167619910Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=cpufreq
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=diskstats
2024-12-04T08:40:33.167639956Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=dmi
2024-12-04T08:40:33.167653853Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=edac
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=entropy
2024-12-04T08:40:33.167661318Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=fibrechannel
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=filefd
2024-12-04T08:40:33.167700466Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=filesystem
2024-12-04T08:40:33.167703446Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=hwmon
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=infiniband
2024-12-04T08:40:33.167708931Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=ipvs
2024-12-04T08:40:33.167711757Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=loadavg
2024-12-04T08:40:33.167714401Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=mdadm
2024-12-04T08:40:33.167726630Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=meminfo
2024-12-04T08:40:33.167729740Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=netclass
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=netdev
2024-12-04T08:40:33.167734993Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=netstat
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=nfs
2024-12-04T08:40:33.167750616Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=nfsd
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=nvme
2024-12-04T08:40:33.167762791Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=os
2024-12-04T08:40:33.167765824Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=powersupplyclass
2024-12-04T08:40:33.167773264Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=pressure
2024-12-04T08:40:33.167776028Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=rapl
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=schedstat
2024-12-04T08:40:33.167781018Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=sockstat
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=softnet
2024-12-04T08:40:33.167790872Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=stat
2024-12-04T08:40:33.167793592Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=tapestats
2024-12-04T08:40:33.167800476Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=textfile
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=thermal_zone
2024-12-04T08:40:33.167805716Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=time
2024-12-04T08:40:33.167810999Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=timex
2024-12-04T08:40:33.167813587Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=udp_queues
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=uname
2024-12-04T08:40:33.167821372Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=vmstat
2024-12-04T08:40:33.167828189Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=xfs
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=zfs
2024-12-04T08:40:33.167961038Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:199 level=info msg="Listening on" address=[0.0.0.0]:9796
2024-12-04T08:40:33.168159794Z ts=2024-12-04T08:40:33.168Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false
panic: runtime error: index out of range [0] with length 0
2024-12-04T08:41:23.971372712Z
2024-12-04T08:41:23.971377402Z goroutine 119 [running]:
2024-12-04T08:41:23.971380878Z github.com/prometheus/procfs.evalStatusLine({0xc00047809e, 0x4b}, {0xc0004780ea, 0x6})
2024-12-04T08:41:23.971385218Z /go/pkg/mod/github.com/prometheus/procfs@v0.7.4-0.20211011103944-1a7a2bd3279f/mdstat.go:170 +0x498
2024-12-04T08:41:23.971388501Z github.com/prometheus/procfs.parseMDStat({0xc000476c00, 0xc0003a8f90, 0xa})
2024-12-04T08:41:23.971401903Z /go/pkg/mod/github.com/prometheus/procfs@v0.7.4-0.20211011103944-1a7a2bd3279f/mdstat.go:105 +0x2e9
2024-12-04T08:41:23.971407103Z github.com/prometheus/procfs.FS.MDStat({{0x7ffe2d2c9fa7, 0xa}})
/go/pkg/mod/github.com/prometheus/procfs@v0.7.4-0.20211011103944-1a7a2bd3279f/mdstat.go:71 +0xfb
2024-12-04T08:41:23.971413562Z github.com/prometheus/node_exporter/collector.(*mdadmCollector).Update(0xc00030b340, 0xc0000fb860)
2024-12-04T08:41:23.971417282Z /app/collector/mdadm_linux.go:111 +0xbb
2024-12-04T08:41:23.971420994Z github.com/prometheus/node_exporter/collector.execute({0xb1130b, 0x0}, {0xbfa980, 0xc00030b340}, 0x0, {0xbfa3c0, 0xc0002bc500})
2024-12-04T08:41:23.971435112Z /app/collector/collector.go:161 +0x9c
2024-12-04T08:41:23.971454659Z github.com/prometheus/node_exporter/collector.NodeCollector.Collect.func1({0xb1130b, 0x0}, {0xbfa980, 0xc00030b340})
2024-12-04T08:41:23.971485454Z /app/collector/collector.go:152 +0x3d
2024-12-04T08:41:23.971489837Z created by github.com/prometheus/node_exporter/collector.NodeCollector.Collect
2024-12-04T08:41:23.971492702Z /app/collector/collector.go:151 +0xd5
Are you running node_exporter in Docker?
yes in kubernetes, using rancher and harvester, as an harvester addon
What did you do that produced an error?
Start the [node_exporter] container.
What did you expect to see?
[node_exporter] running
What did you see instead?
[node_exporter] crashed
From the log I see it was processing /proc/mdstat
And by cat /proc/mdstat shows
Personalities : [raid6] [raid5] [raid4]
md125 : inactive sdc1[29](S) sdb1[27](S) sde1[28](S) sda1[26](S) sdd1[30](S)
41937920 blocks super 1.2
md126 : active raid5 super 1.2 level 5, 64k chunk, algorithm 2 [4/0] [____]
md127 : inactive sdc2[28](S) sdb2[30](S) sda2[27](S) sdd2[29](S) sde2[26](S)
10480640 blocks super 1.2
unused devices: <none>
I don't know exactly which part of this file cause the error.