Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ethtool collector: panic: "node_ethtool_Queue[0] AllocFails" is not a valid metric name #2083

Closed
hhoffstaette opened this issue Jul 16, 2021 · 6 comments · Fixed by #2093
Closed

Ethtool collector: panic: "node_ethtool_Queue[0] AllocFails" is not a valid metric name #2083

hhoffstaette opened this issue Jul 16, 2021 · 6 comments · Fixed by #2093
Assignees
@ventifus
Labels
accepted bug

Comments

@hhoffstaette
Copy link
Contributor

Host operating system: output of uname -a

Linux tux 5.10.51 #1 SMP Thu Jul 15 22:19:10 CEST 2021 x86_64 Intel(R) Core(TM) i5-2400S CPU @ 2.50GHz GenuineIntel GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.2.0 (branch: non-git, revision: b597c12)
build user: portage@localhost
build date: 20210716-16:36:09
go version: go1.16.6
platform: linux/amd64

node_exporter command line flags

--collector.ethtool

Are you running node_exporter in Docker?

No

What did you do that produced an error?

I was excited to try the new ethtool collector in 1.2 and enabled it.

What did you expect to see?

ethtool metrics

What did you see instead?

panic: "node_ethtool_Queue[0] AllocFails" is not a valid metric name

goroutine 96 [running]:
github.com/prometheus/client_golang/prometheus.MustNewConstMetric(...)
	/tmp/portage/app-metrics/node_exporter-1.2.0/homedir/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/value.go:107
github.com/prometheus/node_exporter/collector.(*ethtoolCollector).Update(0xc00026ca40, 0xc000284060, 0x10df580, 0x0)
	/tmp/portage/app-metrics/node_exporter-1.2.0/work/node_exporter-1.2.0/collector/ethtool_linux.go:186 +0x1266
github.com/prometheus/node_exporter/collector.execute(0xc0f535, 0x7, 0xcf1e20, 0xc00026ca40, 0xc000284060, 0xcf19a0, 0xc00023c640)
	/tmp/portage/app-metrics/node_exporter-1.2.0/work/node_exporter-1.2.0/collector/collector.go:161 +0x84
github.com/prometheus/node_exporter/collector.NodeCollector.Collect.func1(0xc000284060, 0xc0003361b0, 0xcf19a0, 0xc00023c640, 0xc0003975e0, 0xc0f535, 0x7, 0xcf1e20, 0xc00026ca40)
	/tmp/portage/app-metrics/node_exporter-1.2.0/work/node_exporter-1.2.0/collector/collector.go:152 +0x71
created by github.com/prometheus/node_exporter/collector.NodeCollector.Collect
	/tmp/portage/app-metrics/node_exporter-1.2.0/work/node_exporter-1.2.0/collector/collector.go:151 +0x13c

ethtool -S eth0 says:


NIC statistics:
     InPackets: 1525155
     InUCast: 1423648
     InMCast: 20857
     InBCast: 80650
     InErrors: 0
     OutPackets: 1615342
     OutUCast: 1610383
     OutMCast: 10
     OutBCast: 4949
     InUCastOctets: 4609239418
     OutUCastOctets: 9663166233
     InMCastOctets: 5196003
     OutMCastOctets: 908
     InBCastOctets: 5169872
     OutBCastOctets: 316736
     InOctets: 4619605293
     OutOctets: 9663483877
     InPacketsDma: 1465120
     OutPacketsDma: 1615342
     InOctetsDma: 4606058808
     OutOctetsDma: 9656856053
     InDroppedDma: 0
     Queue[0] InPackets: 432377
     Queue[0] InJumboPackets: 753636
     Queue[0] InLroPackets: 0
     Queue[0] InErrors: 0
     Queue[0] AllocFails: 0
     Queue[0] SkbAllocFails: 0
     Queue[0] Polls: 615012
     Queue[0] OutPackets: 23072
     Queue[0] Restarts: 0
     Queue[1] InPackets: 433936
     Queue[1] InJumboPackets: 976600
     Queue[1] InLroPackets: 0
     Queue[1] InErrors: 0
     Queue[1] AllocFails: 0
     Queue[1] SkbAllocFails: 0
     Queue[1] Polls: 990271
     Queue[1] OutPackets: 353545
     Queue[1] Restarts: 0
     Queue[2] InPackets: 92279
     Queue[2] InJumboPackets: 234171
     Queue[2] InLroPackets: 0
     Queue[2] InErrors: 1
     Queue[2] AllocFails: 0
     Queue[2] SkbAllocFails: 0
     Queue[2] Polls: 310637
     Queue[2] OutPackets: 172772
     Queue[2] Restarts: 0
     Queue[3] InPackets: 506515
     Queue[3] InJumboPackets: 12372
     Queue[3] InLroPackets: 0
     Queue[3] InErrors: 9
     Queue[3] AllocFails: 0
     Queue[3] SkbAllocFails: 0
     Queue[3] Polls: 712999
     Queue[3] OutPackets: 206824
     Queue[3] Restarts: 0

I suspect this collector was not tested with a Multi-Queue NIC.
@hhoffstaette
Copy link
Contributor Author

Suspicion confirmed: I tried on another system with a single-queue NIC and there it works.

@discordianfish
Copy link
Member

@ventifus wanna look into fixing this?

@ventifus
Copy link
Contributor

Yes I'll take care of this. Please give me a few days I'm on vacation. Thanks!

@discordianfish
Copy link
Member

@ventifus Thanks! Enjoy your vacation!

@bdrung
Copy link
Contributor

bdrung commented Jul 21, 2021 •
edited
Loading

We ran into the same issue for different values:
Tx LPI entry count:

panic: "node_ethtool_Tx LPI entry count" is not a valid metric name

port.VF_admin_queue_requests:

panic: "node_ethtool_port.VF_admin_queue_requests" is not a valid metric name

@bdrung bdrung mentioned this issue Jul 21, 2021
Merged
@Sangshaai
Copy link

I ran into the same issue for different values:

level=debug ts=2021-08-11T07:06:40.325Z caller=node_exporter.go:76 msg="collect query:" filters="unsupported value type"
panic: "node_ethtool_port.VF_admin_queue_requests" is not a valid metric name

bdrung added a commit to bdrung/node_exporter that referenced this issue Aug 16, 2021
@bdrung
ethtool: Sanitize metric names
f13217d 
OpenMetrics and the Prometheus exposition format require the metric name
to consist only of alphanumericals and "_", ":" and they must not start
with digits. The metric names from the ethtool stats might contain
spaces, brackets, and dots. Converting them directly to metric names
will produce invalid metric names.

Therefore sanitize the metric names and convert them to lower case.

Fixes: prometheus#2083
Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com>
discordianfish pushed a commit that referenced this issue Aug 16, 2021
@bdrung @discordianfish
ethtool: Sanitize metric names
6ac6ea2 
OpenMetrics and the Prometheus exposition format require the metric name
to consist only of alphanumericals and "_", ":" and they must not start
with digits. The metric names from the ethtool stats might contain
spaces, brackets, and dots. Converting them directly to metric names
will produce invalid metric names.

Therefore sanitize the metric names and convert them to lower case.

Fixes: #2083
Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com>
oblitorum pushed a commit to shatteredsilicon/node_exporter that referenced this issue Apr 9, 2024
@bdrung @oblitorum
ethtool: Sanitize metric names
eb02086 
OpenMetrics and the Prometheus exposition format require the metric name
to consist only of alphanumericals and "_", ":" and they must not start
with digits. The metric names from the ethtool stats might contain
spaces, brackets, and dots. Converting them directly to metric names
will produce invalid metric names.

Therefore sanitize the metric names and convert them to lower case.

Fixes: prometheus#2083
Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com>
oblitorum pushed a commit to shatteredsilicon/node_exporter that referenced this issue Apr 9, 2024
@bdrung @oblitorum
ethtool: Sanitize metric names
1b385c3 
OpenMetrics and the Prometheus exposition format require the metric name
to consist only of alphanumericals and "_", ":" and they must not start
with digits. The metric names from the ethtool stats might contain
spaces, brackets, and dots. Converting them directly to metric names
will produce invalid metric names.

Therefore sanitize the metric names and convert them to lower case.

Fixes: prometheus#2083
Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ventifus ventifus

Labels
accepted bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ethtool: Sanitize metric names bdrung/node_exporter
5 participants
@bdrung @discordianfish @hhoffstaette @Sangshaai @ventifus