fatal error: all goroutines are asleep - deadlock! #2705

andrgras · 2023-05-30T11:59:15Z

Host operating system: output of `uname -a`

Linux vm 5.4.0-136-generic #153-Ubuntu SMP Thu Nov 24 15:56:58 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of `node_exporter --version`

node_exporter, version 1.6.0 (branch: HEAD, revision: ff7f9d6)
build user: root@f9c3ed0cfbd3
build date: 20230527-12:03:54
go version: go1.20.4
platform: linux/amd64
tags: netgo osusergo static_build

node_exporter command line flags

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter --collector.systemd

node_exporter log output

May 30 14:46:55 vm node_exporter[412095]: fatal error: all goroutines are asleep - deadlock!
May 30 14:46:56 vm node_exporter[412095]: goroutine 1 [semacquire, 1495 minutes]:
May 30 14:46:56 vm node_exporter[412095]: sync.runtime_Semacquire(0xc0000061a0?)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/runtime/sema.go:62 +0x27
May 30 14:46:56 vm node_exporter[412095]: sync.(*WaitGroup).Wait(0x7f3462854401?)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/sync/waitgroup.go:116 +0x4b
May 30 14:46:56 vm node_exporter[412095]: golang.org/x/sync/errgroup.(*Group).Wait(0xc0001faa80)
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:53 +0x27
May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/exporter-toolkit/web.ServeMultiple({0xc0002217b0, 0x1, 0xbcadc7?}, 0xc00016d770, 0xc0001642e8, {0xce3360?, 0xc0000419c0})
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:235 +0x151
May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/exporter-toolkit/web.ListenAndServe(0x11381a0?, 0xc0001642e8, {0xce3360, 0xc0000419c0})
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:268 +0x488
May 30 14:46:56 vm node_exporter[412095]: main.main()
May 30 14:46:56 vm node_exporter[412095]: /app/node_exporter.go:210 +0x164e
May 30 14:46:56 vm node_exporter[412095]: goroutine 16 [IO wait, 1491 minutes]:
May 30 14:46:56 vm node_exporter[412095]: internal/poll.runtime_pollWait(0x7f34629ff898, 0x72)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/runtime/netpoll.go:306 +0x89
May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*pollDesc).wait(0xc000225780?, 0x5?, 0x0)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*pollDesc).waitRead(...)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*FD).Accept(0xc000225780)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd
May 30 14:46:56 vm node_exporter[412095]: net.(*netFD).accept(0xc000225780)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/fd_unix.go:172 +0x35
May 30 14:46:56 vm node_exporter[412095]: net.(*TCPListener).accept(0xc0001fc408)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/tcpsock_posix.go:148 +0x25
May 30 14:46:56 vm node_exporter[412095]: net.(*TCPListener).Accept(0xc0001fc408)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/tcpsock.go:297 +0x3d
May 30 14:46:56 vm node_exporter[412095]: net/http.(*Server).Serve(0xc00016d770, {0xce8800, 0xc0001fc408})
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/http/server.go:3059 +0x385
May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/exporter-toolkit/web.Serve({0xce8800, 0xc0001fc408}, 0xc00016d770, 0xc0001642e8, {0xce3360?, 0xc0000419c0})
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:278 +0x35f
May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/exporter-toolkit/web.ServeMultiple.func1()
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:232 +0x31
May 30 14:46:56 vm node_exporter[412095]: golang.org/x/sync/errgroup.(*Group).Go.func1()
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:75 +0x64
May 30 14:46:56 vm node_exporter[412095]: created by golang.org/x/sync/errgroup.(*Group).Go
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:72 +0xa5
May 30 14:46:56 vm node_exporter[412095]: goroutine 4236 [select]:
May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/client_golang/prometheus.(*Registry).Gather(0xc0002865a0)
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:522 +0x9f1
May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/client_golang/prometheus.Gatherers.Gather({0xc000231d60, 0x2, 0xc0000903c0?})
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:753 +0x166
May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/client_golang/prometheus.(*noTransactionGatherer).Gather(0xee3?)
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:1073 +0x22
May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/client_golang/prometheus/promhttp.HandlerForTransactional.func1({0x7f34627c01b8, 0xc000228190}, 0xc00064a000)
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/http.go:135 +0xfe
May 30 14:46:56 vm node_exporter[412095]: net/http.HandlerFunc.ServeHTTP(0xce5480?, {0x7f34627c01b8?, 0xc000228190?}, 0xc000658180?)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/http/server.go:2122 +0x2f
May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1({0x7f34627c01b8, 0xc000228190}, 0xce8a00?)
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/instrument_server.go:60 +0xd4
May 30 14:46:56 vm node_exporter[412095]: net/http.HandlerFunc.ServeHTTP(0xce8a10?, {0x7f34627c01b8?, 0xc000228190?}, 0x7f3489d88108?)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/http/server.go:2122 +0x2f
May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0xce8a10?, 0xc0001f8000?}, 0xc00064a000)
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/instrument_server.go:147 +0xc5
May 30 14:46:56 vm node_exporter[412095]: net/http.HandlerFunc.ServeHTTP(0xc000228140?, {0xce8a10?, 0xc0001f8000?}, 0x2?)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/http/server.go:2122 +0x2f
May 30 14:46:56 vm node_exporter[412095]: main.(*handler).ServeHTTP(0xc000041b40, {0xce8a10, 0xc0001f8000}, 0xc00064a000)
May 30 14:46:56 vm node_exporter[412095]: /app/node_exporter.go:82 +0x3f2
May 30 14:46:56 vm node_exporter[412095]: net/http.(*ServeMux).ServeHTTP(0x0?, {0xce8a10, 0xc0001f8000}, 0xc00064a000)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/http/server.go:2500 +0x149
May 30 14:46:56 vm node_exporter[412095]: net/http.serverHandler.ServeHTTP({0xc0002cc0f0?}, {0xce8a10, 0xc0001f8000}, 0xc00064a000)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/http/server.go:2936 +0x316
May 30 14:46:56 vm node_exporter[412095]: net/http.(*conn).serve(0xc0000ea1b0, {0xce8f70, 0xc0001ffa70})
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/http/server.go:1995 +0x612
May 30 14:46:56 vm node_exporter[412095]: created by net/http.(*Server).Serve
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/http/server.go:3089 +0x5ed
May 30 14:46:56 vm node_exporter[412095]: goroutine 1664356 [chan receive]:
May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.(*Conn).send.func1()
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:560 +0x45
May 30 14:46:56 vm node_exporter[412095]: created by github.com/godbus/dbus/v5.(*Conn).send
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:559 +0x42c
May 30 14:46:56 vm node_exporter[412095]: goroutine 1664245 [semacquire]:
May 30 14:46:56 vm node_exporter[412095]: sync.runtime_Semacquire(0x0?)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/runtime/sema.go:62 +0x27
May 30 14:46:56 vm node_exporter[412095]: sync.(*WaitGroup).Wait(0xc0003d5790?)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/sync/waitgroup.go:116 +0x4b
May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2()
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:472 +0x2f
May 30 14:46:56 vm node_exporter[412095]: created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:471 +0x61b
May 30 14:46:56 vm node_exporter[412095]: goroutine 1664305 [IO wait]:
May 30 14:46:56 vm node_exporter[412095]: internal/poll.runtime_pollWait(0x7f34629ff5c8, 0x72)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/runtime/netpoll.go:306 +0x89
May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*pollDesc).wait(0xc0005e7200?, 0xc0002b0480?, 0x0)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*pollDesc).waitRead(...)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*FD).ReadMsg(0xc0005e7200, {0xc0002b0480, 0x10, 0x10}, {0xc00001c620, 0x1000, 0x1000}, 0x70000c000066400?)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_unix.go:304 +0x3aa
May 30 14:46:56 vm node_exporter[412095]: net.(*netFD).readMsg(0xc0005e7200, {0xc0002b0480?, 0x10f5118?, 0xc0002b0388?}, {0xc00001c620?, 0xc0002b027e?, 0xc0002b0278?}, 0xc0002b0274?)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/fd_posix.go:78 +0x37
May 30 14:46:56 vm node_exporter[412095]: net.(*UnixConn).readMsg(0xc0002124c8, {0xc0002b0480?, 0xc000310710?, 0xce2ec0?}, {0xc00001c620?, 0xcec200?, 0x11675f8?})
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/unixsock_posix.go:115 +0x4f
May 30 14:46:56 vm node_exporter[412095]: net.(*UnixConn).ReadMsgUnix(0xc0002124c8, {0xc0002b0480?, 0xc000473d50?, 0x43bb60?}, {0xc00001c620?, 0x40dcea?, 0xc0001fd7b8?})
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/unixsock.go:143 +0x3c
May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.(*oobReader).Read(0xc00001c600, {0xc0002b0480?, 0xc000473de8?, 0x40e0c7?})
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/transport_unix.go:21 +0x45
May 30 14:46:56 vm node_exporter[412095]: io.ReadAtLeast({0xce3480, 0xc00001c600}, {0xc0002b0480, 0x10, 0x10}, 0x10)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/io/io.go:332 +0x9a
May 30 14:46:56 vm node_exporter[412095]: io.ReadFull(...)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/io/io.go:351
May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.(*unixTransport).ReadMessage(0xc00060cf60)
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/transport_unix.go:91 +0x11e
May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.(*Conn).inWorker(0xc00016da40)
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:390 +0x48
May 30 14:46:56 vm node_exporter[412095]: created by github.com/godbus/dbus/v5.(*Conn).Auth
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/auth.go:118 +0x99c
May 30 14:46:56 vm node_exporter[412095]: goroutine 1664355 [chan receive]:
May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.(*Conn).send.func1()
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:560 +0x45
May 30 14:46:56 vm node_exporter[412095]: created by github.com/godbus/dbus/v5.(*Conn).send
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:559 +0x42c
May 30 14:46:56 vm node_exporter[412095]: goroutine 1664314 [chan receive]:
May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.(*Object).CallWithContext(0xbca663?, {0xce8f00, 0xc000038048}, {0xbed119, 0x23}, 0x7?, {0xc000285ee0, 0x2, 0x2})
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/object.go:39 +0xdc
May 30 14:46:56 vm node_exporter[412095]: github.com/coreos/go-systemd/v22/dbus.(*Conn).getProperty(0xc00027b880, {0xce8f00, 0xc000038048}, {0xc000400510, 0x10}, {0xc000320e00, 0x20}, {0xbca663, 0x4})
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/methods.go:310 +0x205
May 30 14:46:56 vm node_exporter[412095]: github.com/coreos/go-systemd/v22/dbus.(*Conn).GetUnitTypePropertyContext(...)
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/methods.go:376
May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/node_exporter/collector.(*systemdCollector).collectUnitStatusMetrics(0xc000133900, 0xc0003daf01?, 0x4fdf06?, {0xc00059a000?, 0xcb, 0x0?})
May 30 14:46:56 vm node_exporter[412095]: /app/collector/systemd_linux.go:274 +0x4d3
May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/node_exporter/collector.(*systemdCollector).Update.func1()
May 30 14:46:56 vm node_exporter[412095]: /app/collector/systemd_linux.go:219 +0x108
May 30 14:46:56 vm node_exporter[412095]: created by github.com/prometheus/node_exporter/collector.(*systemdCollector).Update
May 30 14:46:56 vm node_exporter[412095]: /app/collector/systemd_linux.go:216 +0xcca
May 30 14:46:56 vm node_exporter[412095]: goroutine 1664240 [IO wait]:
May 30 14:46:56 vm node_exporter[412095]: internal/poll.runtime_pollWait(0x7f34629ff7a8, 0x72)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/runtime/netpoll.go:306 +0x89
May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*pollDesc).wait(0xc00027a400?, 0xc0002cc101?, 0x0)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*pollDesc).waitRead(...)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*FD).Read(0xc00027a400, {0xc0002cc101, 0x1, 0x1})
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299
May 30 14:46:56 vm node_exporter[412095]: net.(*netFD).Read(0xc00027a400, {0xc0002cc101?, 0xc0000419c0?, 0x10035a3c0?})
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/fd_posix.go:55 +0x29
May 30 14:46:56 vm node_exporter[412095]: net.(*conn).Read(0xc000212000, {0xc0002cc101?, 0x9e471d?, 0xbcf46c?})
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/net.go:183 +0x45
May 30 14:46:56 vm node_exporter[412095]: net/http.(*connReader).backgroundRead(0xc0002cc0f0)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/http/server.go:674 +0x3f
May 30 14:46:56 vm node_exporter[412095]: created by net/http.(*connReader).startBackgroundRead
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/http/server.go:670 +0xca
May 30 14:46:56 vm node_exporter[412095]: goroutine 1664304 [chan receive]:
May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.newConn.func1()
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:303 +0x32
May 30 14:46:56 vm node_exporter[412095]: created by github.com/godbus/dbus/v5.newConn
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:302 +0x4f6
May 30 14:46:56 vm node_exporter[412095]: goroutine 1664246 [semacquire]:
May 30 14:46:56 vm node_exporter[412095]: sync.runtime_Semacquire(0xc000659fb0?)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/runtime/sema.go:62 +0x27
May 30 14:46:56 vm node_exporter[412095]: sync.(*WaitGroup).Wait(0xc0003d7e90?)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/sync/waitgroup.go:116 +0x4b
May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/node_exporter/collector.NodeCollector.Collect({0xc000154c90?, {0xce3360?, 0xc0000419c0?}}, 0xc00042ee40)
May 30 14:46:56 vm node_exporter[412095]: /app/collector/collector.go:156 +0x24f
May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:455 +0x10d
May 30 14:46:56 vm node_exporter[412095]: created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:547 +0xc09
May 30 14:46:56 vm node_exporter[412095]: goroutine 1664307 [chan receive]:
May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.newConn.func1()
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:303 +0x32
May 30 14:46:56 vm node_exporter[412095]: created by github.com/godbus/dbus/v5.newConn
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:302 +0x4f6
May 30 14:46:56 vm node_exporter[412095]: goroutine 1664308 [IO wait]:
May 30 14:46:56 vm node_exporter[412095]: internal/poll.runtime_pollWait(0x7f34629ff4d8, 0x72)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/runtime/netpoll.go:306 +0x89
May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*pollDesc).wait(0xc00027b280?, 0xc00026e5d0?, 0x0)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*pollDesc).waitRead(...)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*FD).ReadMsg(0xc00027b280, {0xc00026e5d0, 0x10, 0x10}, {0xc000426020, 0x1000, 0x1000}, 0x7000000000000c0?)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_unix.go:304 +0x3aa
May 30 14:46:56 vm node_exporter[412095]: net.(*netFD).readMsg(0xc00027b280, {0xc00026e5d0?, 0x10f5118?, 0x11675f8?}, {0xc000426020?, 0xc00026e476?, 0xc00026e470?}, 0xc00026e46c?)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/fd_posix.go:78 +0x37
May 30 14:46:56 vm node_exporter[412095]: net.(*UnixConn).readMsg(0xc000212d08, {0xc00026e5d0?, 0xc00038ad00?, 0x416d8b?}, {0xc000426020?, 0x423127?, 0xab7980?})
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/unixsock_posix.go:115 +0x4f
May 30 14:46:56 vm node_exporter[412095]: net.(*UnixConn).ReadMsgUnix(0xc000212d08, {0xc00026e5d0?, 0xc000479d50?, 0x43bb60?}, {0xc000426020?, 0x40dcea?, 0xc000334e10?})
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/unixsock.go:143 +0x3c
May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.(*oobReader).Read(0xc000426000, {0xc00026e5d0?, 0xc000479de8?, 0x40e0c7?})
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/transport_unix.go:21 +0x45
May 30 14:46:56 vm node_exporter[412095]: io.ReadAtLeast({0xce3480, 0xc000426000}, {0xc00026e5d0, 0x10, 0x10}, 0x10)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/io/io.go:332 +0x9a
May 30 14:46:56 vm node_exporter[412095]: io.ReadFull(...)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/io/io.go:351
May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.(*unixTransport).ReadMessage(0xc000334228)
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/transport_unix.go:91 +0x11e
May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.(*Conn).inWorker(0xc00016dc20)
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:390 +0x48
May 30 14:46:56 vm node_exporter[412095]: created by github.com/godbus/dbus/v5.(*Conn).Auth
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/auth.go:118 +0x99c
May 30 14:46:56 vm node_exporter[412095]: goroutine 1664311 [chan receive]:
May 30 14:46:56 vm node_exporter[412095]: github.com/coreos/go-systemd/v22/dbus.(*Conn).dispatch.func1()
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/subscription.go:55 +0x52
May 30 14:46:56 vm node_exporter[412095]: created by github.com/coreos/go-systemd/v22/dbus.(*Conn).dispatch
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/subscription.go:53 +0x9b
May 30 14:46:56 vm node_exporter[412095]: goroutine 1664316 [chan receive]:
May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.(*Object).CallWithContext(0xbd2627?, {0xce8f00, 0xc000038048}, {0xbed119, 0x23}, 0x6?, {0xc000285fc0, 0x2, 0x2})
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/object.go:39 +0xdc
May 30 14:46:56 vm node_exporter[412095]: github.com/coreos/go-systemd/v22/dbus.(*Conn).getProperty(0xc00027b880, {0xce8f00, 0xc000038048}, {0xc000401780, 0xf}, {0xc000320e80, 0x1f}, {0xbd2627, 0xc})
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/methods.go:310 +0x205
May 30 14:46:56 vm node_exporter[412095]: github.com/coreos/go-systemd/v22/dbus.(*Conn).GetUnitTypePropertyContext(...)
May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/methods.go:376
May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/node_exporter/collector.(*systemdCollector).collectSockets(0xc000133900, 0xce6c98?, 0xc00032f270?, {0xc00059a000?, 0xcb, 0xac6ce0?})
May 30 14:46:56 vm node_exporter[412095]: /app/collector/systemd_linux.go:326 +0x4ec
May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/node_exporter/collector.(*systemdCollector).Update.func5()
May 30 14:46:56 vm node_exporter[412095]: /app/collector/systemd_linux.go:257 +0x108
May 30 14:46:56 vm node_exporter[412095]: created by github.com/prometheus/node_exporter/collector.(*systemdCollector).Update
May 30 14:46:56 vm node_exporter[412095]: /app/collector/systemd_linux.go:254 +0x111b
May 30 14:46:56 vm node_exporter[412095]: goroutine 1664270 [semacquire]:
May 30 14:46:56 vm node_exporter[412095]: sync.runtime_Semacquire(0x8?)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/runtime/sema.go:62 +0x27
May 30 14:46:56 vm node_exporter[412095]: sync.(*WaitGroup).Wait(0x1138300?)
May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/sync/waitgroup.go:116 +0x4b
May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/node_exporter/collector.(*systemdCollector).Update(0xc000133900, 0xc00042ee40)
May 30 14:46:56 vm node_exporter[412095]: /app/collector/systemd_linux.go:267 +0x1369
May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/node_exporter/collector.execute({0xbce40f, 0x7}, {0xce3e40, 0xc000133900}, 0xc000362fd0?, {0xce3360, 0xc0000419c0})
May 30 14:46:56 vm node_exporter[412095]: /app/collector/collector.go:161 +0x9c
May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/node_exporter/collector.NodeCollector.Collect.func1({0xbce40f?, 0xc0003016b0?}, {0xce3e40?, 0xc000133900?})
May 30 14:46:56 vm node_exporter[412095]: /app/collector/collector.go:152 +0x3d
May 30 14:46:56 vm node_exporter[412095]: created by github.com/prometheus/node_exporter/collector.NodeCollector.Collect
May 30 14:46:56 vm node_exporter[412095]: /app/collector/collector.go:151 +0xd0
May 30 14:46:56 vm systemd[1]: node_exporter.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 30 14:46:56 vm systemd[1]: node_exporter.service: Failed with result 'exit-code'.

Are you running node_exporter in Docker?

What did you do that produced an error?

Upgraded node_exporter yesterday from 1.5.0 > 1.6.0

What did you expect to see?

There was no fatal errors on 1.5.0 version

What did you see instead?

The text was updated successfully, but these errors were encountered:

SuperQ · 2023-05-30T12:01:23Z

Please include the requested detailed information about your runtime and configuration.

andrgras · 2023-05-30T13:34:48Z

Please include the requested detailed information about your runtime and configuration.

Added more information

mguegan · 2023-05-31T14:49:26Z

Hi, I'd like to add a similar bug report on version 1.6.0 that happened ~30' after upgrade. For the moment only one host has produced this error.

Host operating system: output of `uname -a`

Linux HOSTNAME 3.10.0-1160.88.1.el7.x86_64 #1 SMP Tue Mar 7 15:41:52 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of `node_exporter --version`

node_exporter, version 1.6.0 (branch: HEAD, revision: ff7f9d69b645cb691dd3e84dc3afc88f5c006962)
  build user:       root@f9c3ed0cfbd3
  build date:       20230527-12:03:54
  go version:       go1.20.4
  platform:         linux/amd64
  tags:             netgo osusergo static_build

node_exporter command line flags

node_exporter --collector.systemd

node_exporter log output

-- Logs begin at Tue 2023-04-18 10:56:35 CEST, end at Wed 2023-05-31 16:36:51 CEST. --                                                                                                                             
May 31 15:37:25 HOSTNAME systemd[1]: Started Prometheus node exporter.                                                                                                                                             
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.860Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.6.0, branch=HEAD, revision=ff7f9d69b645cb691dd3e
84dc3afc88f5c006962)"                                                                                                                                                                                              
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.860Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.20.4, platform=linux/amd64, user=root@f9c3ed0cfbd3, da
te=20230527-12:03:54, tags=netgo osusergo static_build)"                                                                                                                                                           
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.860Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(de
v|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/)                                                                                                                                
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.861Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|
binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.861Z caller=systemd_linux.go:152 level=info collector=systemd msg="Parsed flag --collector.systemd.unit-include" flag=.+
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.861Z caller=systemd_linux.go:154 level=info collector=systemd msg="Parsed flag --collector.systemd.unit-exclude" flag=.+\.(automount|device|m
ount|scope|slice)                                                                                        
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.862Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|
(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$                                                                       
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:110 level=info msg="Enabled collectors"
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=arp
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=bcache
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=bonding
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=btrfs                                                             
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=conntrack 
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=cpu                             
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=cpufreq   
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=diskstats
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=dmi
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=edac
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=entropy
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=fibrechannel
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=filefd
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=filesystem
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=hwmon
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=infiniband
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=ipvs
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=loadavg
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=mdadm
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=meminfo
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=netclass
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=netdev
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=netstat
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=nfs
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=nfsd
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=nvme
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=os
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=powersupplyclass
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=pressure
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=rapl
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=schedstat
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=selinux
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=sockstat
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=softnet
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=stat
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=systemd
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=tapestats
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=textfile
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=thermal_zone
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=time
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=timex
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=udp_queues
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=uname
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=vmstat
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=xfs
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.863Z caller=node_exporter.go:117 level=info collector=zfs
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.865Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100
May 31 15:37:25 HOSTNAME node_exporter[32628]: ts=2023-05-31T13:37:25.865Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100
May 31 16:06:08 HOSTNAME node_exporter[32628]: fatal error: all goroutines are asleep - deadlock!
May 31 16:06:08 HOSTNAME node_exporter[32628]: goroutine 1 [semacquire, 28 minutes]:
May 31 16:06:08 HOSTNAME node_exporter[32628]: sync.runtime_Semacquire(0xc0000061a0?)
May 31 16:06:08 HOSTNAME node_exporter[32628]: /usr/local/go/src/runtime/sema.go:62 +0x27
May 31 16:06:08 HOSTNAME node_exporter[32628]: sync.(*WaitGroup).Wait(0x7fe9c2675e01?)
May 31 16:06:08 HOSTNAME node_exporter[32628]: /usr/local/go/src/sync/waitgroup.go:116 +0x4b
May 31 16:06:08 HOSTNAME node_exporter[32628]: golang.org/x/sync/errgroup.(*Group).Wait(0xc00022ea00)
May 31 16:06:08 HOSTNAME node_exporter[32628]: /go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:53 +0x27
May 31 16:06:08 HOSTNAME node_exporter[32628]: github.com/prometheus/exporter-toolkit/web.ServeMultiple({0xc0002977b0, 0x1, 0xbcadc7?}, 0xc00016a690, 0xc000203788, {0xce3360?, 0xc00022e580})
May 31 16:06:08 HOSTNAME node_exporter[32628]: /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:235 +0x151
May 31 16:06:08 HOSTNAME node_exporter[32628]: github.com/prometheus/exporter-toolkit/web.ListenAndServe(0x11381a0?, 0xc000203788, {0xce3360, 0xc00022e580})
May 31 16:06:08 HOSTNAME node_exporter[32628]: /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:268 +0x488
May 31 16:06:08 HOSTNAME systemd[1]: node_exporter.service: main process exited, code=exited, status=2/INVALIDARGUMENT
May 31 16:06:08 HOSTNAME systemd[1]: Unit node_exporter.service entered failed state.
May 31 16:06:08 HOSTNAME systemd[1]: node_exporter.service failed.

Are you running node_exporter in Docker?

No. Directly on a VM.

What did you do that produced an error?

Upgraded to latest release produced that error.

SuperQ · 2023-05-31T16:04:54Z

Thanks for the report. It seems like this is related to the systemd collector so far.

SuperQ · 2023-05-31T16:09:39Z

I checked the code changes. We made no changes to systemd collection. Either in the collector code or libraries used to access systemd.

andrgras · 2023-05-31T17:55:40Z

I have 3x Linux machines, so far "fatal error: all goroutines are asleep - deadlock!" happened on two of them

dswarbrick · 2023-05-31T20:00:17Z

@SuperQ Doesn't the stacktrace suggest more that the problem lies in exporter-toolkit? I would focus on what changed in exporter-toolkit between v0.8.2 (used by node_exporter v1.5.0) and v0.10.0.

Although a cursory glance at that diff doesn't really scream anything that would cause a deadlock.

SuperQ · 2023-05-31T21:05:58Z

@dswarbrick What I'm thinking is that there is an HA Prometheus setup, but by coincidence, the two instances of Prometheus have very close scrape timestamp.

For the reporters, assuming I'm correct in that you have HA Prometheus. Can you please post the results of an instant query for up{instance="node-that-is-failing:9100}[5m] from each Prometheus? I'd like to see the detailed timing data.

mguegan · 2023-06-01T04:18:56Z

I have a federation setup.
I have now a bunch of exporters that have crashed, I will rollback to the old version, but I will keep the 1.6.0 for testing.
As for the query:

from the 'local' prometheus:

1 
@1685592488.7151 
@1685592498.7151
@1685592508.721 
@1685592518.7151 
@1685592528.7151 
@1685592538.7151 
@1685592548.7151 
@1685592558.7151 
@1685592568.7151 
@1685592578.7151 
@1685592588.7151 
@1685592598.7151 
@1685592608.7151 
@1685592618.7171 
@1685592628.7151 
@1685592638.7151 
@1685592648.7151 
@1685592658.7151 
@1685592668.7151 
@1685592678.7151 
@1685592688.7151 
@1685592698.7151 
@1685592708.7181 
@1685592718.7151 
@1685592728.7151 
@1685592738.7151 
@1685592748.7181 
@1685592758.7151 
@1685592768.7151 
@1685592778.715

from the federation:

1 
@1685592488.7151 
@1685592508.721 
@1685592518.7151
@1685592538.7151 
@1685592548.7151 
@1685592568.7151 
@1685592578.7151 
@1685592598.7151 
@1685592608.7151 
@1685592628.7151 
@1685592638.7151 
@1685592658.7151 
@1685592668.7151 
@1685592688.7151 
@1685592698.7151 
@1685592718.7151 
@1685592728.7151 
@1685592748.7181 
@1685592758.7151 
@1685592778.715

mguegan · 2023-06-01T05:36:29Z

I can confirm that bug/crashes occurs only when exporter is inside a Prometheus federation setup. Other deployments done without federation (with only one Prometheus) are running fine in my case.

SuperQ · 2023-06-01T11:25:47Z

Federation should make no difference, as only one Prometheus is hitting the exporter. The question is about HA, where there are two distinct instances hitting the exporter.

mguegan · 2023-06-01T12:38:37Z

This maybe a coincidence, but I had 6-7 crashes on different VM on a pool of ~150 exporters with Federation. On another pool of ~100 hosts without federation, no problem so far. But I don't want to influence you, just giving more info on my setup.
I have no more than one Prometheus scrapping exporters on each setup. I have just verified on an impacted host with tcpdump.

HinShell · 2023-06-01T13:01:28Z

Hi,
Same issue here, with only one prometheus. Node_exporter installed on six fresh servers, crashed only on one of them.

Host operating system: output of `uname -a`

Linux vm 5.10.0-22-amd64 #1 SMP Debian 5.10.178-3 (2023-04-22) x86_64 GNU/Linux

node_exporter version: output of `node_exporter --version`

node_exporter, version 1.6.0 (branch: HEAD, revision: ff7f9d6)
build user: root@f9c3ed0cfbd3
build date: 20230527-12:03:54
go version: go1.20.4
platform: linux/amd64
tags: netgo osusergo static_build

node_exporter command line flags

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
--web.config.file=/etc/node_exporter/web.yml
--collector.systemd
--collector.processes
--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|run|var/tmp)($|/)
--collector.textfile.directory=/opt/node_exporter
--collector.netdev.address-info
--web.listen-address=ip:9100

node_exporter log output

May 31 21:33:32 vm node_exporter[74299]: fatal error: all goroutines are asleep - deadlock!
May 31 21:33:32 vm node_exporter[74299]: goroutine 1 [semacquire, 3409 minutes]:
May 31 21:33:32 vm node_exporter[74299]: sync.runtime_Semacquire(0xc0000061a0?)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/runtime/sema.go:62 +0x27
May 31 21:33:32 vm node_exporter[74299]: sync.(*WaitGroup).Wait(0x7f491fbcdf01?)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/sync/waitgroup.go:116 +0x4b
May 31 21:33:32 vm node_exporter[74299]: golang.org/x/sync/errgroup.(*Group).Wait(0xc000230d00)
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:53 +0x27
May 31 21:33:32 vm node_exporter[74299]: github.com/prometheus/exporter-toolkit/web.ServeMultiple({0xc000217940, 0x1, 0x7ffc6030cedb?}, 0xc0000fa690, 0xc000115758, {0xce3360?, 0xc0001425c0})
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:235 +0x151
May 31 21:33:32 vm node_exporter[74299]: github.com/prometheus/exporter-toolkit/web.ListenAndServe(0x11381a0?, 0xc000115758, {0xce3360, 0xc0001425c0})
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:268 +0x488
May 31 21:33:32 vm node_exporter[74299]: main.main()
May 31 21:33:32 vm node_exporter[74299]: #11/app/node_exporter.go:210 +0x164e
May 31 21:33:32 vm node_exporter[74299]: goroutine 50 [IO wait, 3409 minutes]:
May 31 21:33:32 vm node_exporter[74299]: internal/poll.runtime_pollWait(0x7f491fb97898, 0x72)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/runtime/netpoll.go:306 +0x89
May 31 21:33:32 vm node_exporter[74299]: internal/poll.(*pollDesc).wait(0xc00021ba00?, 0xc000113998?, 0x0)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
May 31 21:33:32 vm node_exporter[74299]: internal/poll.(*pollDesc).waitRead(...)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
May 31 21:33:32 vm node_exporter[74299]: internal/poll.(*FD).Accept(0xc00021ba00)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd
May 31 21:33:32 vm node_exporter[74299]: net.(*netFD).accept(0xc00021ba00)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/fd_unix.go:172 +0x35
May 31 21:33:32 vm node_exporter[74299]: net.(*TCPListener).accept(0xc0001fc660)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/tcpsock_posix.go:148 +0x25
May 31 21:33:32 vm node_exporter[74299]: net.(*TCPListener).Accept(0xc0001fc660)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/tcpsock.go:297 +0x3d
May 31 21:33:32 vm node_exporter[74299]: crypto/tls.(*listener).Accept(0xc0001fc900)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/crypto/tls/tls.go:66 +0x2d
May 31 21:33:32 vm node_exporter[74299]: net/http.(*Server).Serve(0xc0000fa690, {0xce84d0, 0xc0001fc900})
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/http/server.go:3059 +0x385
May 31 21:33:32 vm node_exporter[74299]: net/http.(*Server).ServeTLS(0xc0000fa690, {0xce8800?, 0xc0001fc660}, {0x0, 0x0}, {0x0, 0x0})
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/http/server.go:3129 +0x4bd
May 31 21:33:32 vm node_exporter[74299]: github.com/prometheus/exporter-toolkit/web.Serve({0xce8800, 0xc0001fc660}, 0xc0000fa690, 0xc000115758, {0xce3360?, 0xc0001425c0})
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:332 +0xa10
May 31 21:33:32 vm node_exporter[74299]: github.com/prometheus/exporter-toolkit/web.ServeMultiple.func1()
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:232 +0x31
May 31 21:33:32 vm node_exporter[74299]: golang.org/x/sync/errgroup.(*Group).Go.func1()
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:75 +0x64
May 31 21:33:32 vm node_exporter[74299]: created by golang.org/x/sync/errgroup.(*Group).Go
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:72 +0xa5
May 31 21:33:32 vm node_exporter[74299]: goroutine 51 [select]:
May 31 21:33:32 vm node_exporter[74299]: net/http.(*http2serverConn).serve(0xc0001a0d00)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/http/h2_bundle.go:4675 +0x8ac
May 31 21:33:32 vm node_exporter[74299]: net/http.(*http2Server).ServeConn(0xc00021ec30, {0xcec678?, 0xc00015bc00}, 0xc000153b20)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/http/h2_bundle.go:4270 +0xbe5
May 31 21:33:32 vm node_exporter[74299]: net/http.http2ConfigureServer.func1(0xc0000fa690, 0xcec678?, {0xce5c80, 0xc000226bc0})
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/http/h2_bundle.go:4060 +0xff
May 31 21:33:32 vm node_exporter[74299]: net/http.(*conn).serve(0xc0000ea750, {0xce8f70, 0xc0002705a0})
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/http/server.go:1903 +0x1293
May 31 21:33:32 vm node_exporter[74299]: created by net/http.(*Server).Serve
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/http/server.go:3089 +0x5ed
May 31 21:33:32 vm node_exporter[74299]: goroutine 54 [IO wait]:
May 31 21:33:32 vm node_exporter[74299]: internal/poll.runtime_pollWait(0x7f491fb977a8, 0x72)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/runtime/netpoll.go:306 +0x89
May 31 21:33:32 vm node_exporter[74299]: internal/poll.(*pollDesc).wait(0xc00021be00?, 0xc000269200?, 0x0)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
May 31 21:33:32 vm node_exporter[74299]: internal/poll.(*pollDesc).waitRead(...)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
May 31 21:33:32 vm node_exporter[74299]: internal/poll.(*FD).Read(0xc00021be00, {0xc000269200, 0x600, 0x600})
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/internal/poll/fd_unix.go:167 +0x299
May 31 21:33:32 vm node_exporter[74299]: net.(*netFD).Read(0xc00021be00, {0xc000269200?, 0xc000269205?, 0x22?})
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/fd_posix.go:55 +0x29
May 31 21:33:32 vm node_exporter[74299]: net.(*conn).Read(0xc000012498, {0xc000269200?, 0x0?, 0xc00015bdb0?})
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/net.go:183 +0x45
May 31 21:33:32 vm node_exporter[74299]: crypto/tls.(*atLeastReader).Read(0xc0003a8090, {0xc000269200?, 0xc0003a8090?, 0x0?})
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/crypto/tls/conn.go:788 +0x3d
May 31 21:33:32 vm node_exporter[74299]: bytes.(*Buffer).ReadFrom(0xc00015be90, {0xce3060, 0xc0003a8090})
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/bytes/buffer.go:202 +0x98
May 31 21:33:32 vm node_exporter[74299]: crypto/tls.(*Conn).readFromUntil(0xc00015bc00, {0xce48e0?, 0xc000012498}, 0x600?)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/crypto/tls/conn.go:810 +0xe5
May 31 21:33:32 vm node_exporter[74299]: crypto/tls.(*Conn).readRecordOrCCS(0xc00015bc00, 0x0)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/crypto/tls/conn.go:617 +0x116
May 31 21:33:32 vm node_exporter[74299]: crypto/tls.(*Conn).readRecord(...)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/crypto/tls/conn.go:583
May 31 21:33:32 vm node_exporter[74299]: crypto/tls.(*Conn).Read(0xc00015bc00, {0xc00024d620, 0x9, 0xc00042800c?})
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/crypto/tls/conn.go:1316 +0x16f
May 31 21:33:32 vm node_exporter[74299]: io.ReadAtLeast({0x7f491f97e738, 0xc00015bc00}, {0xc00024d620, 0x9, 0x9}, 0x9)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/io/io.go:332 +0x9a
May 31 21:33:32 vm node_exporter[74299]: io.ReadFull(...)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/io/io.go:351
May 31 21:33:32 vm node_exporter[74299]: net/http.http2readFrameHeader({0xc00024d620?, 0x9?, 0xc0001e2300?}, {0x7f491f97e738?, 0xc00015bc00?})
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/http/h2_bundle.go:1567 +0x6e
May 31 21:33:32 vm node_exporter[74299]: net/http.(*http2Framer).ReadFrame(0xc00024d5e0)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/http/h2_bundle.go:1831 +0x95
May 31 21:33:32 vm node_exporter[74299]: net/http.(*http2serverConn).readFrames(0xc0001a0d00)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/http/h2_bundle.go:4560 +0x91
May 31 21:33:32 vm node_exporter[74299]: created by net/http.(*http2serverConn).serve
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/http/h2_bundle.go:4667 +0x55c
May 31 21:33:32 vm node_exporter[74299]: goroutine 826380 [select]:
May 31 21:33:32 vm node_exporter[74299]: github.com/prometheus/client_golang/prometheus.(*Registry).Gather(0xc00021e640)
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:522 +0x9f1
May 31 21:33:32 vm node_exporter[74299]: github.com/prometheus/client_golang/prometheus.Gatherers.Gather({0xc00011de40, 0x2, 0xc000090240?})
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:753 +0x166
May 31 21:33:32 vm node_exporter[74299]: github.com/prometheus/client_golang/prometheus.(*noTransactionGatherer).Gather(0x911f32?)
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:1073 +0x22
May 31 21:33:32 vm node_exporter[74299]: github.com/prometheus/client_golang/prometheus/promhttp.HandlerForTransactional.func1({0x7f491f9830f0, 0xc0003d8140}, 0xc0000dc100)
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/http.go:135 +0xfe
May 31 21:33:32 vm node_exporter[74299]: net/http.HandlerFunc.ServeHTTP(0x1138b40?, {0x7f491f9830f0?, 0xc0003d8140?}, 0xc0001e2570?)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/http/server.go:2122 +0x2f
May 31 21:33:32 vm node_exporter[74299]: github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1({0x7f491f9830f0, 0xc0003d8140}, 0xce8900?)
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/instrument_server.go:60 +0xd4
May 31 21:33:32 vm node_exporter[74299]: net/http.HandlerFunc.ServeHTTP(0xce8950?, {0x7f491f9830f0?, 0xc0003d8140?}, 0x7f4946f60108?)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/http/server.go:2122 +0x2f
May 31 21:33:32 vm node_exporter[74299]: github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0xce8950?, 0xc00020a000?}, 0xc0000dc100)
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/instrument_server.go:147 +0xc5
May 31 21:33:32 vm node_exporter[74299]: net/http.HandlerFunc.ServeHTTP(0xc00008e1e0?, {0xce8950?, 0xc00020a000?}, 0x2?)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/http/server.go:2122 +0x2f
May 31 21:33:32 vm node_exporter[74299]: main.(*handler).ServeHTTP(0xc00003f480, {0xce8950, 0xc00020a000}, 0xc0000dc100)
May 31 21:33:32 vm node_exporter[74299]: #11/app/node_exporter.go:82 +0x3f2
May 31 21:33:32 vm node_exporter[74299]: net/http.(*ServeMux).ServeHTTP(0x7ffc6030cdd8?, {0xce8950, 0xc00020a000}, 0xc0000dc100)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/http/server.go:2500 +0x149
May 31 21:33:32 vm node_exporter[74299]: github.com/prometheus/exporter-toolkit/web.(*webHandler).ServeHTTP(0xc000230e40, {0xce8950, 0xc00020a000}, 0x7f491fbcd808?)
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/handler.go:102 +0x42b
May 31 21:33:32 vm node_exporter[74299]: net/http.serverHandler.ServeHTTP({0xb2b7a0?}, {0xce8950, 0xc00020a000}, 0xc0000dc100)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/http/server.go:2936 +0x316
May 31 21:33:32 vm node_exporter[74299]: net/http.initALPNRequest.ServeHTTP({{0xce8f70?, 0xc000270660?}, 0xc00015bc00?, {0xc0000fa690?}}, {0xce8950, 0xc00020a000}, 0xc0000dc100)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/http/server.go:3545 +0x245
May 31 21:33:32 vm node_exporter[74299]: net/http.(*http2serverConn).runHandler(0xc0000901e0?, 0xc0002eef90?, 0xc00023e9c0?, 0xc0002eefa0?)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/http/h2_bundle.go:6049 +0x83
May 31 21:33:32 vm node_exporter[74299]: created by net/http.(*http2serverConn).processHeaders
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/http/h2_bundle.go:5762 +0x68a
May 31 21:33:32 vm node_exporter[74299]: goroutine 826414 [chan receive]:
May 31 21:33:32 vm node_exporter[74299]: github.com/godbus/dbus/v5.(*Object).Call(0x0?, {0xbe1e84, 0x1a}, 0xf0?, {0x0, 0x0, 0x0})
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/object.go:34 +0xd3
May 31 21:33:32 vm node_exporter[74299]: github.com/godbus/dbus/v5.(*Conn).Hello(0xc0000faff0)
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:377 +0x62
May 31 21:33:32 vm node_exporter[74299]: github.com/coreos/go-systemd/v22/dbus.dbusAuthHelloConnection({0xce8f00?, 0xc000036048?}, 0x4160f0?)
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/dbus.go:256 +0x34
May 31 21:33:32 vm node_exporter[74299]: github.com/coreos/go-systemd/v22/dbus.NewSystemConnectionContext.func1()
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/dbus.go:138 +0x28
May 31 21:33:32 vm node_exporter[74299]: github.com/coreos/go-systemd/v22/dbus.NewConnection(0xc0001579f0)
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/dbus.go:195 +0x51
May 31 21:33:32 vm node_exporter[74299]: github.com/coreos/go-systemd/v22/dbus.NewSystemConnectionContext({0xce8f00?, 0xc000036048?})
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/dbus.go:137 +0x4d
May 31 21:33:32 vm node_exporter[74299]: github.com/coreos/go-systemd/v22/dbus.NewWithContext({0xce8f00, 0xc000036048})
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/dbus.go:122 +0x27
May 31 21:33:32 vm node_exporter[74299]: github.com/prometheus/node_exporter/collector.newSystemdDbusConn()
May 31 21:33:32 vm node_exporter[74299]: #11/app/collector/systemd_linux.go:442 +0x52
May 31 21:33:32 vm node_exporter[74299]: github.com/prometheus/node_exporter/collector.(*systemdCollector).Update(0xc00013ce80, 0xc0004b0180)
May 31 21:33:32 vm node_exporter[74299]: #11/app/collector/systemd_linux.go:180 +0xa5
May 31 21:33:32 vm node_exporter[74299]: github.com/prometheus/node_exporter/collector.execute({0xbce40f, 0x7}, {0xce3e40, 0xc00013ce80}, 0xc000320fd0?, {0xce3360, 0xc0001425c0})
May 31 21:33:32 vm node_exporter[74299]: #11/app/collector/collector.go:161 +0x9c
May 31 21:33:32 vm node_exporter[74299]: github.com/prometheus/node_exporter/collector.NodeCollector.Collect.func1({0xbce40f?, 0xc00052df80?}, {0xce3e40?, 0xc00013ce80?})
May 31 21:33:32 vm node_exporter[74299]: #11/app/collector/collector.go:152 +0x3d
May 31 21:33:32 vm node_exporter[74299]: created by github.com/prometheus/node_exporter/collector.NodeCollector.Collect
May 31 21:33:32 vm node_exporter[74299]: #11/app/collector/collector.go:151 +0xd0
May 31 21:33:32 vm node_exporter[74299]: goroutine 826447 [chan receive]:
May 31 21:33:32 vm node_exporter[74299]: github.com/godbus/dbus/v5.newConn.func1()
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:303 +0x32
May 31 21:33:32 vm node_exporter[74299]: created by github.com/godbus/dbus/v5.newConn
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:302 +0x4f6
May 31 21:33:32 vm node_exporter[74299]: goroutine 826452 [chan receive]:
May 31 21:33:32 vm node_exporter[74299]: github.com/godbus/dbus/v5.(*Conn).send.func1()
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:560 +0x45
May 31 21:33:32 vm node_exporter[74299]: created by github.com/godbus/dbus/v5.(*Conn).send
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:559 +0x42c
May 31 21:33:32 vm node_exporter[74299]: goroutine 826386 [semacquire]:
May 31 21:33:32 vm node_exporter[74299]: sync.runtime_Semacquire(0xc000302090?)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/runtime/sema.go:62 +0x27
May 31 21:33:32 vm node_exporter[74299]: sync.(*WaitGroup).Wait(0xc0002eae90?)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/sync/waitgroup.go:116 +0x4b
May 31 21:33:32 vm node_exporter[74299]: github.com/prometheus/node_exporter/collector.NodeCollector.Collect({0xc00018eb70?, {0xce3360?, 0xc0001425c0?}}, 0xc0004b0180)
May 31 21:33:32 vm node_exporter[74299]: #11/app/collector/collector.go:156 +0x24f
May 31 21:33:32 vm node_exporter[74299]: github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:455 +0x10d
May 31 21:33:32 vm node_exporter[74299]: created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
May 31 21:33:32 vm systemd[1]: node_exporter.service: Failed with result 'exit-code'.
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:547 +0xc09
May 31 21:33:32 vm node_exporter[74299]: goroutine 826448 [IO wait]:
May 31 21:33:32 vm node_exporter[74299]: internal/poll.runtime_pollWait(0x7f491fb976b8, 0x72)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/runtime/netpoll.go:306 +0x89
May 31 21:33:32 vm node_exporter[74299]: internal/poll.(*pollDesc).wait(0xc0002be980?, 0xc000345a30?, 0x0)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
May 31 21:33:32 vm node_exporter[74299]: internal/poll.(*pollDesc).waitRead(...)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
May 31 21:33:32 vm node_exporter[74299]: internal/poll.(*FD).ReadMsg(0xc0002be980, {0xc000345a30, 0x10, 0x10}, {0xc00042a020, 0x1000, 0x1000}, 0x700000000000080?)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/internal/poll/fd_unix.go:304 +0x3aa
May 31 21:33:32 vm node_exporter[74299]: net.(*netFD).readMsg(0xc0002be980, {0xc000345a30?, 0x10f5118?, 0xc0003458f4?}, {0xc00042a020?, 0xc0003457ae?, 0xc0003457a8?}, 0xc0003457a4?)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/fd_posix.go:78 +0x37
May 31 21:33:32 vm node_exporter[74299]: net.(*UnixConn).readMsg(0xc00020b598, {0xc000345a30?, 0xc00043a150?, 0xc00001fd90?}, {0xc00042a020?, 0xcec200?, 0x11675f8?})
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/unixsock_posix.go:115 +0x4f
May 31 21:33:32 vm node_exporter[74299]: net.(*UnixConn).ReadMsgUnix(0xc00020b598, {0xc000345a30?, 0x0?, 0x0?}, {0xc00042a020?, 0x40dd1f?, 0xc000458f18?})
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/unixsock.go:143 +0x3c
May 31 21:33:32 vm node_exporter[74299]: github.com/godbus/dbus/v5.(*oobReader).Read(0xc00042a000, {0xc000345a30?, 0xc00001fde8?, 0x40e0c7?})
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/transport_unix.go:21 +0x45
May 31 21:33:32 vm node_exporter[74299]: io.ReadAtLeast({0xce3480, 0xc00042a000}, {0xc000345a30, 0x10, 0x10}, 0x10)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/io/io.go:332 +0x9a
May 31 21:33:32 vm node_exporter[74299]: io.ReadFull(...)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/io/io.go:351
May 31 21:33:32 vm node_exporter[74299]: github.com/godbus/dbus/v5.(*unixTransport).ReadMessage(0xc00033ed80)
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/transport_unix.go:91 +0x11e
May 31 21:33:32 vm node_exporter[74299]: github.com/godbus/dbus/v5.(*Conn).inWorker(0xc0000fae10)
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:390 +0x48
May 31 21:33:32 vm node_exporter[74299]: created by github.com/godbus/dbus/v5.(*Conn).Auth
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/auth.go:118 +0x99c
May 31 21:33:32 vm node_exporter[74299]: goroutine 826450 [chan receive]:
May 31 21:33:32 vm node_exporter[74299]: github.com/godbus/dbus/v5.newConn.func1()
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:303 +0x32
May 31 21:33:32 vm node_exporter[74299]: created by github.com/godbus/dbus/v5.newConn
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:302 +0x4f6
May 31 21:33:32 vm node_exporter[74299]: goroutine 826451 [IO wait]:
May 31 21:33:32 vm node_exporter[74299]: internal/poll.runtime_pollWait(0x7f491fb972f8, 0x72)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/runtime/netpoll.go:306 +0x89
May 31 21:33:32 vm node_exporter[74299]: internal/poll.(*pollDesc).wait(0xc00021ac00?, 0xc000345b40?, 0x0)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
May 31 21:33:32 vm node_exporter[74299]: internal/poll.(*pollDesc).waitRead(...)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
May 31 21:33:32 vm node_exporter[74299]: internal/poll.(*FD).ReadMsg(0xc00021ac00, {0xc000345b40, 0x10, 0x10}, {0xc00042b320, 0x1000, 0x1000}, 0xc000375c30?)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/internal/poll/fd_unix.go:304 +0x3aa
May 31 21:33:32 vm node_exporter[74299]: net.(*netFD).readMsg(0xc00021ac00, {0xc000345b40?, 0x40e0c7?, 0xa?}, {0xc00042b320?, 0x10?, 0x7f491fbcdf88?}, 0x0?)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/fd_posix.go:78 +0x37
May 31 21:33:32 vm node_exporter[74299]: net.(*UnixConn).readMsg(0xc000140820, {0xc000345b40?, 0xc000314d10?, 0x416d8b?}, {0xc00042b320?, 0x423127?, 0xb62d40?})
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/unixsock_posix.go:115 +0x4f
May 31 21:33:32 vm node_exporter[74299]: net.(*UnixConn).ReadMsgUnix(0xc000140820, {0xc000345b40?, 0x41b511?, 0x1168760?}, {0xc00042b320?, 0x40de48?, 0xc00042b300?})
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/net/unixsock.go:143 +0x3c
May 31 21:33:32 vm node_exporter[74299]: github.com/godbus/dbus/v5.(*oobReader).Read(0xc00042b300, {0xc000345b40?, 0xc000314de8?, 0x40e0c7?})
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/transport_unix.go:21 +0x45
May 31 21:33:32 vm node_exporter[74299]: io.ReadAtLeast({0xce3480, 0xc00042b300}, {0xc000345b40, 0x10, 0x10}, 0x10)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/io/io.go:332 +0x9a
May 31 21:33:32 vm node_exporter[74299]: io.ReadFull(...)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/io/io.go:351
May 31 21:33:32 vm node_exporter[74299]: github.com/godbus/dbus/v5.(*unixTransport).ReadMessage(0xc000458f48)
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/transport_unix.go:91 +0x11e
May 31 21:33:32 vm node_exporter[74299]: github.com/godbus/dbus/v5.(*Conn).inWorker(0xc0000faff0)
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:390 +0x48
May 31 21:33:32 vm node_exporter[74299]: created by github.com/godbus/dbus/v5.(*Conn).Auth
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/auth.go:118 +0x99c
May 31 21:33:32 vm node_exporter[74299]: goroutine 826385 [semacquire]:
May 31 21:33:32 vm node_exporter[74299]: sync.runtime_Semacquire(0x0?)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/runtime/sema.go:62 +0x27
May 31 21:33:32 vm node_exporter[74299]: sync.(*WaitGroup).Wait(0xc00030df90?)
May 31 21:33:32 vm node_exporter[74299]: #11/usr/local/go/src/sync/waitgroup.go:116 +0x4b
May 31 21:33:32 vm node_exporter[74299]: github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2()
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:472 +0x2f
May 31 21:33:32 vm node_exporter[74299]: created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
May 31 21:33:32 vm node_exporter[74299]: #11/go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:471 +0x61b

Are you running node_exporter in Docker?

Nop, on VM.

What did you do that produced an error?

fresh install.

What did you expect to see?

No fatal error.

What did you see instead?

fatal error: all goroutines are asleep - deadlock!

aminvakil · 2023-06-01T16:15:50Z

I've upgraded a couple of my servers to 1.6.0 and today one of them crashed with this error, I'm not sure if this is the same error or I should open an issue, also I'm using a single node prometheus:

Host operating system: output of `uname -a`

Linux hostname 4.19.0-24-amd64 #1 SMP Debian 4.19.282-1 (2023-04-29) x86_64 GNU/Linux

node_exporter version: output of `node_exporter --version`

node_exporter, version 1.6.0 (branch: HEAD, revision: ff7f9d69b645cb691dd3e84dc3afc88f5c006962)
  build user:       root@f9c3ed0cfbd3
  build date:       20230527-12:03:54
  go version:       go1.20.4
  platform:         linux/amd64
  tags:             netgo osusergo static_build

node_exporter command line flags

node_exporter --collector.systemd --collector.systemd.unit-include=(nginx|node_exporter|sshd).service --collector.processes --web.listen-address=localhost:9100 --no-collector.softnet --no-collector.rapl

Not sure if it's related, but I've removed a couple of other custom systemd services which we are monitoring, and we have not problem with them since ever we're using prometheus, all services were fine when this happened.

node_exporter log output

node_exporter[804]: runtime: checkdead: find g 294180 in status 1
node_exporter[804]: fatal error: checkdead: runnable g
node_exporter[804]: runtime stack:
node_exporter[804]: runtime.throw({0xbdbcc7?, 0x0?})
node_exporter[804]:         /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0xc000443d58 sp=0xc000443d28 pc=0x438d5d
node_exporter[804]: runtime.checkdead.func1(0xc000319520)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:5216 +0xe5 fp=0xc000443d90 sp=0xc000443d58 pc=0x446a85
node_exporter[804]: runtime.forEachG(0xc000443df0)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:591 +0x5c fp=0xc000443dc0 sp=0xc000443d90 pc=0x43c4fc
node_exporter[804]: runtime.checkdead()
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:5203 +0xf7 fp=0xc000443e10 sp=0xc000443dc0 pc=0x446777
node_exporter[804]: runtime.mput(0x0?)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:5696 +0x33 fp=0xc000443e20 sp=0xc000443e10 pc=0x447cf3
node_exporter[804]: runtime.stopm()
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:2332 +0x67 fp=0xc000443e50 sp=0xc000443e20 pc=0x43f927
node_exporter[804]: runtime.findRunnable()
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:3007 +0xabc fp=0xc000443f58 sp=0xc000443e50 pc=0x44119c
node_exporter[804]: runtime.schedule()
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:3360 +0xb1 fp=0xc000443f90 sp=0xc000443f58 pc=0x441fd1
node_exporter[804]: runtime.park_m(0xc000319520?)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:3511 +0x12d fp=0xc000443fc0 sp=0xc000443f90 pc=0x4424ed
node_exporter[804]: runtime.mcall()
node_exporter[804]:         /usr/local/go/src/runtime/asm_amd64.s:452 +0x43 fp=0xc000443fd0 sp=0xc000443fc0 pc=0x46cb03
node_exporter[804]: goroutine 1 [semacquire, 1334 minutes]:
node_exporter[804]: runtime.gopark(0x40dcea?, 0xc000206af0?, 0x0?, 0x0?, 0x48?)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000441950 sp=0xc000441930 pc=0x43ba76
node_exporter[804]: runtime.goparkunlock(...)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:387
node_exporter[804]: runtime.semacquire1(0xc00015ad90, 0x40?, 0x1, 0x0, 0xe0?)
node_exporter[804]:         /usr/local/go/src/runtime/sema.go:160 +0x20f fp=0xc0004419b8 sp=0xc000441950 pc=0x44cdaf
node_exporter[804]: sync.runtime_Semacquire(0xc0000061a0?)
node_exporter[804]:         /usr/local/go/src/runtime/sema.go:62 +0x27 fp=0xc0004419f0 sp=0xc0004419b8 pc=0x46ac07
node_exporter[804]: sync.(*WaitGroup).Wait(0x7fc4859ce301?)
node_exporter[804]:         /usr/local/go/src/sync/waitgroup.go:116 +0x4b fp=0xc000441a18 sp=0xc0004419f0 pc=0x48a76b
node_exporter[804]: golang.org/x/sync/errgroup.(*Group).Wait(0xc00015ad80)
node_exporter[804]:         /go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:53 +0x27 fp=0xc000441a30 sp=0xc000441a18 pc=0x92c867
node_exporter[804]: github.com/prometheus/exporter-toolkit/web.ServeMultiple({0xc0001177e0, 0x1, 0x7fff716a7ec6?}, 0xc0002b1860, 0xc00029e6c0, {0xce3360?, 0xc0001e65c0})
node_exporter[804]:         /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:235 +0x151 fp=0xc000441a80 sp=0xc000441a30 pc=0x92f831
node_exporter[804]: github.com/prometheus/exporter-toolkit/web.ListenAndServe(0x11381a0?, 0xc00029e6c0, {0xce3360, 0xc0001e65c0})
node_exporter[804]:         /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:268 +0x488 fp=0xc000441b58 sp=0xc000441a80 pc=0x92fd88
node_exporter[804]: main.main()
node_exporter[804]:         /app/node_exporter.go:210 +0x164e fp=0xc000441f80 sp=0xc000441b58 pc=0xa645ee
node_exporter[804]: runtime.main()
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:250 +0x207 fp=0xc000441fe0 sp=0xc000441f80 pc=0x43b647
node_exporter[804]: runtime.goexit()
node_exporter[804]:         /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000441fe8 sp=0xc000441fe0 pc=0x46ec01
node_exporter[804]: goroutine 2 [force gc (idle), 1081 minutes]:
node_exporter[804]: runtime.gopark(0xe46bcbdb167?, 0x0?, 0x0?, 0x0?, 0x0?)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00007cfb0 sp=0xc00007cf90 pc=0x43ba76
node_exporter[804]: runtime.goparkunlock(...)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:387
node_exporter[804]: runtime.forcegchelper()
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:305 +0xb0 fp=0xc00007cfe0 sp=0xc00007cfb0 pc=0x43b8b0
node_exporter[804]: runtime.goexit()
node_exporter[804]:         /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00007cfe8 sp=0xc00007cfe0 pc=0x46ec01
node_exporter[804]: created by runtime.init.6
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:293 +0x25
node_exporter[804]: goroutine 3 [GC sweep wait]:
node_exporter[804]: runtime.gopark(0x1137d01?, 0x0?, 0x0?, 0x0?, 0x0?)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00007d780 sp=0xc00007d760 pc=0x43ba76
node_exporter[804]: runtime.goparkunlock(...)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:387
node_exporter[804]: runtime.bgsweep(0x0?)
node_exporter[804]:         /usr/local/go/src/runtime/mgcsweep.go:319 +0xde fp=0xc00007d7c8 sp=0xc00007d780 pc=0x425ede
node_exporter[804]: runtime.gcenable.func1()
node_exporter[804]:         /usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc00007d7e0 sp=0xc00007d7c8 pc=0x41b146
node_exporter[804]: runtime.goexit()
node_exporter[804]:         /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00007d7e8 sp=0xc00007d7e0 pc=0x46ec01
node_exporter[804]: created by runtime.gcenable
node_exporter[804]:         /usr/local/go/src/runtime/mgc.go:178 +0x6b
node_exporter[804]: goroutine 4 [runnable]:
node_exporter[804]: runtime.gopark(0x49361a3bd829?, 0x34f91a?, 0x0?, 0x0?, 0x0?)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00007df70 sp=0xc00007df50 pc=0x43ba76
node_exporter[804]: runtime.goparkunlock(...)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:387
node_exporter[804]: runtime.(*scavengerState).park(0x11384a0)
node_exporter[804]:         /usr/local/go/src/runtime/mgcscavenge.go:400 +0x53 fp=0xc00007dfa0 sp=0xc00007df70 pc=0x423db3
node_exporter[804]: runtime.bgscavenge(0x0?)
node_exporter[804]:         /usr/local/go/src/runtime/mgcscavenge.go:633 +0x65 fp=0xc00007dfc8 sp=0xc00007dfa0 pc=0x4243a5
node_exporter[804]: runtime.gcenable.func2()
node_exporter[804]:         /usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc00007dfe0 sp=0xc00007dfc8 pc=0x41b0e6
node_exporter[804]: runtime.goexit()
node_exporter[804]:         /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00007dfe8 sp=0xc00007dfe0 pc=0x46ec01
node_exporter[804]: created by runtime.gcenable
node_exporter[804]:         /usr/local/go/src/runtime/mgc.go:179 +0xaa
node_exporter[804]: goroutine 5 [finalizer wait, 1334 minutes]:
node_exporter[804]: runtime.gopark(0x43bdf2?, 0x7fc4acd9b908?, 0x0?, 0x0?, 0xc00007c770?)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00007c628 sp=0xc00007c608 pc=0x43ba76
node_exporter[804]: runtime.runfinq()
node_exporter[804]:         /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc00007c7e0 sp=0xc00007c628 pc=0x41a187
node_exporter[804]: runtime.goexit()
node_exporter[804]:         /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00007c7e8 sp=0xc00007c7e0 pc=0x46ec01
node_exporter[804]: created by runtime.createfing
node_exporter[804]:         /usr/local/go/src/runtime/mfinal.go:163 +0x45
node_exporter[804]: goroutine 52 [IO wait]:
node_exporter[804]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000b2a78 sp=0xc0000b2a58 pc=0x43ba76
node_exporter[804]: runtime.netpollblock(0x7fc4859fe7a8?, 0x40482f?, 0x0?)
node_exporter[804]:         /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0000b2ab0 sp=0xc0000b2a78 pc=0x4343d7
node_exporter[804]: internal/poll.runtime_pollWait(0x7fc4859cb958, 0x72)
node_exporter[804]:         /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0000b2ad0 sp=0xc0000b2ab0 pc=0x469349
node_exporter[804]: internal/poll.(*pollDesc).wait(0xc000157980?, 0x4?, 0x0)
node_exporter[804]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0000b2af8 sp=0xc0000b2ad0 pc=0x4d9bf2
node_exporter[804]: internal/poll.(*pollDesc).waitRead(...)
node_exporter[804]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
node_exporter[804]: internal/poll.(*FD).Accept(0xc000157980)
node_exporter[804]:         /usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc0000b2ba0 sp=0xc0000b2af8 pc=0x4df4fd
node_exporter[804]: net.(*netFD).accept(0xc000157980)
node_exporter[804]:         /usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc0000b2c58 sp=0xc0000b2ba0 pc=0x5c12b5
node_exporter[804]: net.(*TCPListener).accept(0xc00015c2e8)
node_exporter[804]:         /usr/local/go/src/net/tcpsock_posix.go:148 +0x25 fp=0xc0000b2c80 sp=0xc0000b2c58 pc=0x5d85a5
node_exporter[804]: net.(*TCPListener).Accept(0xc00015c2e8)
node_exporter[804]:         /usr/local/go/src/net/tcpsock.go:297 +0x3d fp=0xc0000b2cb0 sp=0xc0000b2c80 pc=0x5d769d
node_exporter[804]: net/http.(*onceCloseListener).Accept(0xc000186990?)
node_exporter[804]:         <autogenerated>:1 +0x2a fp=0xc0000b2cc8 sp=0xc0000b2cb0 pc=0x6fd76a
node_exporter[804]: net/http.(*Server).Serve(0xc0002b1860, {0xce8800, 0xc00015c2e8})
node_exporter[804]:         /usr/local/go/src/net/http/server.go:3059 +0x385 fp=0xc0000b2df8 sp=0xc0000b2cc8 pc=0x6d7ba5
node_exporter[804]: github.com/prometheus/exporter-toolkit/web.Serve({0xce8800, 0xc00015c2e8}, 0xc0002b1860, 0xc00029e6c0, {0xce3360?, 0xc0001e65c0})
node_exporter[804]:         /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:278 +0x35f fp=0xc0000b2f38 sp=0xc0000b2df8 pc=0x9301bf
node_exporter[804]: github.com/prometheus/exporter-toolkit/web.ServeMultiple.func1()
node_exporter[804]:         /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:232 +0x31 fp=0xc0000b2f78 sp=0xc0000b2f38 pc=0x92f8d1
node_exporter[804]: golang.org/x/sync/errgroup.(*Group).Go.func1()
node_exporter[804]:         /go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:75 +0x64 fp=0xc0000b2fe0 sp=0xc0000b2f78 pc=0x92ca04
node_exporter[804]: runtime.goexit()
node_exporter[804]:         /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000b2fe8 sp=0xc0000b2fe0 pc=0x46ec01
node_exporter[804]: created by golang.org/x/sync/errgroup.(*Group).Go
node_exporter[804]:         /go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:72 +0xa5
node_exporter[804]: goroutine 294251 [chan receive]:
node_exporter[804]: runtime.gopark(0xc0000b4f28?, 0x472e45?, 0x28?, 0x4f?, 0x4fdf06?)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000b4f00 sp=0xc0000b4ee0 pc=0x43ba76
node_exporter[804]: runtime.chanrecv(0xc0000ac120, 0x0, 0x1)
node_exporter[804]:         /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc0000b4f90 sp=0xc0000b4f00 pc=0x406f9d
node_exporter[804]: runtime.chanrecv1(0xc0003fe050?, 0x9584e6?)
node_exporter[804]:         /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc0000b4fb8 sp=0xc0000b4f90 pc=0x406a98
node_exporter[804]: github.com/godbus/dbus/v5.newConn.func1()
node_exporter[804]:         /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:303 +0x32 fp=0xc0000b4fe0 sp=0xc0000b4fb8 pc=0x95ac12
node_exporter[804]: runtime.goexit()
node_exporter[804]:         /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000b4fe8 sp=0xc0000b4fe0 pc=0x46ec01
node_exporter[804]: created by github.com/godbus/dbus/v5.newConn
node_exporter[804]:         /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:302 +0x4f6
node_exporter[804]: goroutine 294162 [semacquire]:
node_exporter[804]: runtime.gopark(0xc00040a780?, 0x40704a?, 0xc0?, 0x63?, 0x472e45?)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00040a6f0 sp=0xc00040a6d0 pc=0x43ba76
node_exporter[804]: runtime.goparkunlock(...)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:387
node_exporter[804]: runtime.semacquire1(0xc0002a2ad8, 0x50?, 0x1, 0x0, 0x0?)
node_exporter[804]:         /usr/local/go/src/runtime/sema.go:160 +0x20f fp=0xc00040a758 sp=0xc00040a6f0 pc=0x44cdaf
node_exporter[804]: sync.runtime_Semacquire(0xc0000ad1a0?)
node_exporter[804]:         /usr/local/go/src/runtime/sema.go:62 +0x27 fp=0xc00040a790 sp=0xc00040a758 pc=0x46ac07
node_exporter[804]: sync.(*WaitGroup).Wait(0xc0003fe1e0?)
node_exporter[804]:         /usr/local/go/src/sync/waitgroup.go:116 +0x4b fp=0xc00040a7b8 sp=0xc00040a790 pc=0x48a76b
node_exporter[804]: github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2()
node_exporter[804]:         /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:472 +0x2f fp=0xc00040a7e0 sp=0xc00040a7b8 pc=0x8bd52f
node_exporter[804]: runtime.goexit()
node_exporter[804]:         /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00040a7e8 sp=0xc00040a7e0 pc=0x46ec01
node_exporter[804]: created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
node_exporter[804]:         /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:471 +0x61b
node_exporter[804]: goroutine 116 [GC worker (idle)]:
node_exporter[804]: runtime.gopark(0x49361eeb6b90?, 0x2?, 0xfd?, 0x56?, 0x8?)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00030af50 sp=0xc00030af30 pc=0x43ba76
node_exporter[804]: runtime.gcBgMarkWorker()
node_exporter[804]:         /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc00030afe0 sp=0xc00030af50 pc=0x41ceb1
node_exporter[804]: runtime.goexit()
node_exporter[804]:         /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00030afe8 sp=0xc00030afe0 pc=0x46ec01
node_exporter[804]: created by runtime.gcBgMarkStartWorkers
node_exporter[804]:         /usr/local/go/src/runtime/mgc.go:1199 +0x25
node_exporter[804]: goroutine 294156 [select]:
node_exporter[804]: runtime.gopark(0xc000512f10?, 0x2?, 0x2c?, 0xe3?, 0xc000512d2c?)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000512b68 sp=0xc000512b48 pc=0x43ba76
node_exporter[804]: runtime.selectgo(0xc000512f10, 0xc000512d28, 0x18?, 0x0, 0x0?, 0x1)
node_exporter[804]:         /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000512ca8 sp=0xc000512b68 pc=0x44bcbe
node_exporter[804]: github.com/prometheus/client_golang/prometheus.(*Registry).Gather(0xc0002064b0)
node_exporter[804]:         /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:522 +0x9f1 fp=0xc000513188 sp=0xc000512ca8 pc=0x8bd071
node_exporter[804]: github.com/prometheus/client_golang/prometheus.Gatherers.Gather({0xc00011df00, 0x2, 0x2?})
node_exporter[804]:         /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:753 +0x166 fp=0xc0005134f0 sp=0xc000513188 pc=0x8be6e6
node_exporter[804]: github.com/prometheus/client_golang/prometheus.(*Gatherers).Gather(0x65ad1c?)
node_exporter[804]:         <autogenerated>:1 +0x38 fp=0xc000513518 sp=0xc0005134f0 pc=0x8c7d58
node_exporter[804]: github.com/prometheus/client_golang/prometheus.(*noTransactionGatherer).Gather(0xe74?)
node_exporter[804]:         /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:1073 +0x22 fp=0xc000513530 sp=0xc000513518 pc=0x8c0ec2
node_exporter[804]: github.com/prometheus/client_golang/prometheus/promhttp.HandlerForTransactional.func1({0x7fc4858252f8, 0xc0000aa820}, 0xc00059ff00)
node_exporter[804]:         /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/http.go:135 +0xfe fp=0xc0005137c0 sp=0xc000513530 pc=0x8c9d1e
node_exporter[804]: net/http.HandlerFunc.ServeHTTP(0xce5480?, {0x7fc4858252f8?, 0xc0000aa820?}, 0xc0005d7380?)
node_exporter[804]:         /usr/local/go/src/net/http/server.go:2122 +0x2f fp=0xc0005137e8 sp=0xc0005137c0 pc=0x6d44ef
node_exporter[804]: github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1({0x7fc4858252f8, 0xc0000aa820}, 0xce8a00?)
node_exporter[804]:         /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/instrument_server.go:60 +0xd4 fp=0xc000513868 sp=0xc0005137e8 pc=0x8cb194
node_exporter[804]: net/http.HandlerFunc.ServeHTTP(0xce8a10?, {0x7fc4858252f8?, 0xc0000aa820?}, 0x7fc4acd90108?)
node_exporter[804]:         /usr/local/go/src/net/http/server.go:2122 +0x2f fp=0xc000513890 sp=0xc000513868 pc=0x6d44ef
node_exporter[804]: github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0xce8a10?, 0xc0001581c0?}, 0xc00059ff00)
node_exporter[804]:         /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/instrument_server.go:147 +0xc5 fp=0xc0005139a0 sp=0xc000513890 pc=0x8cb965
node_exporter[804]: net/http.HandlerFunc.ServeHTTP(0xc0000aa7d0?, {0xce8a10?, 0xc0001581c0?}, 0x2?)
node_exporter[804]:         /usr/local/go/src/net/http/server.go:2122 +0x2f fp=0xc0005139c8 sp=0xc0005139a0 pc=0x6d44ef
node_exporter[804]: main.(*handler).ServeHTTP(0xc0001e6740, {0xce8a10, 0xc0001581c0}, 0xc00059ff00)
node_exporter[804]:         /app/node_exporter.go:82 +0x3f2 fp=0xc000513a80 sp=0xc0005139c8 pc=0xa62632
node_exporter[804]: net/http.(*ServeMux).ServeHTTP(0x0?, {0xce8a10, 0xc0001581c0}, 0xc00059ff00)
node_exporter[804]:         /usr/local/go/src/net/http/server.go:2500 +0x149 fp=0xc000513ad0 sp=0xc000513a80 pc=0x6d5e89
node_exporter[804]: net/http.serverHandler.ServeHTTP({0xc0005d7170?}, {0xce8a10, 0xc0001581c0}, 0xc00059ff00)
node_exporter[804]:         /usr/local/go/src/net/http/server.go:2936 +0x316 fp=0xc000513b80 sp=0xc000513ad0 pc=0x6d7636
node_exporter[804]: net/http.(*conn).serve(0xc000186990, {0xce8f70, 0xc00015ff80})
node_exporter[804]:         /usr/local/go/src/net/http/server.go:1995 +0x612 fp=0xc000513fb8 sp=0xc000513b80 pc=0x6d3012
node_exporter[804]: net/http.(*Server).Serve.func3()
node_exporter[804]:         /usr/local/go/src/net/http/server.go:3089 +0x2e fp=0xc000513fe0 sp=0xc000513fb8 pc=0x6d7f8e
node_exporter[804]: runtime.goexit()
node_exporter[804]:         /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000513fe8 sp=0xc000513fe0 pc=0x46ec01
node_exporter[804]: created by net/http.(*Server).Serve
node_exporter[804]:         /usr/local/go/src/net/http/server.go:3089 +0x5ed
node_exporter[804]: goroutine 294163 [semacquire]:
node_exporter[804]: runtime.gopark(0xc2c548?, 0xaa?, 0x0?, 0x63?, 0x7fc4859fc0e8?)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000418d68 sp=0xc000418d48 pc=0x43ba76
node_exporter[804]: runtime.goparkunlock(...)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:387
node_exporter[804]: runtime.semacquire1(0xc0002a2ae8, 0x30?, 0x1, 0x0, 0x91?)
node_exporter[804]:         /usr/local/go/src/runtime/sema.go:160 +0x20f fp=0xc000418dd0 sp=0xc000418d68 pc=0x44cdaf
node_exporter[804]: sync.runtime_Semacquire(0xc0003d8210?)
node_exporter[804]:         /usr/local/go/src/runtime/sema.go:62 +0x27 fp=0xc000418e08 sp=0xc000418dd0 pc=0x46ac07
node_exporter[804]: sync.(*WaitGroup).Wait(0xc000418e90?)
node_exporter[804]:         /usr/local/go/src/sync/waitgroup.go:116 +0x4b fp=0xc000418e30 sp=0xc000418e08 pc=0x48a76b
node_exporter[804]: github.com/prometheus/node_exporter/collector.NodeCollector.Collect({0xc0001f3dd0?, {0xce3360?, 0xc0001e65c0?}}, 0xc00037e960)
node_exporter[804]:         /app/collector/collector.go:156 +0x24f fp=0xc000418f00 sp=0xc000418e30 pc=0x9e462f
node_exporter[804]: github.com/prometheus/node_exporter/collector.(*NodeCollector).Collect(0xc000418fb0?, 0xc000418f60?)
node_exporter[804]:         <autogenerated>:1 +0x3e fp=0xc000418f30 sp=0xc000418f00 pc=0xa615de
node_exporter[804]: github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()
node_exporter[804]:         /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:455 +0x10d fp=0xc000418fe0 sp=0xc000418f30 pc=0x8bd66d
node_exporter[804]: runtime.goexit()
node_exporter[804]:         /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000418fe8 sp=0xc000418fe0 pc=0x46ec01
node_exporter[804]: created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
node_exporter[804]:         /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:547 +0xc09
node_exporter[804]: goroutine 294180 [runnable]:
node_exporter[804]: runtime.gopark(0xc0001dc418?, 0xb?, 0x0?, 0x0?, 0xa?)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0005151f0 sp=0xc0005151d0 pc=0x43ba76
node_exporter[804]: runtime.netpollblock(0x4be225?, 0x40482f?, 0x0?)
node_exporter[804]:         /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000515228 sp=0xc0005151f0 pc=0x4343d7
node_exporter[804]: internal/poll.runtime_pollWait(0x7fc4857549d8, 0x72)
node_exporter[804]:         /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000515248 sp=0xc000515228 pc=0x469349
node_exporter[804]: internal/poll.(*pollDesc).wait(0xc0001dc400?, 0xc0004f6000?, 0x0)
node_exporter[804]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000515270 sp=0xc000515248 pc=0x4d9bf2
node_exporter[804]: internal/poll.(*pollDesc).waitRead(...)
node_exporter[804]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
node_exporter[804]: internal/poll.(*FD).Read(0xc0001dc400, {0xc0004f6000, 0x1000, 0x1000})
node_exporter[804]:         /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc000515308 sp=0xc000515270 pc=0x4dafd9
node_exporter[804]: net.(*netFD).Read(0xc0001dc400, {0xc0004f6000?, 0x0?, 0x1?})
node_exporter[804]:         /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000515350 sp=0xc000515308 pc=0x5bf129
node_exporter[804]: net.(*conn).Read(0xc0005c0628, {0xc0004f6000?, 0x4d9ab2?, 0xc0005153c8?})
node_exporter[804]:         /usr/local/go/src/net/net.go:183 +0x45 fp=0xc000515398 sp=0xc000515350 pc=0x5cf545
node_exporter[804]: github.com/godbus/dbus/v5.(*unixTransport).Read(0xc000515478?, {0xc0004f6000?, 0xc0001dc400?, 0xc000459698?})
node_exporter[804]:         <autogenerated>:1 +0x31 fp=0xc0005153c8 sp=0xc000515398 pc=0x97e7b1
node_exporter[804]: bufio.(*Reader).fill(0xc000515800)
node_exporter[804]:         /usr/local/go/src/bufio/bufio.go:106 +0xff fp=0xc000515400 sp=0xc0005153c8 pc=0x65a0bf
node_exporter[804]: bufio.(*Reader).ReadSlice(0xc000515800, 0x0?)
node_exporter[804]:         /usr/local/go/src/bufio/bufio.go:372 +0x2f fp=0xc000515450 sp=0xc000515400 pc=0x65acaf
node_exporter[804]: bufio.(*Reader).collectFragments(0x97f491?, 0x28?)
node_exporter[804]:         /usr/local/go/src/bufio/bufio.go:447 +0x74 fp=0xc000515510 sp=0xc000515450 pc=0x65b0f4
node_exporter[804]: bufio.(*Reader).ReadBytes(0x977a1c?, 0xd0?)
node_exporter[804]:         /usr/local/go/src/bufio/bufio.go:474 +0x1d fp=0xc000515590 sp=0xc000515510 pc=0x65b31d
node_exporter[804]: github.com/godbus/dbus/v5.authReadLine(0xce33a0?)
node_exporter[804]:         /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/auth.go:229 +0x25 fp=0xc000515600 sp=0xc000515590 pc=0x959045
node_exporter[804]: github.com/godbus/dbus/v5.(*Conn).Auth(0xc0002b1950, {0xc0005158c0?, 0x0?, 0x0?})
node_exporter[804]:         /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/auth.go:101 +0x7ca fp=0xc000515868 sp=0xc000515600 pc=0x95822a
node_exporter[804]: github.com/coreos/go-systemd/v22/dbus.dbusAuthConnection({0xce8f00?, 0xc000036048}, 0xc2a610)
node_exporter[804]:         /go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/dbus.go:241 +0x153 fp=0xc0005158f0 sp=0xc000515868 pc=0x9d4133
node_exporter[804]: github.com/coreos/go-systemd/v22/dbus.dbusAuthHelloConnection({0xce8f00?, 0xc000036048?}, 0x4160f0?)
node_exporter[804]:         /go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/dbus.go:251 +0x25 fp=0xc000515930 sp=0xc0005158f0 pc=0x9d4245
node_exporter[804]: github.com/coreos/go-systemd/v22/dbus.NewSystemConnectionContext.func1()
node_exporter[804]:         /go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/dbus.go:138 +0x28 fp=0xc000515958 sp=0xc000515930 pc=0x9d3aa8
node_exporter[804]: github.com/coreos/go-systemd/v22/dbus.NewConnection(0xc0005159f0)
node_exporter[804]:         /go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/dbus.go:195 +0x51 fp=0xc0005159e8 sp=0xc000515958 pc=0x9d3c31
node_exporter[804]: github.com/coreos/go-systemd/v22/dbus.NewSystemConnectionContext({0xce8f00?, 0xc000036048?})
node_exporter[804]:         /go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/dbus.go:137 +0x4d fp=0xc000515a18 sp=0xc0005159e8 pc=0x9d3a4d
node_exporter[804]: github.com/coreos/go-systemd/v22/dbus.NewWithContext({0xce8f00, 0xc000036048})
node_exporter[804]:         /go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/dbus.go:122 +0x27 fp=0xc000515a70 sp=0xc000515a18 pc=0x9d3967
node_exporter[804]: github.com/prometheus/node_exporter/collector.newSystemdDbusConn()
node_exporter[804]:         /app/collector/systemd_linux.go:442 +0x52 fp=0xc000515a90 sp=0xc000515a70 pc=0xa401d2
node_exporter[804]: github.com/prometheus/node_exporter/collector.(*systemdCollector).Update(0xc0001dd580, 0xc00037e960)
node_exporter[804]:         /app/collector/systemd_linux.go:180 +0xa5 fp=0xc000515e28 sp=0xc000515a90 pc=0xa3b0c5
node_exporter[804]: github.com/prometheus/node_exporter/collector.execute({0xbce40f, 0x7}, {0xce3e40, 0xc0001dd580}, 0xc000232a80?, {0xce3360, 0xc0001e65c0})
node_exporter[804]:         /app/collector/collector.go:161 +0x9c fp=0xc000515f60 sp=0xc000515e28 pc=0x9e481c
node_exporter[804]: github.com/prometheus/node_exporter/collector.NodeCollector.Collect.func1({0xbce40f?, 0xc00030ff90?}, {0xce3e40?, 0xc0001dd580?})
node_exporter[804]:         /app/collector/collector.go:152 +0x3d fp=0xc000515fb0 sp=0xc000515f60 pc=0x9e471d
node_exporter[804]: github.com/prometheus/node_exporter/collector.NodeCollector.Collect.func2()
node_exporter[804]:         /app/collector/collector.go:154 +0x36 fp=0xc000515fe0 sp=0xc000515fb0 pc=0x9e46b6
node_exporter[804]: runtime.goexit()
node_exporter[804]:         /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000515fe8 sp=0xc000515fe0 pc=0x46ec01
node_exporter[804]: created by github.com/prometheus/node_exporter/collector.NodeCollector.Collect
node_exporter[804]:         /app/collector/collector.go:151 +0xd0
node_exporter[804]: goroutine 294249 [IO wait]:
node_exporter[804]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000511a38 sp=0xc000511a18 pc=0x43ba76
node_exporter[804]: runtime.netpollblock(0x0?, 0x40482f?, 0x0?)
node_exporter[804]:         /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000511a70 sp=0xc000511a38 pc=0x4343d7
node_exporter[804]: internal/poll.runtime_pollWait(0x7fc485754d98, 0x72)
node_exporter[804]:         /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000511a90 sp=0xc000511a70 pc=0x469349
node_exporter[804]: internal/poll.(*pollDesc).wait(0xc0001dd500?, 0xc00042e4e0?, 0x0)
node_exporter[804]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000511ab8 sp=0xc000511a90 pc=0x4d9bf2
node_exporter[804]: internal/poll.(*pollDesc).waitRead(...)
node_exporter[804]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
node_exporter[804]: internal/poll.(*FD).ReadMsg(0xc0001dd500, {0xc00042e4e0, 0x10, 0x10}, {0xc0004be020, 0x1000, 0x1000}, 0xc000511c08?)
node_exporter[804]:         /usr/local/go/src/internal/poll/fd_unix.go:304 +0x3aa fp=0xc000511ba8 sp=0xc000511ab8 pc=0x4dc2ea
node_exporter[804]: net.(*netFD).readMsg(0xc0001dd500, {0xc00042e4e0?, 0x0?, 0x0?}, {0xc0004be020?, 0xc000511c78?, 0x4275c5?}, 0x0?)
node_exporter[804]:         /usr/local/go/src/net/fd_posix.go:78 +0x37 fp=0xc000511c30 sp=0xc000511ba8 pc=0x5bf597
node_exporter[804]: net.(*UnixConn).readMsg(0xc00055a900, {0xc00042e4e0?, 0xc000511d10?, 0x416d8b?}, {0xc0004be020?, 0x423127?, 0x7fc4acd9a200?})
node_exporter[804]:         /usr/local/go/src/net/unixsock_posix.go:115 +0x4f fp=0xc000511cc0 sp=0xc000511c30 pc=0x5de8cf
node_exporter[804]: net.(*UnixConn).ReadMsgUnix(0xc00055a900, {0xc00042e4e0?, 0x41b511?, 0x1168760?}, {0xc0004be020?, 0x40dcea?, 0xc000200ea0?})
node_exporter[804]:         /usr/local/go/src/net/unixsock.go:143 +0x3c fp=0xc000511d38 sp=0xc000511cc0 pc=0x5dcb5c
node_exporter[804]: github.com/godbus/dbus/v5.(*oobReader).Read(0xc0004be000, {0xc00042e4e0?, 0xc000511de8?, 0x40e0c7?})
node_exporter[804]:         /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/transport_unix.go:21 +0x45 fp=0xc000511db0 sp=0xc000511d38 pc=0x972f65
node_exporter[804]: io.ReadAtLeast({0xce3480, 0xc0004be000}, {0xc00042e4e0, 0x10, 0x10}, 0x10)
node_exporter[804]:         /usr/local/go/src/io/io.go:332 +0x9a fp=0xc000511df8 sp=0xc000511db0 pc=0x4b7bfa
node_exporter[804]: io.ReadFull(...)
node_exporter[804]:         /usr/local/go/src/io/io.go:351
node_exporter[804]: github.com/godbus/dbus/v5.(*unixTransport).ReadMessage(0xc000201230)
node_exporter[804]:         /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/transport_unix.go:91 +0x11e fp=0xc000511f58 sp=0xc000511df8 pc=0x97357e
node_exporter[804]: github.com/godbus/dbus/v5.(*Conn).inWorker(0xc0002b1ef0)
node_exporter[804]:         /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:390 +0x48 fp=0xc000511fc8 sp=0xc000511f58 pc=0x95b0e8
node_exporter[804]: github.com/godbus/dbus/v5.(*Conn).Auth.func1()
node_exporter[804]:         /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/auth.go:118 +0x26 fp=0xc000511fe0 sp=0xc000511fc8 pc=0x9584e6
node_exporter[804]: runtime.goexit()
node_exporter[804]:         /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000511fe8 sp=0xc000511fe0 pc=0x46ec01
node_exporter[804]: created by github.com/godbus/dbus/v5.(*Conn).Auth
node_exporter[804]:         /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/auth.go:118 +0x99c
node_exporter[804]: goroutine 294218 [chan receive]:
node_exporter[804]: runtime.gopark(0xc00030ff28?, 0x472e45?, 0x28?, 0xff?, 0x4fdf06?)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00030ff00 sp=0xc00030fee0 pc=0x43ba76
node_exporter[804]: runtime.chanrecv(0xc000558d20, 0x0, 0x1)
node_exporter[804]:         /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc00030ff90 sp=0xc00030ff00 pc=0x406f9d
node_exporter[804]: runtime.chanrecv1(0xc0000ab360?, 0xce3e40?)
node_exporter[804]:         /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc00030ffb8 sp=0xc00030ff90 pc=0x406a98
node_exporter[804]: github.com/godbus/dbus/v5.newConn.func1()
node_exporter[804]:         /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:303 +0x32 fp=0xc00030ffe0 sp=0xc00030ffb8 pc=0x95ac12
node_exporter[804]: runtime.goexit()
node_exporter[804]:         /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00030ffe8 sp=0xc00030ffe0 pc=0x46ec01
node_exporter[804]: created by github.com/godbus/dbus/v5.newConn
node_exporter[804]:         /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:302 +0x4f6
node_exporter[804]: goroutine 294157 [IO wait]:
node_exporter[804]: runtime.gopark(0xbdacc7?, 0xb?, 0x0?, 0x0?, 0x7?)
node_exporter[804]:         /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00030bda0 sp=0xc00030bd80 pc=0x43ba76
node_exporter[804]: runtime.netpollblock(0x4be225?, 0x40482f?, 0x0?)
node_exporter[804]:         /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc00030bdd8 sp=0xc00030bda0 pc=0x4343d7
node_exporter[804]: internal/poll.runtime_pollWait(0x7fc4859cb868, 0x72)
node_exporter[804]:         /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc00030bdf8 sp=0xc00030bdd8 pc=0x469349
node_exporter[804]: internal/poll.(*pollDesc).wait(0xc0001dd080?, 0xc0005d7181?, 0x0)
node_exporter[804]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc00030be20 sp=0xc00030bdf8 pc=0x4d9bf2
node_exporter[804]: internal/poll.(*pollDesc).waitRead(...)
node_exporter[804]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
node_exporter[804]: internal/poll.(*FD).Read(0xc0001dd080, {0xc0005d7181, 0x1, 0x1})
node_exporter[804]:         /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc00030beb8 sp=0xc00030be20 pc=0x4dafd9
node_exporter[804]: net.(*netFD).Read(0xc0001dd080, {0xc0005d7181?, 0x94b452?, 0xc000020ed0?})
node_exporter[804]:         /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc00030bf00 sp=0xc00030beb8 pc=0x5bf129
node_exporter[804]: net.(*conn).Read(0xc00055a8d8, {0xc0005d7181?, 0x9e471d?, 0xc0002d159b?})
node_exporter[804]:         /usr/local/go/src/net/net.go:183 +0x45 fp=0xc00030bf48 sp=0xc00030bf00 pc=0x5cf545
node_exporter[804]: net.(*TCPConn).Read(0xc0001e72c0?, {0xc0005d7181?, 0x100000000ce3360?, 0xc0001e7800?})
node_exporter[804]:         <autogenerated>:1 +0x29 fp=0xc00030bf78 sp=0xc00030bf48 pc=0x5e1ea9
node_exporter[804]: net/http.(*connReader).backgroundRead(0xc0005d7170)
node_exporter[804]:         /usr/local/go/src/net/http/server.go:674 +0x3f fp=0xc00030bfc8 sp=0xc00030bf78 pc=0x6ccd3f
node_exporter[804]: net/http.(*connReader).startBackgroundRead.func2()
node_exporter[804]:         /usr/local/go/src/net/http/server.go:670 +0x26 fp=0xc00030bfe0 sp=0xc00030bfc8 pc=0x6ccc66
node_exporter[804]: runtime.goexit()
node_exporter[804]:         /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00030bfe8 sp=0xc00030bfe0 pc=0x46ec01
node_exporter[804]: created by net/http.(*connReader).startBackgroundRead
node_exporter[804]:         /usr/local/go/src/net/http/server.go:670 +0xca

Are you running node_exporter in Docker?

No.

What did you do that produced an error?

Running node_exporter using a systemd service.

What did you expect to see?

Working fine.

What did you see instead?

fatal error: checkdead: runnable g

mguegan · 2023-06-02T06:24:34Z

Just a comment to add that I had a crash on another host but without federation.

SuperQ · 2023-06-02T08:29:21Z

One data point that may be useful, some graphs of max without (collector) (node_scrape_collector_duration_seconds) for the crashed instances.

mguegan · 2023-06-02T08:39:24Z

discordianfish · 2023-06-02T12:27:22Z

@SuperQ Should we revert the exporter-toolkit bump? Seems to be an issue for many users..

SuperQ · 2023-06-02T12:40:40Z

@discordianfish I can't see anything in the exporter-toolkit changes that would cause this.

discordianfish · 2023-06-02T13:23:09Z

Yeah fair enough, maybe it's the go version? I'd try rebuilding with go 1.19 and see if the problem still happens

dswarbrick · 2023-06-02T13:31:50Z

I'm also starting to lean more towards it being a runtime bug. I don't see any recent changes in exporter-toolkit or golang.org/x/sync that would provoke this.

I'm reading a few reports of checkdead firing erroneously on single-core instances. It would be interesting to know if the affected systems in this issue are all single-core.

My search led me to these so far: https://go-review.googlesource.com/c/go/+/487316 and https://go-review.googlesource.com/c/go/+/487375

mguegan · 2023-06-02T13:47:25Z

I have multi-core VMs impacted. I had also crashes on OpenBSD systems.

discordianfish · 2023-06-02T13:48:49Z

The bug report for the fix in go is here: golang/go#59600
Could some verify if running with --runtime.gomaxprocs=2 fixes it?

mguegan · 2023-06-02T19:47:00Z

If nobody can't do it before me, I could prepare a test with a bunch of impacted exporters on Monday.

aminvakil · 2023-06-03T09:08:17Z

One data point that may be useful, some graphs of max without (collector) (node_scrape_collector_duration_seconds) for the crashed instances.

mguegan · 2023-06-05T07:04:21Z

I have deployed the exporter with --runtime.gomaxprocs=2 in a batch of 140 hosts including OpenBSD and Linux VMs.
🤞
EDIT: no crash after 6h runtime

mguegan · 2023-06-06T07:45:43Z

Still no crash after 24h period.

SuperQ · 2023-06-07T10:34:40Z

Looks like 1.20.5 was released without the possible bugfix for this.

lcoffe-botify · 2023-06-12T18:16:20Z

You might want to get a coredump and search for the race condition described in golang/go#59600

If I can help in any way (and if you can share the coredump publicly), I can have a look at it too

ia-mfriegang · 2023-06-26T18:32:41Z

I was running into this issue on a few hosts and adding the flag --runtime.gomaxprocs=2 seemed to fix it for me as well

SuperQ · 2023-06-26T18:37:56Z

Upstream Go has backported the fix into the 1.20 branch. The next patch release of Go 1.20 should also fix the issue. When that is released we can cut a bugfix build of the node_exporter.

SuperQ · 2023-07-11T18:55:59Z

Go 1.20.6 has been released if anyone wants to manually build and test it. A new release here will take some time.

SuperQ · 2023-07-17T12:43:25Z

Please test the latest release:
https://github.com/prometheus/node_exporter/releases/tag/v1.6.1

andrgras · 2023-07-18T05:41:57Z

Installed 1.6.1 on all Linux machines, so far stable no crashes

discordianfish · 2023-07-31T11:59:35Z

I'll consider this as fixed then. We can re-open if it can be reproduced with 1.6.1

discordianfish mentioned this issue Jul 1, 2023

Go panic: fatal error: checkdead: runnable g #2730

Closed

SuperQ mentioned this issue Jul 17, 2023

Release v1.6.1 #2747

Merged

discordianfish closed this as completed Jul 31, 2023

fatal error: all goroutines are asleep - deadlock! #2705

fatal error: all goroutines are asleep - deadlock! #2705

Comments

andrgras commented May 30, 2023 • edited Loading

Host operating system: output of uname -a

node_exporter version: output of node_exporter --version

node_exporter command line flags

node_exporter log output

Are you running node_exporter in Docker?

What did you do that produced an error?

What did you expect to see?

What did you see instead?

SuperQ commented May 30, 2023

andrgras commented May 30, 2023

mguegan commented May 31, 2023

Host operating system: output of uname -a

node_exporter version: output of node_exporter --version

node_exporter command line flags

node_exporter log output

Are you running node_exporter in Docker?

What did you do that produced an error?

SuperQ commented May 31, 2023

SuperQ commented May 31, 2023

andrgras commented May 31, 2023

dswarbrick commented May 31, 2023 • edited Loading

SuperQ commented May 31, 2023

mguegan commented Jun 1, 2023

mguegan commented Jun 1, 2023

SuperQ commented Jun 1, 2023

mguegan commented Jun 1, 2023 • edited Loading

HinShell commented Jun 1, 2023

Host operating system: output of uname -a

node_exporter version: output of node_exporter --version

node_exporter command line flags

node_exporter log output

Are you running node_exporter in Docker?

What did you do that produced an error?

What did you expect to see?

What did you see instead?

aminvakil commented Jun 1, 2023

Host operating system: output of uname -a

node_exporter version: output of node_exporter --version

node_exporter command line flags

node_exporter log output

Are you running node_exporter in Docker?

What did you do that produced an error?

What did you expect to see?

What did you see instead?

mguegan commented Jun 2, 2023

SuperQ commented Jun 2, 2023

mguegan commented Jun 2, 2023

discordianfish commented Jun 2, 2023

SuperQ commented Jun 2, 2023

discordianfish commented Jun 2, 2023

dswarbrick commented Jun 2, 2023 • edited Loading

mguegan commented Jun 2, 2023

discordianfish commented Jun 2, 2023

mguegan commented Jun 2, 2023 • edited Loading

aminvakil commented Jun 3, 2023 • edited Loading

mguegan commented Jun 5, 2023 • edited Loading

mguegan commented Jun 6, 2023

SuperQ commented Jun 7, 2023

lcoffe-botify commented Jun 12, 2023

ia-mfriegang commented Jun 26, 2023

SuperQ commented Jun 26, 2023 • edited Loading

SuperQ commented Jul 11, 2023

SuperQ commented Jul 17, 2023

andrgras commented Jul 18, 2023 • edited Loading

discordianfish commented Jul 31, 2023

andrgras commented May 30, 2023 •

edited

Loading

Host operating system: output of `uname -a`

node_exporter version: output of `node_exporter --version`

Host operating system: output of `uname -a`

node_exporter version: output of `node_exporter --version`

dswarbrick commented May 31, 2023 •

edited

Loading

mguegan commented Jun 1, 2023 •

edited

Loading

Host operating system: output of `uname -a`

node_exporter version: output of `node_exporter --version`

Host operating system: output of `uname -a`

node_exporter version: output of `node_exporter --version`

dswarbrick commented Jun 2, 2023 •

edited

Loading

mguegan commented Jun 2, 2023 •

edited

Loading

aminvakil commented Jun 3, 2023 •

edited

Loading

mguegan commented Jun 5, 2023 •

edited

Loading

SuperQ commented Jun 26, 2023 •

edited

Loading

andrgras commented Jul 18, 2023 •

edited

Loading