Stuck at 13107517 with header not found

[splix]

System information

Erigon version: erigon version 2021.10.1-alpha

OS & Version: Linux

Commit hash : tag v2021.10.01

Backtrace

erigon[1787100]: [INFO] [10-04|00:44:50.814] [4/18 Bodies] Processing bodies...       from=13107517 to=13349505
erigon[1787100]: [EROR] [10-04|00:44:50.814] Staged Sync                              error="[4/18 Bodies] request more bodies: header not found: %!w(<nil>), blockNum=13107518, trace=[github.com/ledgerwatch/erigon/turbo/stages/bodydownload.(*BodyDownload).RequestMoreBodies github.com/ledgerwatch/erigon/eth/stagedsync.BodiesForward github.com/ledgerwatch/erigon/eth/stagedsync.DefaultStages.func10 github.com/ledgerwatch/erigon/eth/stagedsync.(*Sync).runStage github.com/ledgerwatch/erigon/eth/stagedsync.(*Sync).Run github.com/ledgerwatch/erigon/turbo/stages.StageLoopStep github.com/ledgerwatch/erigon/turbo/stages.StageLoop]"

Tried to restart multiple times, still stuck with block 13107517.

When I request the block details through JSON RPC it returns a correct hash. And returns null for blocks 13107518+

[AskAlexSharov]

Hi, please post output from:

pkill -SIGUSR1 erigon

[splix]

Oct 04 01:02:02 xhost erigon[1788474]: goroutine 72 [running]:
Oct 04 01:02:02 xhost erigon[1788474]: runtime/pprof.writeGoroutineStacks(0x208f1e0, 0xc000134008, 0x490dec, 0xc0005d8058)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/pprof/pprof.go:693 +0x9f
Oct 04 01:02:02 xhost erigon[1788474]: runtime/pprof.writeGoroutine(0x208f1e0, 0xc000134008, 0x2, 0x2, 0x1010010eb40)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/pprof/pprof.go:682 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: runtime/pprof.(*Profile).WriteTo(0x2879d40, 0x208f1e0, 0xc000134008, 0x2, 0x2, 0x1)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/pprof/pprof.go:331 +0x3f2
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/internal/debug.ListenSignals(0x208d140, 0xc00015ec60)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/internal/debug/signal.go:40 +0x1c5
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/cmd/utils.StartNode
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/utils/cmd.go:62 +0xc5
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 1 [chan receive, 3 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/node.(*Node).Wait(...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/node/node.go:388
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/turbo/node.(*ErigonNode).Serve(0xc0001fca30, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/turbo/node/node.go:30 +0x85
Oct 04 01:02:02 xhost erigon[1788474]: main.runErigon(0xc0001622c0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/erigon/main.go:36 +0x2fe
Oct 04 01:02:02 xhost erigon[1788474]: github.com/urfave/cli.HandleAction(0x13f0900, 0x1f30058, 0xc0001622c0, 0xc0001622c0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/urfave/cli@v1.22.5/app.go:526 +0x59
Oct 04 01:02:02 xhost erigon[1788474]: github.com/urfave/cli.(*App).Run(0xc00015ce00, 0xc000138000, 0x8, 0x8, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/urfave/cli@v1.22.5/app.go:286 +0x675
Oct 04 01:02:02 xhost erigon[1788474]: main.main()
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/erigon/main.go:18 +0xa9
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 19 [sleep]:
Oct 04 01:02:02 xhost erigon[1788474]: time.Sleep(0x22ecb25c00)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/time.go:193 +0xd2
Oct 04 01:02:02 xhost erigon[1788474]: github.com/VictoriaMetrics/metrics.summariesSwapCron(0x45d964b800)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/VictoriaMetrics/metrics@v1.18.0/summary.go:237 +0x1d7
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/VictoriaMetrics/metrics.registerSummaryLocked
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/VictoriaMetrics/metrics@v1.18.0/summary.go:211 +0x13b
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 50 [sync.Cond.Wait, 3 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: sync.runtime_notifyListWait(0xc0002080d8, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/sema.go:513 +0xf8
Oct 04 01:02:02 xhost erigon[1788474]: sync.(*Cond).Wait(0xc0002080c8)
Oct 04 01:02:02 xhost erigon[1788474]:         sync/cond.go:56 +0x99
Oct 04 01:02:02 xhost erigon[1788474]: github.com/anacrolix/stm.(*Tx).wait(0xc000208090)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/anacrolix/stm@v0.2.1-0.20201002073511-c35a2c748c6a/tx.go:71 +0xd3
Oct 04 01:02:02 xhost erigon[1788474]: github.com/anacrolix/stm.Atomically(0xc000248100, 0xc000248100, 0x2aefe60)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/anacrolix/stm@v0.2.1-0.20201002073511-c35a2c748c6a/funcs.go:80 +0x193
Oct 04 01:02:02 xhost erigon[1788474]: github.com/anacrolix/stm/rate.(*Limiter).tokenGenerator(0xc00061c020, 0x2625a00)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/anacrolix/stm@v0.2.1-0.20201002073511-c35a2c748c6a/rate/ratelimit.go:59 +0x193
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/anacrolix/stm/rate.NewLimiter
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/anacrolix/stm@v0.2.1-0.20201002073511-c35a2c748c6a/rate/ratelimit.go:45 +0x27f
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 38 [chan receive, 3 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/anacrolix/dht/v2.dnsResolverRefresher()
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/anacrolix/dht/v2@v2.8.0/dht.go:103 +0x7a
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/anacrolix/dht/v2.init.0
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/anacrolix/dht/v2@v2.8.0/dht.go:109 +0x35
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 7 [chan receive]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/metrics.(*meterArbiter).tick(0x2aee3c0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/metrics/meter.go:295 +0x9e
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/metrics.NewMeterForced
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/metrics/meter.go:73 +0x105
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 8 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/consensus/ethash.(*remoteSealer).loop(0xc000724f00)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/consensus/ethash/sealer.go:160 +0x20c
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/consensus/ethash.startRemoteSealer
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/consensus/ethash/sealer.go:144 +0x2b2
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 9 [select, 1 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/eth.New.func6(0xc0000aa820)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/eth/backend.go:345 +0x34c
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/eth.New
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/eth/backend.go:338 +0x3c85
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 12 [IO wait, 3 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.runtime_pollWait(0x7f41169d9eb8, 0x72, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/netpoll.go:222 +0x55
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).wait(0xc00003ca18, 0x72, 0x0, 0x0, 0x163596b)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:87 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).waitRead(...)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:92
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*FD).Accept(0xc00003ca00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_unix.go:401 +0x212
Oct 04 01:02:02 xhost erigon[1788474]: net.(*netFD).accept(0xc00003ca00, 0x37e11d600, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/fd_unix.go:172 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: net.(*TCPListener).accept(0xc0001284b0, 0xc005037e58, 0xc005037e60, 0x28)
Oct 04 01:02:02 xhost erigon[1788474]:         net/tcpsock_posix.go:139 +0x32
Oct 04 01:02:02 xhost erigon[1788474]: net.(*TCPListener).Accept(0xc0001284b0, 0x1f2ebf0, 0xc000340e00, 0x20a6f90, 0xc0001284b0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/tcpsock.go:261 +0x65
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc.(*Server).Serve(0xc000340e00, 0x20a6f90, 0xc0001284b0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:786 +0x27f
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/ethdb/privateapi.StartGrpc.func1(0xc000340e00, 0x20a6f90, 0xc0001284b0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/ethdb/privateapi/all.go:35 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/ethdb/privateapi.StartGrpc
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/ethdb/privateapi/all.go:34 +0x35f
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 13 [select, 1 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/turbo/stages/txpropagate.BroadcastPendingTxsToNetwork(0x20abf98, 0xc00003f500, 0xc000193b00, 0xc002e46d50, 0xc0007250e0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/turbo/stages/txpropagate/deprecated.go:38 +0x5d3
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/eth.New
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/eth/backend.go:464 +0x2c87
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 14 [select, 3 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/eth.New.func9(0xc0000aa820, 0x20abf98, 0xc00003f500)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/eth/backend.go:471 +0x12e
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/eth.New
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/eth/backend.go:465 +0x2cc6
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 15 [select, 3 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/eth.New.func12(0xc0000aa820, 0x20bbfb0, 0xc00022e380)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/eth/backend.go:522 +0x105
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/eth.New
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/eth/backend.go:519 +0x20f4
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 16 [chan receive]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon-lib/direct.(*SentryReceiveClientDirect).Recv(0xc0059a2360, 0xc00003f500, 0xc0042523c0, 0x20bf108)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/direct/sentry_client.go:223 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.RecvMessage(0x20abf98, 0xc00003f500, 0x20bf108, 0xc00000e4c8, 0xc01b560f10, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/downloader.go:298 +0x595
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.RecvMessageLoop(0x20abf98, 0xc00003f500, 0x20bf108, 0xc00000e4c8, 0xc0007250e0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/downloader.go:253 +0x5c9
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/eth.(*Ethereum).Start.func1(0xc0000aa820, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/eth/backend.go:767 +0x86
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/eth.(*Ethereum).Start
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/eth/backend.go:766 +0x65
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 66 [chan receive, 2 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon-lib/direct.(*SentryReceiveClientDirect).Recv(0xc005f0bb00, 0xc00003f500, 0xc001b66000, 0x20bf108)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/direct/sentry_client.go:223 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.RecvUploadMessage(0x20abf98, 0xc00003f500, 0x20bf108, 0xc00000e4c8, 0xc007af5f10, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/downloader.go:108 +0x3f5
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.RecvUploadMessageLoop(0x20abf98, 0xc00003f500, 0x20bf108, 0xc00000e4c8, 0xc0007250e0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/downloader.go:73 +0x5b6
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/eth.(*Ethereum).Start.func2(0xc0000aa820, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/eth/backend.go:770 +0x86
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/eth.(*Ethereum).Start
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/eth/backend.go:769 +0x94
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 67 [chan receive]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon-lib/direct.(*SentryReceiveClientDirect).Recv(0xc0059a2240, 0xc00003f500, 0xc0042526e0, 0x20bf108)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/direct/sentry_client.go:223 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.RecvUploadHeadersMessage(0x20abf98, 0xc00003f500, 0x20bf108, 0xc00000e4c8, 0xc006e87f10, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/downloader.go:197 +0x342
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.RecvUploadHeadersMessageLoop(0x20abf98, 0xc00003f500, 0x20bf108, 0xc00000e4c8, 0xc0007250e0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/downloader.go:164 +0x5b6
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/eth.(*Ethereum).Start.func3(0xc0000aa820, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/eth/backend.go:773 +0x86
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/eth.(*Ethereum).Start
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/eth/backend.go:772 +0xc5
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 68 [chan receive]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon-lib/direct.(*SentryReceiveClientDirect).Recv(0xc005c7fd10, 0xc00003f500, 0xc004252690, 0x20bf108)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/direct/sentry_client.go:223 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.RecvMessage(0x20abf98, 0xc00003f500, 0x20bf108, 0xc00000e4f8, 0xc0064f7f10, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/downloader.go:298 +0x595
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.RecvMessageLoop(0x20abf98, 0xc00003f500, 0x20bf108, 0xc00000e4f8, 0xc0007250e0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/downloader.go:253 +0x5c9
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/eth.(*Ethereum).Start.func1(0xc0000aa820, 0x1)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/eth/backend.go:767 +0x86
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/eth.(*Ethereum).Start
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/eth/backend.go:766 +0x65
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 69 [chan receive, 3 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon-lib/direct.(*SentryReceiveClientDirect).Recv(0xc0042c63f0, 0x20abf98, 0xc0042c83c0, 0xc0042c8400)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/direct/sentry_client.go:223 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.RecvUploadMessage(0x20abf98, 0xc00003f500, 0x20bf108, 0xc00000e4f8, 0xc007af7f10, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/downloader.go:108 +0x395
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.RecvUploadMessageLoop(0x20abf98, 0xc00003f500, 0x20bf108, 0xc00000e4f8, 0xc0007250e0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/downloader.go:73 +0x5b6
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/eth.(*Ethereum).Start.func2(0xc0000aa820, 0x1)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/eth/backend.go:770 +0x86
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/eth.(*Ethereum).Start
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/eth/backend.go:769 +0x94
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 70 [chan receive]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon-lib/direct.(*SentryReceiveClientDirect).Recv(0xc0042c6540, 0xc00003f500, 0xc007c84000, 0x20bf108)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/direct/sentry_client.go:223 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.RecvUploadHeadersMessage(0x20abf98, 0xc00003f500, 0x20bf108, 0xc00000e4f8, 0xc006490f10, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/downloader.go:197 +0x342
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.RecvUploadHeadersMessageLoop(0x20abf98, 0xc00003f500, 0x20bf108, 0xc00000e4f8, 0xc0007250e0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/downloader.go:164 +0x5b6
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/eth.(*Ethereum).Start.func3(0xc0000aa820, 0x1)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/eth/backend.go:773 +0x86
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/eth.(*Ethereum).Start
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/eth/backend.go:772 +0xc5
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 71 [runnable]:
Oct 04 01:02:02 xhost erigon[1788474]: golang.org/x/crypto/sha3.keccakF1600(0xc0135b81c0)
Oct 04 01:02:02 xhost erigon[1788474]:         golang.org/x/crypto@v0.0.0-20210616213533-5ff15b29337e/sha3/keccakf_amd64.s:324 +0x4c96
Oct 04 01:02:02 xhost erigon[1788474]: golang.org/x/crypto/sha3.(*state).Write(0xc0135b81c0, 0xc00726e408, 0x215, 0x2f8, 0x1575540, 0xc0163ea000, 0x16)
Oct 04 01:02:02 xhost erigon[1788474]:         golang.org/x/crypto@v0.0.0-20210616213533-5ff15b29337e/sha3/sha3.go:140 +0x13f
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/rlp.(*encbuf).toWriter(0xc013247260, 0x7f4114640050, 0xc0135b81c0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/rlp/encode.go:277 +0x230
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/rlp.Encode(0x7f4114640050, 0xc0135b81c0, 0x1575540, 0xc0163ea000, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/rlp/encode.go:75 +0x1a5
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/core/types.rlpHash(0x1575540, 0xc0163ea000, 0x4d165c, 0x2876d80, 0xc000580c00, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/core/types/hashing.go:174 +0x16d
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/core/types.(*Header).Hash(...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/core/types/block.go:546
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/turbo/stages/headerdownload.(*HeaderDownload).addHeaderAsLink(0xc000724ff0, 0xc0163ea000, 0x201, 0x1575540)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/turbo/stages/headerdownload/header_algos.go:679 +0x69
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/turbo/stages/headerdownload.(*HeaderDownload).RecoverFromDb.func1(0x20bd478, 0xc01289a060, 0xc000132040, 0x20bd478)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/turbo/stages/headerdownload/header_algos.go:475 +0xdf
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon-lib/kv/mdbx.(*MdbxKV).View(0xc00003c580, 0x20abfd0, 0xc000132040, 0xc0002ac050, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/kv/mdbx/kv_mdbx.go:599 +0x11e
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/turbo/stages/headerdownload.(*HeaderDownload).RecoverFromDb(0xc000724ff0, 0x7f41144ccbd8, 0xc00003c580, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/turbo/stages/headerdownload/header_algos.go:461 +0x15e
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/turbo/stages.StageLoop(0x20abf98, 0xc00003f500, 0x20b4ad8, 0xc00003c580, 0xc00015f080, 0xc000724ff0, 0xc0009da5e0, 0xc0001189f0, 0xc0000f40c0, 0xdf8475800)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/turbo/stages/stageloop.go:62 +0x2d4
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/eth.(*Ethereum).Start
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/eth/backend.go:777 +0x1b4
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 74 [syscall]:
Oct 04 01:02:02 xhost erigon[1788474]: os/signal.signal_recv(0x20a1130)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/sigqueue.go:168 +0xa5
Oct 04 01:02:02 xhost erigon[1788474]: os/signal.loop()
Oct 04 01:02:02 xhost erigon[1788474]:         os/signal/signal_unix.go:23 +0x25
Oct 04 01:02:02 xhost erigon[1788474]: created by os/signal.Notify.func1.1
Oct 04 01:02:02 xhost erigon[1788474]:         os/signal/signal.go:151 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 115 [select, 3 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.(*SentryServerImpl).Messages(0xc000191560, 0xc0042c8400, 0x20b70b8, 0xc0042c63c0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:950 +0x27d
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon-lib/direct.(*SentryClientDirect).Messages.func1(0xc00000e4f8, 0xc0042c8400, 0xc0042c63c0, 0xc0042aa360)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/direct/sentry_client.go:264 +0x58
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon-lib/direct.(*SentryClientDirect).Messages
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/direct/sentry_client.go:263 +0x125
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 83 [select, 3 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/nat.Map(0x20ac858, 0xc00003eac0, 0xc00645a660, 0x1631eb3, 0x3, 0x7660, 0x7660, 0x163b7a3, 0xc)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/nat/nat.go:114 +0x42a
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Server).setupListening.func1(0xc0000b42c0, 0xc006458fc0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:697 +0xad
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Server).setupListening
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:695 +0x245
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 84 [IO wait, 3 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.runtime_pollWait(0x7f41169d9dd0, 0x72, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/netpoll.go:222 +0x55
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).wait(0xc000138798, 0x72, 0x0, 0x0, 0x163596b)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:87 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).waitRead(...)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:92
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*FD).Accept(0xc000138780, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_unix.go:401 +0x212
Oct 04 01:02:02 xhost erigon[1788474]: net.(*netFD).accept(0xc000138780, 0xc0005e2058, 0xa2ac20, 0xc000100000)
Oct 04 01:02:02 xhost erigon[1788474]:         net/fd_unix.go:172 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: net.(*TCPListener).accept(0xc00645e798, 0x44c28b, 0xc0005e2000, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/tcpsock_posix.go:139 +0x32
Oct 04 01:02:02 xhost erigon[1788474]: net.(*TCPListener).Accept(0xc00645e798, 0x0, 0xc0005e2000, 0xc0005de000, 0x2)
Oct 04 01:02:02 xhost erigon[1788474]:         net/tcpsock.go:261 +0x65
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Server).listenLoop(0xc0000b42c0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:895 +0x25c
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Server).setupListening
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:704 +0x16d
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 85 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/enode.(*FairMix).runSource(0xc0064543c0, 0xc00645aa20, 0xc00645d100)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/enode/iter.go:284 +0x192
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p/enode.(*FairMix).AddSource
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/enode/iter.go:182 +0x153
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 86 [select, 3 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/nat.Map(0x20ac858, 0xc00003eac0, 0xc00645a660, 0x1631ec5, 0x3, 0x7660, 0x7660, 0x164311b, 0x12)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/nat/nat.go:114 +0x42a
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Server).setupDiscovery.func1(0xc0000b42c0, 0xc006459080)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:585 +0xad
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Server).setupDiscovery
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:583 +0x825
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 116 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.(*SentryServerImpl).Messages(0xc000191560, 0xc0042c8480, 0x20b70b8, 0xc0042c6510, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:950 +0x27d
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon-lib/direct.(*SentryClientDirect).Messages.func1(0xc00000e4f8, 0xc0042c8480, 0xc0042c6510, 0xc0042aa540)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/direct/sentry_client.go:264 +0x58
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon-lib/direct.(*SentryClientDirect).Messages
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/direct/sentry_client.go:263 +0x125
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 708 [IO wait]:
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.runtime_pollWait(0x7f41169d9038, 0x72, 0xffffffffffffffff)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/netpoll.go:222 +0x55
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).wait(0xc007a8f598, 0x72, 0x30000, 0x30000, 0xffffffffffffffff)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:87 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).waitRead(...)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:92
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*FD).Read(0xc007a8f580, 0xc002e52000, 0x30000, 0x30000, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_unix.go:166 +0x1d5
Oct 04 01:02:02 xhost erigon[1788474]: net.(*netFD).Read(0xc007a8f580, 0xc002e52000, 0x30000, 0x30000, 0xc019eca700, 0xc0135b9430, 0x20)
Oct 04 01:02:02 xhost erigon[1788474]:         net/fd_posix.go:55 +0x4f
Oct 04 01:02:02 xhost erigon[1788474]: net.(*conn).Read(0xc0005ea1d0, 0xc002e52000, 0x30000, 0x30000, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/net.go:183 +0x91
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*meteredConn).Read(0xc0077d9300, 0xc002e52000, 0x30000, 0x30000, 0x490dec, 0xc0078ddc18, 0xa1922a)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/metrics.go:63 +0x51
Oct 04 01:02:02 xhost erigon[1788474]: io.ReadAtLeast(0x7f41145b89a8, 0xc0077d9300, 0xc002e52000, 0x30000, 0x30000, 0x20, 0x1448120, 0x7f41169d9118, 0x78)
Oct 04 01:02:02 xhost erigon[1788474]:         io/io.go:328 +0x87
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*readBuffer).read(0xc0075dbf00, 0x7f41145b89a8, 0xc0077d9300, 0x20, 0x45043f, 0xc000032000, 0x14820e0, 0x1552d40, 0x7f41145b89a8)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/buffer.go:64 +0x12e
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*sessionState).readFrame(0xc0075dbe00, 0x7f41145b89a8, 0xc0077d9300, 0x7f41145b89a8, 0xc0077d9300, 0x2aefe60, 0x72, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/rlpx.go:166 +0x105
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*Conn).Read(0xc007ab04b0, 0xc04eb28a057099e0, 0x31777cd383, 0x2aefe60, 0x0, 0x0, 0x2aefe60, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/rlpx.go:136 +0x98
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*rlpxTransport).ReadMsg(0xc0077f1a40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/transport.go:67 +0x154
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).readLoop(0xc007a1baa0, 0xc00757b080)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:298 +0xd1
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).run
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:233 +0xfb
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 87 [select, 3 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/enode.(*DB).expirer(0xc006454240)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/enode/nodedb.go:456 +0xdb
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p/enode.(*DB).ensureExpirer.func1
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/enode/nodedb.go:447 +0x3e
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 710 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*protoRW).ReadMsg(0xc00789b0e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:456 +0xcf
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.runPeer(0x20abf98, 0xc00003f500, 0xc0075736c0, 0x40, 0x42, 0x209f0b0, 0xc00789b0e0, 0xc00757b0e0, 0xc00644fe70, 0xc00644fe60, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:289 +0x1f3
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.NewSentryServer.func1(0xc007a1baa0, 0x209f0b0, 0xc00789b0e0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:496 +0x79d
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).startProtocols.func1(0xc007a1baa0, 0xc00789b0e0, 0x209f0b0, 0xc00789b0e0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:399 +0xb4
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).startProtocols
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:396 +0x205
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 24 [select, 3 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/nat.Map(0x20ac858, 0xc00003eac0, 0xc0050406c0, 0x1631eb3, 0x3, 0x7d05, 0x7d05, 0x163b7a3, 0xc)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/nat/nat.go:114 +0x42a
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Server).setupListening.func1(0xc005066000, 0xc00503cfc0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:697 +0xad
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Server).setupListening
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:695 +0x245
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 25 [IO wait]:
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.runtime_pollWait(0x7f41169d9c00, 0x72, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/netpoll.go:222 +0x55
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).wait(0xc0027c4698, 0x72, 0x0, 0x0, 0x163596b)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:87 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).waitRead(...)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:92
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*FD).Accept(0xc0027c4680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_unix.go:401 +0x212
Oct 04 01:02:02 xhost erigon[1788474]: net.(*netFD).accept(0xc0027c4680, 0xc00010ee98, 0xa2ac20, 0x26fd930)
Oct 04 01:02:02 xhost erigon[1788474]:         net/fd_unix.go:172 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: net.(*TCPListener).accept(0xc00062e858, 0x44c28b, 0xc00010ee40, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/tcpsock_posix.go:139 +0x32
Oct 04 01:02:02 xhost erigon[1788474]: net.(*TCPListener).Accept(0xc00062e858, 0x0, 0xc005066000, 0x20b9f48, 0xc0075ec0f0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/tcpsock.go:261 +0x65
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Server).listenLoop(0xc005066000)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:895 +0x25c
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Server).setupListening
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:704 +0x16d
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 26 [select, 2 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/enode.(*FairMix).runSource(0xc005044240, 0xc005040a80, 0xc005043100)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/enode/iter.go:284 +0x192
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p/enode.(*FairMix).AddSource
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/enode/iter.go:182 +0x153
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 27 [select, 3 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/nat.Map(0x20ac858, 0xc00003eac0, 0xc0050406c0, 0x1631ec5, 0x3, 0x7d05, 0x7d05, 0x164311b, 0x12)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/nat/nat.go:114 +0x42a
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Server).setupDiscovery.func1(0xc005066000, 0xc00503d080)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:585 +0xad
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Server).setupDiscovery
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:583 +0x825
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 100 [select, 3 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/enode.(*DB).expirer(0xc005044180)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/enode/nodedb.go:456 +0xdb
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p/enode.(*DB).ensureExpirer.func1
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/enode/nodedb.go:447 +0x3e
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 88 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/discover.(*Table).loop(0xc0000a03c0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/discover/table.go:238 +0x309
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p/discover.ListenV4
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:152 +0x2d9
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 89 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).loop(0xc0000db3b0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:461 +0x517
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p/discover.ListenV4
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:155 +0x316
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 90 [IO wait]:
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.runtime_pollWait(0x7f41169d9ce8, 0x72, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/netpoll.go:222 +0x55
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).wait(0xc000138818, 0x72, 0x500, 0x500, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:87 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).waitRead(...)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:92
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*FD).ReadFrom(0xc000138800, 0xc0050f2000, 0x500, 0x500, 0x0, 0x0, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_unix.go:222 +0x1e6
Oct 04 01:02:02 xhost erigon[1788474]: net.(*netFD).readFrom(0xc000138800, 0xc0050f2000, 0x500, 0x500, 0xf8449a57372d28d2, 0x7018b911797d42ea, 0x4cc310733723898a, 0x4e468a13efcd32ef, 0xb5910b1e70ed7783)
Oct 04 01:02:02 xhost erigon[1788474]:         net/fd_posix.go:61 +0x5b
Oct 04 01:02:02 xhost erigon[1788474]: net.(*UDPConn).readFrom(0xc000010048, 0xc0050f2000, 0x500, 0x500, 0x0, 0xc01ba36120, 0xc000142040, 0x209f100)
Oct 04 01:02:02 xhost erigon[1788474]:         net/udpsock_posix.go:47 +0x6a
Oct 04 01:02:02 xhost erigon[1788474]: net.(*UDPConn).ReadFromUDP(0xc000010048, 0xc0050f2000, 0x500, 0x500, 0x500, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/udpsock.go:106 +0x5d
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).readLoop(0xc0000db3b0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:540 +0x126
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p/discover.ListenV4
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:156 +0x345
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 91 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/enode.(*FairMix).runSource(0xc0064543c0, 0xc00645aa20, 0xc00645d6a0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/enode/iter.go:284 +0x192
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p/enode.(*FairMix).AddSource
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/enode/iter.go:182 +0x153
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 92 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*dialScheduler).readNodes(0xc005c78160, 0x20a63c0, 0xc0064543c0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/dial.go:334 +0x168
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.newDialScheduler
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/dial.go:188 +0x376
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 93 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*dialScheduler).loop(0xc005c78160, 0x20a63c0, 0xc0064543c0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/dial.go:252 +0x2b4
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.newDialScheduler
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/dial.go:189 +0x3b5
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 94 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Server).run(0xc0000b42c0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:747 +0x4d9
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Server).Start
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:505 +0x38c
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 95 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.(*SentryServerImpl).Messages(0xc000191560, 0xc000636bc0, 0x20b70b8, 0xc005c7fce0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:950 +0x27d
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon-lib/direct.(*SentryClientDirect).Messages.func1(0xc00000e4f8, 0xc000636bc0, 0xc005c7fce0, 0xc005cb8960)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/direct/sentry_client.go:264 +0x58
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon-lib/direct.(*SentryClientDirect).Messages
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/direct/sentry_client.go:263 +0x125
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 1820 [select, 2 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).run(0xc019086c00, 0x13b10e0, 0xc003126980, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:243 +0x265
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Server).runPeer(0xc005066000, 0xc019086c00)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:1074 +0x205
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Server).launchPeer
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:1056 +0xc5
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 379 [IO wait]:
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.runtime_pollWait(0x7f41169d94c0, 0x72, 0xffffffffffffffff)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/netpoll.go:222 +0x55
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).wait(0xc0072b5c98, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:87 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).waitRead(...)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:92
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*FD).Read(0xc0072b5c80, 0xc0072f4000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_unix.go:166 +0x1d5
Oct 04 01:02:02 xhost erigon[1788474]: net.(*netFD).Read(0xc0072b5c80, 0xc0072f4000, 0x8000, 0x8000, 0xc06a1e, 0x800000601, 0x480efc)
Oct 04 01:02:02 xhost erigon[1788474]:         net/fd_posix.go:55 +0x4f
Oct 04 01:02:02 xhost erigon[1788474]: net.(*conn).Read(0xc000134ca0, 0xc0072f4000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/net.go:183 +0x91
Oct 04 01:02:02 xhost erigon[1788474]: bufio.(*Reader).Read(0xc0072dcd80, 0xc007290118, 0x9, 0x9, 0xc0072bd438, 0x18, 0x7f413ee66878)
Oct 04 01:02:02 xhost erigon[1788474]:         bufio/bufio.go:227 +0x222
Oct 04 01:02:02 xhost erigon[1788474]: io.ReadAtLeast(0x208b3a0, 0xc0072dcd80, 0xc007290118, 0x9, 0x9, 0x9, 0x5d342238e4199, 0x27cd4649, 0x27cd4649072bd3e0)
Oct 04 01:02:02 xhost erigon[1788474]:         io/io.go:328 +0x87
Oct 04 01:02:02 xhost erigon[1788474]: io.ReadFull(...)
Oct 04 01:02:02 xhost erigon[1788474]:         io/io.go:347
Oct 04 01:02:02 xhost erigon[1788474]: golang.org/x/net/http2.readFrameHeader(0xc007290118, 0x9, 0x9, 0x208b3a0, 0xc0072dcd80, 0x0, 0x0, 0xc0061af380, 0x29381517ed)
Oct 04 01:02:02 xhost erigon[1788474]:         golang.org/x/net@v0.0.0-20210805182204-aaa1db679c0d/http2/frame.go:237 +0x89
Oct 04 01:02:02 xhost erigon[1788474]: golang.org/x/net/http2.(*Framer).ReadFrame(0xc0072900e0, 0xc01a694b70, 0x2aefe60, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         golang.org/x/net@v0.0.0-20210805182204-aaa1db679c0d/http2/frame.go:492 +0xa5
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0061af380, 0xc0072d5e00, 0x1f2ec30)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/internal/transport/http2_server.go:541 +0x9b
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc.(*Server).serveStreams(0xc000340e00, 0x20bb058, 0xc0061af380)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:925 +0x103
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc.(*Server).handleRawConn.func1(0xc000340e00, 0x20bb058, 0xc0061af380, 0xc0072e03b0, 0xe)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:875 +0x3f
Oct 04 01:02:02 xhost erigon[1788474]: created by google.golang.org/grpc.(*Server).handleRawConn
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:874 +0x5f4
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 388 [chan receive, 3 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/ethdb/privateapi.(*TxPoolServer).OnAdd(0xc002e41dc0, 0xc007370cf0, 0x20b7478, 0xc0070e78c0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/ethdb/privateapi/txpool.go:141 +0x2f9
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon-lib/gointerfaces/txpool._Txpool_OnAdd_Handler(0x15529c0, 0xc002e41dc0, 0x20b4e38, 0xc005c67440, 0x20, 0x20)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/gointerfaces/txpool/txpool_grpc.pb.go:289 +0x113
Oct 04 01:02:02 xhost erigon[1788474]: github.com/grpc-ecosystem/go-grpc-middleware/recovery.StreamServerInterceptor.func1(0x15529c0, 0xc002e41dc0, 0x20b4e38, 0xc005c67440, 0xc0073606d8, 0x1f2d2c8, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:51 +0xc2
Oct 04 01:02:02 xhost erigon[1788474]: github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x15529c0, 0xc002e41dc0, 0x20b4e38, 0xc005c67440, 0xc006e85c20, 0x454078)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:49 +0x5f
Oct 04 01:02:02 xhost erigon[1788474]: github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1(0x15529c0, 0xc002e41dc0, 0x20b4e38, 0xc005c67440, 0xc0073606d8, 0x1f2d2c8, 0x20ac040, 0xc007370cc0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:58 +0xd1
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc.(*Server).processStreamingRPC(0xc000340e00, 0x20bb058, 0xc0061af380, 0xc007168480, 0xc002e476b0, 0x2876460, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:1549 +0x6d8
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc.(*Server).handleStream(0xc000340e00, 0x20bb058, 0xc0061af380, 0xc007168480, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:1622 +0xca5
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0072e03e0, 0xc000340e00, 0x20bb058, 0xc0061af380, 0xc007168480)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:941 +0xab
Oct 04 01:02:02 xhost erigon[1788474]: created by google.golang.org/grpc.(*Server).serveStreams.func1
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:939 +0x1fd
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 101 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/discover.(*Table).loop(0xc00507c000)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/discover/table.go:238 +0x309
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p/discover.ListenV4
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:152 +0x2d9
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 102 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).loop(0xc005048ab0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:461 +0x517
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p/discover.ListenV4
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:155 +0x316
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 103 [IO wait]:
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.runtime_pollWait(0x7f41169d9a30, 0x72, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/netpoll.go:222 +0x55
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).wait(0xc0027c4718, 0x72, 0x500, 0x500, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:87 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).waitRead(...)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:92
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*FD).ReadFrom(0xc0027c4700, 0xc005d98000, 0x500, 0x500, 0x0, 0x0, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_unix.go:222 +0x1e6
Oct 04 01:02:02 xhost erigon[1788474]: net.(*netFD).readFrom(0xc0027c4700, 0xc005d98000, 0x500, 0x500, 0xe36b34c44afa36fb, 0x8069e29ddc8b8177, 0x6278e6df5e949641, 0x6859c398ab5e1baa, 0x77d4ff622c7ee2ef)
Oct 04 01:02:02 xhost erigon[1788474]:         net/fd_posix.go:61 +0x5b
Oct 04 01:02:02 xhost erigon[1788474]: net.(*UDPConn).readFrom(0xc00506a010, 0xc005d98000, 0x500, 0x500, 0x0, 0xc0038a82a0, 0xc000142040, 0x209f1a0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/udpsock_posix.go:47 +0x6a
Oct 04 01:02:02 xhost erigon[1788474]: net.(*UDPConn).ReadFromUDP(0xc00506a010, 0xc005d98000, 0x500, 0x500, 0x500, 0x208b760, 0xc0001fc880, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/udpsock.go:106 +0x5d
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).readLoop(0xc005048ab0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:540 +0x126
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p/discover.ListenV4
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:156 +0x345
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 104 [select, 2 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/enode.(*FairMix).runSource(0xc005044240, 0xc005040a80, 0xc003122400)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/enode/iter.go:284 +0x192
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p/enode.(*FairMix).AddSource
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/enode/iter.go:182 +0x153
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 105 [select, 2 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*dialScheduler).readNodes(0xc00313a2c0, 0x20a63c0, 0xc005044240)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/dial.go:334 +0x168
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.newDialScheduler
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/dial.go:188 +0x376
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 106 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*dialScheduler).loop(0xc00313a2c0, 0x20a63c0, 0xc005044240)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/dial.go:252 +0x2b4
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.newDialScheduler
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/dial.go:189 +0x3b5
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 107 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Server).run(0xc005066000)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:747 +0x4d9
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Server).Start
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:505 +0x38c
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 108 [select, 2 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.(*SentryServerImpl).Messages(0xc000191440, 0xc003124540, 0x20b70b8, 0xc005f0bad0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:950 +0x27d
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon-lib/direct.(*SentryClientDirect).Messages.func1(0xc00000e4c8, 0xc003124540, 0xc005f0bad0, 0xc005f44660)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/direct/sentry_client.go:264 +0x58
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon-lib/direct.(*SentryClientDirect).Messages
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/direct/sentry_client.go:263 +0x125
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 75 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.(*SentryServerImpl).Messages(0xc000191440, 0xc0059ac100, 0x20b70b8, 0xc0059a2210, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:950 +0x27d
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon-lib/direct.(*SentryClientDirect).Messages.func1(0xc00000e4c8, 0xc0059ac100, 0xc0059a2210, 0xc00599faa0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/direct/sentry_client.go:264 +0x58
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon-lib/direct.(*SentryClientDirect).Messages
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/direct/sentry_client.go:263 +0x125
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 378 [select, 3 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0061af380)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/internal/transport/http2_server.go:1061 +0x265
Oct 04 01:02:02 xhost erigon[1788474]: created by google.golang.org/grpc/internal/transport.NewServerTransport
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/internal/transport/http2_server.go:313 +0x11d5
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 76 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.(*SentryServerImpl).Messages(0xc000191440, 0xc0059ac180, 0x20b70b8, 0xc0059a2330, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:950 +0x27d
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon-lib/direct.(*SentryClientDirect).Messages.func1(0xc00000e4c8, 0xc0059ac180, 0xc0059a2330, 0xc00599fc80)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/direct/sentry_client.go:264 +0x58
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon-lib/direct.(*SentryClientDirect).Messages
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/direct/sentry_client.go:263 +0x125
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 2890 [select, 1 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).run(0xc010db0120, 0x13b10e0, 0xc0050c8100, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:243 +0x265
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Server).runPeer(0xc005066000, 0xc010db0120)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:1074 +0x205
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Server).launchPeer
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:1056 +0xc5
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 1481 [IO wait]:
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.runtime_pollWait(0x7f41169d92f0, 0x72, 0xffffffffffffffff)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/netpoll.go:222 +0x55
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).wait(0xc004388298, 0x72, 0x400, 0x400, 0xffffffffffffffff)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:87 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).waitRead(...)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:92
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*FD).Read(0xc004388280, 0xc000164c00, 0x400, 0x400, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_unix.go:166 +0x1d5
Oct 04 01:02:02 xhost erigon[1788474]: net.(*netFD).Read(0xc004388280, 0xc000164c00, 0x400, 0x400, 0xc005aeea80, 0xc00015d0b0, 0x20)
Oct 04 01:02:02 xhost erigon[1788474]:         net/fd_posix.go:55 +0x4f
Oct 04 01:02:02 xhost erigon[1788474]: net.(*conn).Read(0xc00062c058, 0xc000164c00, 0x400, 0x400, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/net.go:183 +0x91
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*meteredConn).Read(0xc0075ec0c0, 0xc000164c00, 0x400, 0x400, 0xc005aeea80, 0xc007af9b70, 0xa1922a)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/metrics.go:63 +0x51
Oct 04 01:02:02 xhost erigon[1788474]: io.ReadAtLeast(0x7f41145b89a8, 0xc0075ec0c0, 0xc000164c00, 0x400, 0x400, 0x20, 0x1448120, 0x7f41169d93d0, 0x5c)
Oct 04 01:02:02 xhost erigon[1788474]:         io/io.go:328 +0x87
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*readBuffer).read(0xc005b35b40, 0x7f41145b89a8, 0xc0075ec0c0, 0x20, 0x45043f, 0xc000032000, 0x14820e0, 0x1552d40, 0x7f41145b89a8)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/buffer.go:64 +0x12e
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*sessionState).readFrame(0xc005b35a40, 0x7f41145b89a8, 0xc0075ec0c0, 0x7f41145b89a8, 0xc0075ec0c0, 0x2aefe60, 0x72, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/rlpx.go:166 +0x105
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*Conn).Read(0xc004252190, 0xc04eb28783d3291e, 0x2f21d37ec6, 0x2aefe60, 0x0, 0x0, 0x2aefe60, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/rlpx.go:136 +0x98
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*rlpxTransport).ReadMsg(0xc002df6280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/transport.go:67 +0x154
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).readLoop(0xc00324aa20, 0xc00324aae0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:298 +0xd1
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).run
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:233 +0xfb
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 387 [select, 3 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/ethdb/privateapi.(*EthBackendServer).Subscribe(0xc002e46fc0, 0xc0072d5ec0, 0x20b71a8, 0xc0072aec00, 0xc002e46fc0, 0x3200000000000000)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/ethdb/privateapi/ethbackend.go:108 +0x1b1
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon-lib/gointerfaces/remote._ETHBACKEND_Subscribe_Handler(0x15528e0, 0xc002e46fc0, 0x20b4e38, 0xc0000ecc00, 0x20, 0x20)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/gointerfaces/remote/ethbackend_grpc.pb.go:297 +0x113
Oct 04 01:02:02 xhost erigon[1788474]: github.com/grpc-ecosystem/go-grpc-middleware/recovery.StreamServerInterceptor.func1(0x15528e0, 0xc002e46fc0, 0x20b4e38, 0xc0000ecc00, 0xc0072b6e28, 0x1f2d1d0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:51 +0xc2
Oct 04 01:02:02 xhost erigon[1788474]: github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x15528e0, 0xc002e46fc0, 0x20b4e38, 0xc0000ecc00, 0xc006496c20, 0x454078)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:49 +0x5f
Oct 04 01:02:02 xhost erigon[1788474]: github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1(0x15528e0, 0xc002e46fc0, 0x20b4e38, 0xc0000ecc00, 0xc0072b6e28, 0x1f2d1d0, 0x20ac040, 0xc0072d5e90)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:58 +0xd1
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc.(*Server).processStreamingRPC(0xc000340e00, 0x20bb058, 0xc0061af380, 0xc007168360, 0xc002e47620, 0x2876440, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:1549 +0x6d8
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc.(*Server).handleStream(0xc000340e00, 0x20bb058, 0xc0061af380, 0xc007168360, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:1622 +0xca5
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0072e03e0, 0xc000340e00, 0x20bb058, 0xc0061af380, 0xc007168360)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:941 +0xab
Oct 04 01:02:02 xhost erigon[1788474]: created by google.golang.org/grpc.(*Server).serveStreams.func1
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:939 +0x1fd
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 377 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0072895e0, 0x1, 0x0, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/internal/transport/controlbuf.go:406 +0xff
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00716ac00, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/internal/transport/controlbuf.go:533 +0x1dd
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc/internal/transport.NewServerTransport.func2(0xc0061af380)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/internal/transport/http2_server.go:304 +0xd7
Oct 04 01:02:02 xhost erigon[1788474]: created by google.golang.org/grpc/internal/transport.NewServerTransport
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/internal/transport/http2_server.go:301 +0x11b0
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 4256 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: net.(*netFD).connect.func2(0x20ac008, 0xc013ab8360, 0xc003126300, 0xc0064cc3c0, 0xc0064cc2a0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/fd_unix.go:118 +0x8a
Oct 04 01:02:02 xhost erigon[1788474]: created by net.(*netFD).connect
Oct 04 01:02:02 xhost erigon[1788474]:         net/fd_unix.go:117 +0x253
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 1482 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).pingLoop(0xc00324aa20)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:281 +0x13b
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).run
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:234 +0x125
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 740 [select, 2 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).run(0xc007a1baa0, 0x13b10e0, 0xc007b02400, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:243 +0x265
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Server).runPeer(0xc005066000, 0xc007a1baa0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:1074 +0x205
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Server).launchPeer
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:1056 +0xc5
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 389 [chan receive, 3 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/ethdb/privateapi.(*MiningServer).OnPendingLogs(0xc00022e380, 0xc0073829f0, 0x20b7428, 0xc00749b200, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/ethdb/privateapi/mining.go:92 +0xb3
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon-lib/gointerfaces/txpool._Mining_OnPendingLogs_Handler(0x159b7e0, 0xc00022e380, 0x20b4e38, 0xc006fc6900, 0x20, 0x20)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/gointerfaces/txpool/mining_grpc.pb.go:361 +0x113
Oct 04 01:02:02 xhost erigon[1788474]: github.com/grpc-ecosystem/go-grpc-middleware/recovery.StreamServerInterceptor.func1(0x159b7e0, 0xc00022e380, 0x20b4e38, 0xc006fc6900, 0xc007201908, 0x1f2d290, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:51 +0xc2
Oct 04 01:02:02 xhost erigon[1788474]: github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x159b7e0, 0xc00022e380, 0x20b4e38, 0xc006fc6900, 0xc003110c20, 0x454078)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:49 +0x5f
Oct 04 01:02:02 xhost erigon[1788474]: github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1(0x159b7e0, 0xc00022e380, 0x20b4e38, 0xc006fc6900, 0xc007201908, 0x1f2d290, 0x20ac040, 0xc0073829c0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:58 +0xd1
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc.(*Server).processStreamingRPC(0xc000340e00, 0x20bb058, 0xc0061af380, 0xc0071685a0, 0xc002e47740, 0x2881660, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:1549 +0x6d8
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc.(*Server).handleStream(0xc000340e00, 0x20bb058, 0xc0061af380, 0xc0071685a0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:1622 +0xca5
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0072e03e0, 0xc000340e00, 0x20bb058, 0xc0061af380, 0xc0071685a0)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:941 +0xab
Oct 04 01:02:02 xhost erigon[1788474]: created by google.golang.org/grpc.(*Server).serveStreams.func1
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:939 +0x1fd
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 390 [select, 3 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon-lib/kv/remotedbserver.(*KvServer).StateChanges(0xc0006a0e10, 0xc0071f39b0, 0x20b71f8, 0xc007043d90, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/kv/remotedbserver/server.go:263 +0x127
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon-lib/gointerfaces/remote._KV_StateChanges_Handler(0x14e3760, 0xc0006a0e10, 0x20b4e38, 0xc0064e6840, 0x20, 0x20)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/gointerfaces/remote/kv_grpc.pb.go:205 +0x113
Oct 04 01:02:02 xhost erigon[1788474]: github.com/grpc-ecosystem/go-grpc-middleware/recovery.StreamServerInterceptor.func1(0x14e3760, 0xc0006a0e10, 0x20b4e38, 0xc0064e6840, 0xc0071e2f00, 0x1f2d1e0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:51 +0xc2
Oct 04 01:02:02 xhost erigon[1788474]: github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x14e3760, 0xc0006a0e10, 0x20b4e38, 0xc0064e6840, 0xc0050f8c20, 0x454078)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:49 +0x5f
Oct 04 01:02:02 xhost erigon[1788474]: github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1(0x14e3760, 0xc0006a0e10, 0x20b4e38, 0xc0064e6840, 0xc0071e2f00, 0x1f2d1e0, 0x20ac040, 0xc0071f3980)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:58 +0xd1
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc.(*Server).processStreamingRPC(0xc000340e00, 0x20bb058, 0xc0061af380, 0xc0071686c0, 0xc002e477d0, 0x287dd60, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:1549 +0x6d8
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc.(*Server).handleStream(0xc000340e00, 0x20bb058, 0xc0061af380, 0xc0071686c0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:1622 +0xca5
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0072e03e0, 0xc000340e00, 0x20bb058, 0xc0061af380, 0xc0071686c0)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:941 +0xab
Oct 04 01:02:02 xhost erigon[1788474]: created by google.golang.org/grpc.(*Server).serveStreams.func1
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:939 +0x1fd
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 391 [select, 3 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/ethdb/privateapi.(*MiningServer).OnPendingBlock(0xc00022e380, 0xc007179350, 0x20b73d8, 0xc00749b150, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/ethdb/privateapi/mining.go:109 +0x128
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon-lib/gointerfaces/txpool._Mining_OnPendingBlock_Handler(0x159b7e0, 0xc00022e380, 0x20b4e38, 0xc0050e2600, 0x20, 0x20)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon-lib@v0.0.0-20210930024651-a59261cb3242/gointerfaces/txpool/mining_grpc.pb.go:319 +0x113
Oct 04 01:02:02 xhost erigon[1788474]: github.com/grpc-ecosystem/go-grpc-middleware/recovery.StreamServerInterceptor.func1(0x159b7e0, 0xc00022e380, 0x20b4e38, 0xc0050e2600, 0xc0071669f0, 0x1f2d288, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:51 +0xc2
Oct 04 01:02:02 xhost erigon[1788474]: github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x159b7e0, 0xc00022e380, 0x20b4e38, 0xc0050e2600, 0xc005036c20, 0x454078)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:49 +0x5f
Oct 04 01:02:02 xhost erigon[1788474]: github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1(0x159b7e0, 0xc00022e380, 0x20b4e38, 0xc0050e2600, 0xc0071669f0, 0x1f2d288, 0x20ac040, 0xc007179320)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:58 +0xd1
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc.(*Server).processStreamingRPC(0xc000340e00, 0x20bb058, 0xc0061af380, 0xc0071687e0, 0xc002e47740, 0x2881620, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:1549 +0x6d8
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc.(*Server).handleStream(0xc000340e00, 0x20bb058, 0xc0061af380, 0xc0071687e0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:1622 +0xca5
Oct 04 01:02:02 xhost erigon[1788474]: google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0072e03e0, 0xc000340e00, 0x20bb058, 0xc0061af380, 0xc0071687e0)
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:941 +0xab
Oct 04 01:02:02 xhost erigon[1788474]: created by google.golang.org/grpc.(*Server).serveStreams.func1
Oct 04 01:02:02 xhost erigon[1788474]:         google.golang.org/grpc@v1.39.1/server.go:939 +0x1fd
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 3700 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).pingLoop(0xc015b3b680)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:281 +0x13b
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).run
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:234 +0x125
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 1038 [IO wait]:
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.runtime_pollWait(0x7f41169d9860, 0x72, 0xffffffffffffffff)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/netpoll.go:222 +0x55
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).wait(0xc006f01418, 0x72, 0x1a000, 0x1a000, 0xffffffffffffffff)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:87 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).waitRead(...)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:92
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*FD).Read(0xc006f01400, 0xc005150000, 0x1a000, 0x1a000, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_unix.go:166 +0x1d5
Oct 04 01:02:02 xhost erigon[1788474]: net.(*netFD).Read(0xc006f01400, 0xc005150000, 0x1a000, 0x1a000, 0xc019eca260, 0xc0041e75f0, 0x20)
Oct 04 01:02:02 xhost erigon[1788474]:         net/fd_posix.go:55 +0x4f
Oct 04 01:02:02 xhost erigon[1788474]: net.(*conn).Read(0xc0005ea180, 0xc005150000, 0x1a000, 0x1a000, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/net.go:183 +0x91
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*meteredConn).Read(0xc007ba9df0, 0xc005150000, 0x1a000, 0x1a000, 0x490dec, 0xc0005e2d18, 0xa1922a)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/metrics.go:63 +0x51
Oct 04 01:02:02 xhost erigon[1788474]: io.ReadAtLeast(0x7f41145b89a8, 0xc007ba9df0, 0xc005150000, 0x1a000, 0x1a000, 0x20, 0x1448120, 0x7f41169d9940, 0x1722)
Oct 04 01:02:02 xhost erigon[1788474]:         io/io.go:328 +0x87
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*readBuffer).read(0xc0071f5000, 0x7f41145b89a8, 0xc007ba9df0, 0x20, 0x45043f, 0xc000032000, 0x14820e0, 0x1552d40, 0x7f41145b89a8)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/buffer.go:64 +0x12e
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*sessionState).readFrame(0xc0071f4f00, 0x7f41145b89a8, 0xc007ba9df0, 0x7f41145b89a8, 0xc007ba9df0, 0x2aefe60, 0x72, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/rlpx.go:166 +0x105
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*Conn).Read(0xc00702edc0, 0xc04eb28a1a9f4682, 0x318cab8026, 0x2aefe60, 0x0, 0x0, 0x2aefe60, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/rlpx.go:136 +0x98
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*rlpxTransport).ReadMsg(0xc0077f0880, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/transport.go:67 +0x154
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).readLoop(0xc00716baa0, 0xc00716bb60)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:298 +0xd1
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).run
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:233 +0xfb
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 3194 [IO wait]:
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.runtime_pollWait(0x7f4114246830, 0x72, 0xffffffffffffffff)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/netpoll.go:222 +0x55
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).wait(0xc0050c8298, 0x72, 0x1a000, 0x1a000, 0xffffffffffffffff)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:87 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).waitRead(...)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:92
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*FD).Read(0xc0050c8280, 0xc01c560000, 0x1a000, 0x1a000, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_unix.go:166 +0x1d5
Oct 04 01:02:02 xhost erigon[1788474]: net.(*netFD).Read(0xc0050c8280, 0xc01c560000, 0x1a000, 0x1a000, 0xc019eca1e0, 0xc0041e6b70, 0x20)
Oct 04 01:02:02 xhost erigon[1788474]:         net/fd_posix.go:55 +0x4f
Oct 04 01:02:02 xhost erigon[1788474]: net.(*conn).Read(0xc000010180, 0xc01c560000, 0x1a000, 0x1a000, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/net.go:183 +0x91
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*meteredConn).Read(0xc0077d80e0, 0xc01c560000, 0x1a000, 0x1a000, 0x490dec, 0xc0064cc3b8, 0xa1922a)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/metrics.go:63 +0x51
Oct 04 01:02:02 xhost erigon[1788474]: io.ReadAtLeast(0x7f41145b89a8, 0xc0077d80e0, 0xc01c560000, 0x1a000, 0x1a000, 0x20, 0x1448120, 0x7f4114246910, 0xefc)
Oct 04 01:02:02 xhost erigon[1788474]:         io/io.go:328 +0x87
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*readBuffer).read(0xc006db1f00, 0x7f41145b89a8, 0xc0077d80e0, 0x20, 0x45043f, 0xc000032000, 0x14820e0, 0x1552d40, 0x7f41145b89a8)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/buffer.go:64 +0x12e
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*sessionState).readFrame(0xc006db1e00, 0x7f41145b89a8, 0xc0077d80e0, 0x7f41145b89a8, 0xc0077d80e0, 0x2aefe60, 0x72, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/rlpx.go:166 +0x105
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*Conn).Read(0xc007b920f0, 0xc04eb28a1a960155, 0x318ca23aff, 0x2aefe60, 0x0, 0x0, 0x2aefe60, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/rlpx.go:136 +0x98
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*rlpxTransport).ReadMsg(0xc00713e1c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/transport.go:67 +0x154
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).readLoop(0xc01ae74360, 0xc01ae74420)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:298 +0xd1
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).run
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:233 +0xfb
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 4146 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*protoRW).ReadMsg(0xc0031201e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:456 +0xcf
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.runPeer(0x20abf98, 0xc00003f500, 0xc0074c6740, 0x40, 0x42, 0x209f0b0, 0xc0031201e0, 0xc01530a4e0, 0xc005039e70, 0xc005039e60, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:289 +0x1f3
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.NewSentryServer.func1(0xc01530a3c0, 0x209f0b0, 0xc0031201e0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:496 +0x79d
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).startProtocols.func1(0xc01530a3c0, 0xc0031201e0, 0x209f0b0, 0xc0031201e0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:399 +0xb4
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).startProtocols
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:396 +0x205
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 1483 [select, 2 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*protoRW).ReadMsg(0xc00013e3c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:456 +0xcf
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.runPeer(0x20abf98, 0xc00003f500, 0xc00424fa80, 0x40, 0x42, 0x209f0b0, 0xc00013e3c0, 0xc00324ab40, 0xc006e80e70, 0xc006e80e60, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:289 +0x1f3
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.NewSentryServer.func1(0xc00324aa20, 0x209f0b0, 0xc00013e3c0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:496 +0x79d
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).startProtocols.func1(0xc00324aa20, 0xc00013e3c0, 0x209f0b0, 0xc00013e3c0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:399 +0xb4
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).startProtocols
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:396 +0x205
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 709 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).pingLoop(0xc007a1baa0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:281 +0x13b
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).run
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:234 +0x125
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 1821 [IO wait]:
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.runtime_pollWait(0x7f41169d9948, 0x72, 0xffffffffffffffff)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/netpoll.go:222 +0x55
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).wait(0xc004388498, 0x72, 0x2a000, 0x2a000, 0xffffffffffffffff)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:87 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).waitRead(...)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:92
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*FD).Read(0xc004388480, 0xc01bf0e000, 0x2a000, 0x2a000, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_unix.go:166 +0x1d5
Oct 04 01:02:02 xhost erigon[1788474]: net.(*netFD).Read(0xc004388480, 0xc01bf0e000, 0x2a000, 0x2a000, 0xc012736020, 0xc0041e62b0, 0x20)
Oct 04 01:02:02 xhost erigon[1788474]:         net/fd_posix.go:55 +0x4f
Oct 04 01:02:02 xhost erigon[1788474]: net.(*conn).Read(0xc0064c2078, 0xc01bf0e000, 0x2a000, 0x2a000, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/net.go:183 +0x91
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*meteredConn).Read(0xc00749afd0, 0xc01bf0e000, 0x2a000, 0x2a000, 0x490dec, 0xc0071da958, 0xc003c801f0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/metrics.go:63 +0x51
Oct 04 01:02:02 xhost erigon[1788474]: io.ReadAtLeast(0x7f41145b89a8, 0xc00749afd0, 0xc01bf0e000, 0x2a000, 0x2a000, 0x20, 0x1448120, 0x7f41169d9a28, 0x114)
Oct 04 01:02:02 xhost erigon[1788474]:         io/io.go:328 +0x87
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*readBuffer).read(0xc00758ec40, 0x7f41145b89a8, 0xc00749afd0, 0x20, 0x45043f, 0xc000032000, 0x14820e0, 0x1552d40, 0x7f41145b89a8)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/buffer.go:64 +0x12e
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*sessionState).readFrame(0xc00758eb40, 0x7f41145b89a8, 0xc00749afd0, 0x7f41145b89a8, 0xc00749afd0, 0x2aefe60, 0x72, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/rlpx.go:166 +0x105
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*Conn).Read(0xc00428a230, 0xc04eb28933a4d4d0, 0x30b745e67f, 0x2aefe60, 0x0, 0x0, 0x2aefe60, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/rlpx.go:136 +0x98
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*rlpxTransport).ReadMsg(0xc00713e080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/transport.go:67 +0x154
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).readLoop(0xc019086c00, 0xc019086cc0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:298 +0xd1
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).run
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:233 +0xfb
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 1480 [select, 2 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).run(0xc00324aa20, 0x13b10e0, 0xc004388700, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:243 +0x265
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Server).runPeer(0xc005066000, 0xc00324aa20)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:1074 +0x205
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Server).launchPeer
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:1056 +0xc5
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 1037 [select, 2 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).run(0xc00716baa0, 0x13b10e0, 0xc0070ca780, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:243 +0x265
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Server).runPeer(0xc005066000, 0xc00716baa0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:1074 +0x205
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Server).launchPeer
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:1056 +0xc5
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 3193 [select, 1 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).run(0xc01ae74360, 0x13b10e0, 0xc0042e6280, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:243 +0x265
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Server).runPeer(0xc0000b42c0, 0xc01ae74360)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:1074 +0x205
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Server).launchPeer
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:1056 +0xc5
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 1823 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*protoRW).ReadMsg(0xc0050c40a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:456 +0xcf
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.runPeer(0x20abf98, 0xc00003f500, 0xc001b6d700, 0x40, 0x42, 0x209f0b0, 0xc0050c40a0, 0xc019086d20, 0xc0064f8e70, 0xc0064f8e60, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:289 +0x1f3
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.NewSentryServer.func1(0xc019086c00, 0x209f0b0, 0xc0050c40a0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:496 +0x79d
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).startProtocols.func1(0xc019086c00, 0xc0050c40a0, 0x209f0b0, 0xc0050c40a0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:399 +0xb4
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).startProtocols
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:396 +0x205
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 1040 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*protoRW).ReadMsg(0xc0042d5ea0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:456 +0xcf
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.runPeer(0x20abf98, 0xc00003f500, 0xc0078835c0, 0x40, 0x42, 0x209f0b0, 0xc0042d5ea0, 0xc00716bbc0, 0xc0079cde70, 0xc0079cde60, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:289 +0x1f3
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.NewSentryServer.func1(0xc00716baa0, 0x209f0b0, 0xc0042d5ea0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:496 +0x79d
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).startProtocols.func1(0xc00716baa0, 0xc0042d5ea0, 0x209f0b0, 0xc0042d5ea0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:399 +0xb4
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).startProtocols
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:396 +0x205
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 1039 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).pingLoop(0xc00716baa0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:281 +0x13b
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).run
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:234 +0x125
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 3196 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*protoRW).ReadMsg(0xc003120140, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:456 +0xcf
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.runPeer(0x20abf98, 0xc00003f500, 0xc0074c68c0, 0x40, 0x41, 0x209f0b0, 0xc003120140, 0xc01ae74480, 0xc006e83e70, 0xc006e83e60, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:289 +0x1f3
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.NewSentryServer.func1(0xc01ae74360, 0x209f0b0, 0xc003120140, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:496 +0x79d
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).startProtocols.func1(0xc01ae74360, 0xc003120140, 0x209f0b0, 0xc003120140)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:399 +0xb4
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).startProtocols
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:396 +0x205
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 3059 [select, 1 minutes]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).run(0xc0191fa300, 0x13b10e0, 0xc0064b2100, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:243 +0x265
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Server).runPeer(0xc005066000, 0xc0191fa300)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:1074 +0x205
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Server).launchPeer
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:1056 +0xc5
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 4338 [chan receive]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).ping(0xc0000db3b0, 0xc00702db90, 0x7, 0x3, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:220 +0x18c
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/discover.(*Table).doRevalidate(0xc0000a03c0, 0xc0064cc0c0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/discover/table.go:330 +0xe2
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p/discover.(*Table).loop
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/discover/table.go:258 +0x596
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 1822 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).pingLoop(0xc019086c00)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:281 +0x13b
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).run
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:234 +0x125
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 3195 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).pingLoop(0xc01ae74360)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:281 +0x13b
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).run
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:234 +0x125
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 3061 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).pingLoop(0xc0191fa300)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:281 +0x13b
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).run
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:234 +0x125
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 2893 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*protoRW).ReadMsg(0xc003120000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:456 +0xcf
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.runPeer(0x20abf98, 0xc00003f500, 0xc0043cc0c0, 0x40, 0x42, 0x209f0b0, 0xc003120000, 0xc010db0240, 0xc006e86e70, 0xc006e86e60, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:289 +0x1f3
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.NewSentryServer.func1(0xc010db0120, 0x209f0b0, 0xc003120000, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:496 +0x79d
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).startProtocols.func1(0xc010db0120, 0xc003120000, 0x209f0b0, 0xc003120000)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:399 +0xb4
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).startProtocols
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:396 +0x205
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 2891 [IO wait]:
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.runtime_pollWait(0x7f4114246578, 0x72, 0xffffffffffffffff)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/netpoll.go:222 +0x55
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).wait(0xc004388798, 0x72, 0x2d800, 0x2d868, 0xffffffffffffffff)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:87 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).waitRead(...)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:92
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*FD).Read(0xc004388780, 0xc01c502798, 0x2d868, 0x2d868, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_unix.go:166 +0x1d5
Oct 04 01:02:02 xhost erigon[1788474]: net.(*netFD).Read(0xc004388780, 0xc01c502798, 0x2d868, 0x2d868, 0xb50, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/fd_posix.go:55 +0x4f
Oct 04 01:02:02 xhost erigon[1788474]: net.(*conn).Read(0xc0064c2188, 0xc01c502798, 0x2d868, 0x2d868, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/net.go:183 +0x91
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*meteredConn).Read(0xc0072ae190, 0xc01c502798, 0x2d868, 0x2d868, 0xb50, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/metrics.go:63 +0x51
Oct 04 01:02:02 xhost erigon[1788474]: io.ReadAtLeast(0x7f41145b89a8, 0xc0072ae190, 0xc01c5010f8, 0x2ef08, 0x2ef08, 0xef28, 0x30000, 0xc01c500000, 0x10)
Oct 04 01:02:02 xhost erigon[1788474]:         io/io.go:328 +0x87
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*readBuffer).read(0xc006e99780, 0x7f41145b89a8, 0xc0072ae190, 0x10000, 0xc01c500000, 0x10, 0x30000, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/buffer.go:64 +0x12e
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*sessionState).readFrame(0xc006e99680, 0x7f41145b89a8, 0xc0072ae190, 0x7f41145b89a8, 0xc0072ae190, 0x2aefe60, 0x72, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/rlpx.go:187 +0x299
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*Conn).Read(0xc00428a4b0, 0xc04eb289917b44f4, 0x310c51ea95, 0x2aefe60, 0x0, 0x0, 0x2aefe60, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/rlpx.go:136 +0x98
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*rlpxTransport).ReadMsg(0xc003124a40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/transport.go:67 +0x154
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).readLoop(0xc010db0120, 0xc010db01e0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:298 +0xd1
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).run
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:233 +0xfb
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 3701 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*protoRW).ReadMsg(0xc000622000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:456 +0xcf
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.runPeer(0x20abf98, 0xc00003f500, 0xc003151600, 0x40, 0x41, 0x209f0b0, 0xc000622000, 0xc015b3b7a0, 0xc0079cbe70, 0xc0079cbe60, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:289 +0x1f3
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.NewSentryServer.func1(0xc015b3b680, 0x209f0b0, 0xc000622000, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:496 +0x79d
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).startProtocols.func1(0xc015b3b680, 0xc000622000, 0x209f0b0, 0xc000622000)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:399 +0xb4
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).startProtocols
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:396 +0x205
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 3699 [IO wait]:
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.runtime_pollWait(0x7f41169d9778, 0x72, 0xffffffffffffffff)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/netpoll.go:222 +0x55
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).wait(0xc004388618, 0x72, 0x400, 0x400, 0xffffffffffffffff)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:87 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).waitRead(...)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:92
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*FD).Read(0xc004388600, 0xc000165000, 0x400, 0x400, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_unix.go:166 +0x1d5
Oct 04 01:02:02 xhost erigon[1788474]: net.(*netFD).Read(0xc004388600, 0xc000165000, 0x400, 0x400, 0xc012a06220, 0xc0126b9b30, 0x20)
Oct 04 01:02:02 xhost erigon[1788474]:         net/fd_posix.go:55 +0x4f
Oct 04 01:02:02 xhost erigon[1788474]: net.(*conn).Read(0xc000010310, 0xc000165000, 0x400, 0x400, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/net.go:183 +0x91
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*meteredConn).Read(0xc0077d81b0, 0xc000165000, 0x400, 0x400, 0xc012a06220, 0xc00310ab70, 0xa1922a)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/metrics.go:63 +0x51
Oct 04 01:02:02 xhost erigon[1788474]: io.ReadAtLeast(0x7f41145b89a8, 0xc0077d81b0, 0xc000165000, 0x400, 0x400, 0x20, 0x1448120, 0x7f41169d9858, 0x1e2)
Oct 04 01:02:02 xhost erigon[1788474]:         io/io.go:328 +0x87
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*readBuffer).read(0xc0070b6d80, 0x7f41145b89a8, 0xc0077d81b0, 0x20, 0x45043f, 0xc000032000, 0x14820e0, 0x1552d40, 0x7f41145b89a8)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/buffer.go:64 +0x12e
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*sessionState).readFrame(0xc0070b6c80, 0x7f41145b89a8, 0xc0077d81b0, 0x7f41145b89a8, 0xc0077d81b0, 0x2aefe60, 0x72, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/rlpx.go:166 +0x105
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*Conn).Read(0xc000097220, 0xc04eb288a9045456, 0x30356fd206, 0x2aefe60, 0x0, 0x0, 0x2aefe60, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/rlpx.go:136 +0x98
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*rlpxTransport).ReadMsg(0xc000636280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/transport.go:67 +0x154
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).readLoop(0xc015b3b680, 0xc015b3b740)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:298 +0xd1
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).run
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:233 +0xfb
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 4111 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).run(0xc01530a3c0, 0x13b10e0, 0xc003126080, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:243 +0x265
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Server).runPeer(0xc005066000, 0xc01530a3c0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:1074 +0x205
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Server).launchPeer
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:1056 +0xc5
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 3698 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).run(0xc015b3b680, 0x13b10e0, 0xc0042e6900, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:243 +0x265
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Server).runPeer(0xc0000b42c0, 0xc015b3b680)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:1074 +0x205
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Server).launchPeer
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/server.go:1056 +0xc5
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 2892 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).pingLoop(0xc010db0120)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:281 +0x13b
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).run
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:234 +0x125
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 3062 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*protoRW).ReadMsg(0xc0031200a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:456 +0xcf
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.runPeer(0x20abf98, 0xc00003f500, 0xc0043cc280, 0x40, 0x42, 0x209f0b0, 0xc0031200a0, 0xc018258000, 0xc006e82e70, 0xc006e82e60, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:289 +0x1f3
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/cmd/sentry/download.NewSentryServer.func1(0xc0191fa300, 0x209f0b0, 0xc0031200a0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/cmd/sentry/download/sentry.go:496 +0x79d
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).startProtocols.func1(0xc0191fa300, 0xc0031200a0, 0x209f0b0, 0xc0031200a0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:399 +0xb4
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).startProtocols
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:396 +0x205
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 3060 [IO wait]:
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.runtime_pollWait(0x7f41169d9690, 0x72, 0xffffffffffffffff)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/netpoll.go:222 +0x55
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).wait(0xc000138298, 0x72, 0x1a000, 0x1a000, 0xffffffffffffffff)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:87 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).waitRead(...)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:92
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*FD).Read(0xc000138280, 0xc000bd6000, 0x1a000, 0x1a000, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_unix.go:166 +0x1d5
Oct 04 01:02:02 xhost erigon[1788474]: net.(*netFD).Read(0xc000138280, 0xc000bd6000, 0x1a000, 0x1a000, 0xc019ecac00, 0xc00742c0f0, 0x20)
Oct 04 01:02:02 xhost erigon[1788474]:         net/fd_posix.go:55 +0x4f
Oct 04 01:02:02 xhost erigon[1788474]: net.(*conn).Read(0xc00062c128, 0xc000bd6000, 0x1a000, 0x1a000, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/net.go:183 +0x91
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*meteredConn).Read(0xc0077d8000, 0xc000bd6000, 0x1a000, 0x1a000, 0x490dec, 0xc0071da058, 0xa1922a)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/metrics.go:63 +0x51
Oct 04 01:02:02 xhost erigon[1788474]: io.ReadAtLeast(0x7f41145b89a8, 0xc0077d8000, 0xc000bd6000, 0x1a000, 0x1a000, 0x20, 0x1448120, 0x7f41169d9770, 0x27b)
Oct 04 01:02:02 xhost erigon[1788474]:         io/io.go:328 +0x87
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*readBuffer).read(0xc0050d18c0, 0x7f41145b89a8, 0xc0077d8000, 0x20, 0x45043f, 0xc000032000, 0x14820e0, 0x1552d40, 0x7f41145b89a8)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/buffer.go:64 +0x12e
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*sessionState).readFrame(0xc0050d17c0, 0x7f41145b89a8, 0xc0077d8000, 0x7f41145b89a8, 0xc0077d8000, 0x2aefe60, 0x72, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/rlpx.go:166 +0x105
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*Conn).Read(0xc004252000, 0xc04eb28a126d61d6, 0x3184799b78, 0x2aefe60, 0x0, 0x0, 0x2aefe60, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/rlpx.go:136 +0x98
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*rlpxTransport).ReadMsg(0xc0064b0000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/transport.go:67 +0x154
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).readLoop(0xc0191fa300, 0xc0191fab40)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:298 +0xd1
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).run
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:233 +0xfb
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 4113 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).pingLoop(0xc01530a3c0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:281 +0x13b
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).run
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:234 +0x125
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 4255 [IO wait]:
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.runtime_pollWait(0x7f4114246660, 0x77, 0xc0073b7200)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/netpoll.go:222 +0x55
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).wait(0xc003126318, 0x77, 0x20ac000, 0xc013ab8360, 0xc003126300)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:87 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).waitWrite(...)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:96
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*FD).WaitWrite(...)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_unix.go:528
Oct 04 01:02:02 xhost erigon[1788474]: net.(*netFD).connect(0xc003126300, 0x20ac008, 0xc013ab8360, 0x0, 0x0, 0x208f2e0, 0xc005ae4500, 0x0, 0x0, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         net/fd_unix.go:141 +0x27b
Oct 04 01:02:02 xhost erigon[1788474]: net.(*netFD).dial(0xc003126300, 0x20ac008, 0xc013ab8360, 0x20b52b8, 0x0, 0x20b52b8, 0xc0070743f0, 0x0, 0x7f4114e8ce18, 0x10)
Oct 04 01:02:02 xhost erigon[1788474]:         net/sock_posix.go:149 +0x10b
Oct 04 01:02:02 xhost erigon[1788474]: net.socket(0x20ac008, 0xc013ab8360, 0x1631eb3, 0x3, 0x2, 0x1, 0x0, 0x0, 0x20b52b8, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         net/sock_posix.go:70 +0x1c5
Oct 04 01:02:02 xhost erigon[1788474]: net.internetSocket(0x20ac008, 0xc013ab8360, 0x1631eb3, 0x3, 0x20b52b8, 0x0, 0x20b52b8, 0xc0070743f0, 0x1, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         net/ipsock_posix.go:141 +0x145
Oct 04 01:02:02 xhost erigon[1788474]: net.(*sysDialer).doDialTCP(0xc003126280, 0x20ac008, 0xc013ab8360, 0x0, 0xc0070743f0, 0x1467e80, 0x2b29130, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/tcpsock_posix.go:65 +0xc5
Oct 04 01:02:02 xhost erigon[1788474]: net.(*sysDialer).dialTCP(0xc003126280, 0x20ac008, 0xc013ab8360, 0x0, 0xc0070743f0, 0xc0064f6ba0, 0x44b410, 0x60)
Oct 04 01:02:02 xhost erigon[1788474]:         net/tcpsock_posix.go:61 +0xd7
Oct 04 01:02:02 xhost erigon[1788474]: net.(*sysDialer).dialSingle(0xc003126280, 0x20ac008, 0xc013ab8360, 0x20a0078, 0xc0070743f0, 0x0, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/dial.go:580 +0x5e8
Oct 04 01:02:02 xhost erigon[1788474]: net.(*sysDialer).dialSerial(0xc003126280, 0x20ac008, 0xc013ab8360, 0xc0075ec370, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/dial.go:548 +0x15e
Oct 04 01:02:02 xhost erigon[1788474]: net.(*Dialer).DialContext(0xc005cb8900, 0x20ac008, 0xc013ab8360, 0x1631eb3, 0x3, 0xc0036ca060, 0x11, 0x0, 0x0, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         net/dial.go:425 +0x6e5
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.tcpDialer.Dial(0xc005cb8900, 0x20abf98, 0xc000636b40, 0xc0042f58c0, 0x0, 0xc005072ee8, 0x4d2b52, 0xc0144a2270)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/dial.go:69 +0x91
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*dialTask).dial(0xc0070742d0, 0xc005c78160, 0xc0042f58c0, 0xc002df60c0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/dial.go:549 +0x72
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*dialTask).run(0xc0070742d0, 0xc005c78160)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/dial.go:499 +0x59
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*dialScheduler).startDial.func1(0xc0070742d0, 0xc005c78160)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/dial.go:470 +0x5b
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*dialScheduler).startDial
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/dial.go:468 +0x3fc
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 4164 [IO wait]:
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.runtime_pollWait(0x7f41169d9208, 0x77, 0xc005cfd500)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/netpoll.go:222 +0x55
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).wait(0xc0042e6418, 0x77, 0x20ac000, 0xc016672660, 0xc0042e6400)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:87 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).waitWrite(...)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:96
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*FD).WaitWrite(...)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_unix.go:528
Oct 04 01:02:02 xhost erigon[1788474]: net.(*netFD).connect(0xc0042e6400, 0x20ac008, 0xc016672660, 0x0, 0x0, 0x208f2e0, 0xc016542c40, 0x0, 0x0, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         net/fd_unix.go:141 +0x27b
Oct 04 01:02:02 xhost erigon[1788474]: net.(*netFD).dial(0xc0042e6400, 0x20ac008, 0xc016672660, 0x20b52b8, 0x0, 0x20b52b8, 0xc007854a20, 0x0, 0x7f4114e8ab90, 0x10)
Oct 04 01:02:02 xhost erigon[1788474]:         net/sock_posix.go:149 +0x10b
Oct 04 01:02:02 xhost erigon[1788474]: net.socket(0x20ac008, 0xc016672660, 0x1631eb3, 0x3, 0x2, 0x1, 0x0, 0x0, 0x20b52b8, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         net/sock_posix.go:70 +0x1c5
Oct 04 01:02:02 xhost erigon[1788474]: net.internetSocket(0x20ac008, 0xc016672660, 0x1631eb3, 0x3, 0x20b52b8, 0x0, 0x20b52b8, 0xc007854a20, 0x1, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         net/ipsock_posix.go:141 +0x145
Oct 04 01:02:02 xhost erigon[1788474]: net.(*sysDialer).doDialTCP(0xc0042e6380, 0x20ac008, 0xc016672660, 0x0, 0xc007854a20, 0x1467e80, 0x2b29130, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/tcpsock_posix.go:65 +0xc5
Oct 04 01:02:02 xhost erigon[1788474]: net.(*sysDialer).dialTCP(0xc0042e6380, 0x20ac008, 0xc016672660, 0x0, 0xc007854a20, 0xc007af3ba0, 0x44b410, 0x60)
Oct 04 01:02:02 xhost erigon[1788474]:         net/tcpsock_posix.go:61 +0xd7
Oct 04 01:02:02 xhost erigon[1788474]: net.(*sysDialer).dialSingle(0xc0042e6380, 0x20ac008, 0xc016672660, 0x20a0078, 0xc007854a20, 0x0, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/dial.go:580 +0x5e8
Oct 04 01:02:02 xhost erigon[1788474]: net.(*sysDialer).dialSerial(0xc0042e6380, 0x20ac008, 0xc016672660, 0xc00749b2c0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/dial.go:548 +0x15e
Oct 04 01:02:02 xhost erigon[1788474]: net.(*Dialer).DialContext(0xc005cb8900, 0x20ac008, 0xc016672660, 0x1631eb3, 0x3, 0xc013cca090, 0x12, 0x0, 0x0, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         net/dial.go:425 +0x6e5
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.tcpDialer.Dial(0xc005cb8900, 0x20abf98, 0xc000636b40, 0xc0042f53b0, 0x14907c0, 0x7d071500, 0x7f41145c1458, 0xc0050d8f80)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/dial.go:69 +0x91
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*dialTask).dial(0xc007854870, 0xc005c78160, 0xc0042f53b0, 0xc0006360c0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/dial.go:549 +0x72
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*dialTask).run(0xc007854870, 0xc005c78160)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/dial.go:499 +0x59
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*dialScheduler).startDial.func1(0xc007854870, 0xc005c78160)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/dial.go:470 +0x5b
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*dialScheduler).startDial
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/dial.go:468 +0x3fc
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 4165 [select]:
Oct 04 01:02:02 xhost erigon[1788474]: net.(*netFD).connect.func2(0x20ac008, 0xc016672660, 0xc0042e6400, 0xc0042e8840, 0xc0042e87e0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/fd_unix.go:118 +0x8a
Oct 04 01:02:02 xhost erigon[1788474]: created by net.(*netFD).connect
Oct 04 01:02:02 xhost erigon[1788474]:         net/fd_unix.go:117 +0x253
Oct 04 01:02:02 xhost erigon[1788474]: goroutine 4112 [IO wait]:
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.runtime_pollWait(0x7f41169d95a8, 0x72, 0xffffffffffffffff)
Oct 04 01:02:02 xhost erigon[1788474]:         runtime/netpoll.go:222 +0x55
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).wait(0xc003126498, 0x72, 0x400, 0x400, 0xffffffffffffffff)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:87 +0x45
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*pollDesc).waitRead(...)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_poll_runtime.go:92
Oct 04 01:02:02 xhost erigon[1788474]: internal/poll.(*FD).Read(0xc003126480, 0xc013452800, 0x400, 0x400, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         internal/poll/fd_unix.go:166 +0x1d5
Oct 04 01:02:02 xhost erigon[1788474]: net.(*netFD).Read(0xc003126480, 0xc013452800, 0x400, 0x400, 0xc0141d6bc0, 0xc00232e9b0, 0x20)
Oct 04 01:02:02 xhost erigon[1788474]:         net/fd_posix.go:55 +0x4f
Oct 04 01:02:02 xhost erigon[1788474]: net.(*conn).Read(0xc0064c2248, 0xc013452800, 0x400, 0x400, 0x0, 0x0, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         net/net.go:183 +0x91
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*meteredConn).Read(0xc0077d80f0, 0xc013452800, 0x400, 0x400, 0x10000028a, 0xc01bb06840, 0x7f4114611678)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/metrics.go:63 +0x51
Oct 04 01:02:02 xhost erigon[1788474]: io.ReadAtLeast(0x7f41145b89a8, 0xc0077d80f0, 0xc013452800, 0x400, 0x400, 0x20, 0x1448120, 0x7f41169d9688, 0x163)
Oct 04 01:02:02 xhost erigon[1788474]:         io/io.go:328 +0x87
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*readBuffer).read(0xc0075dbc80, 0x7f41145b89a8, 0xc0077d80f0, 0x20, 0x45043f, 0xc000032000, 0x14820e0, 0x1552d40, 0x7f41145b89a8)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/buffer.go:64 +0x12e
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*sessionState).readFrame(0xc0075dbb80, 0x7f41145b89a8, 0xc0077d80f0, 0x7f41145b89a8, 0xc0077d80f0, 0x2aefe60, 0x72, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/rlpx.go:166 +0x105
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p/rlpx.(*Conn).Read(0xc004252780, 0xc04eb286c2d24337, 0x2e6e023ade, 0x2aefe60, 0x0, 0x0, 0x2aefe60, 0x0)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/rlpx/rlpx.go:136 +0x98
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*rlpxTransport).ReadMsg(0xc00012cac0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/transport.go:67 +0x154
Oct 04 01:02:02 xhost erigon[1788474]: github.com/ledgerwatch/erigon/p2p.(*Peer).readLoop(0xc01530a3c0, 0xc01530a480)
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:298 +0xd1
Oct 04 01:02:02 xhost erigon[1788474]: created by github.com/ledgerwatch/erigon/p2p.(*Peer).run
Oct 04 01:02:02 xhost erigon[1788474]:         github.com/ledgerwatch/erigon/p2p/peer.go:233 +0xfb

[AskAlexSharov]

I see that sync is inside RecoverFromDb func now and it does call addHeaderAsLink - mean't it's silently doing something (not stuck). You don't see anything in logs now?

[AskAlexSharov]

Add progress logs to RecoverFromDb func: #2769

[splix]

Hm, it was doing that for past few hours. I restarted it a few times, but I think I had given it at least an hour before first time I restarted. But I heard you, I'll give it more time.

Is it possible that it does some db upgrade from a previous version? It stuck right after I upgraded from 2021.07.04-alpha.

[AskAlexSharov]

No, because recoverFromDB func running after all db migrations applied. It's already part of normal app flow.
But header not found may be related to some old fixed bug.

[AskAlexSharov]

Do you see header not found error in logs after restart?

[splix]

yes, always. same error after each restart

[AskAlexSharov]

then wait will not help

[splix]

oh, that's bad. can a prune help there? or may be some other way to roll back to a previous block?

[AskAlexSharov]

Let's try:

./build/bin/integration stage_bodies --unwind=10_000 --datadir=<your_datadir>

[splix]

Same issue.

Unwind:

./build/bin/integration stage_bodies --unwind=10_000 --datadir=/blockchain/erigon
INFO[10-04|01:54:09.870] Progress                                 bodies=13097517
INFO[10-04|01:54:09.877] database closed                          label=chaindata

and run:

Oct 04 01:54:26 xhost erigon[1791797]: [INFO] [10-04|01:54:26.045] Build info                               git_branch=HEAD git_tag=v2021.10.01 git_commit=ef80fd54b1c6c82aae92f5fa24cbd381e8577b50
Oct 04 01:54:26 xhost erigon[1791797]: [INFO] [10-04|01:54:26.046] Starting Erigon on Ethereum mainnet...
Oct 04 01:54:26 xhost erigon[1791797]: [INFO] [10-04|01:54:26.047] Maximum peer count                       ETH=10 total=10
Oct 04 01:54:26 xhost erigon[1791797]: [INFO] [10-04|01:54:26.047] Set global gas cap                       cap=50000000
Oct 04 01:54:26 xhost erigon[1791797]: [INFO] [10-04|01:54:26.095] Opening Database                         label=chaindata path=/blockchain/erigon/chaindata
Oct 04 01:54:26 xhost erigon[1791797]: [INFO] [10-04|01:54:26.106] Initialised chain configuration          config="{ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, Berlin: 12244000, London: 12965000, Engine: ethash}"
Oct 04 01:54:26 xhost erigon[1791797]: [INFO] [10-04|01:54:26.106] Disk storage enabled for ethash DAGs     dir=/blockchain/erigon/ethash-dags count=2
Oct 04 01:54:26 xhost erigon[1791797]: [INFO] [10-04|01:54:26.106] Initialising Ethereum protocol           network=1
Oct 04 01:54:26 xhost erigon[1791797]: [INFO] [10-04|01:54:26.106] Effective                                prune="--prune="
Oct 04 01:54:26 xhost erigon[1791797]: [INFO] [10-04|01:54:26.669] Starting private RPC server              on=localhost:9091
Oct 04 01:54:26 xhost erigon[1791797]: [INFO] [10-04|01:54:26.670] [1/18 Headers] Waiting for headers...    from=13349636
Oct 04 01:54:26 xhost erigon[1791797]: [INFO] [10-04|01:54:26.695] Started P2P networking                   version=65 self=enode://XXXX@127.0.0.1:30304 name=erigon/v2021.10.1-alpha-ef80fd54/linux-amd64/go1.16
Oct 04 01:54:26 xhost erigon[1791797]: [INFO] [10-04|01:54:26.699] Started P2P networking                   version=66 self=enode://XXXX@127.0.0.1:32005 name=erigon/v2021.10.1-alpha-ef80fd54/linux-amd64/go1.16
Oct 04 01:54:29 xhost erigon[1791797]: [INFO] [10-04|01:54:29.597] Mapped network port                      proto=tcp extport=30304 intport=30304 interface="UPNP IGDv1-IP1"
Oct 04 01:54:30 xhost erigon[1791797]: [INFO] [10-04|01:54:30.406] Mapped network port                      proto=udp extport=30304 intport=30304 interface="UPNP IGDv1-IP1"
Oct 04 01:54:30 xhost erigon[1791797]: [INFO] [10-04|01:54:30.609] Mapped network port                      proto=tcp extport=32005 intport=32005 interface="UPNP IGDv1-IP1"
Oct 04 01:54:30 xhost erigon[1791797]: [INFO] [10-04|01:54:30.813] Mapped network port                      proto=udp extport=32005 intport=32005 interface="UPNP IGDv1-IP1"
Oct 04 01:54:31 xhost erigon[1791797]: [INFO] [10-04|01:54:31.087] event subscription channel established with the RPC daemon
Oct 04 01:54:56 xhost erigon[1791797]: [INFO] [10-04|01:54:56.671] [1/18 Headers] Wrote block headers       number=13349636 blk/second=0.000 alloc="82.02 MiB" sys="206.41 MiB"
Oct 04 01:55:26 xhost erigon[1791797]: [INFO] [10-04|01:55:26.671] [1/18 Headers] Wrote block headers       number=13349836 blk/second=6.667 alloc="225.42 MiB" sys="338.79 MiB"
Oct 04 01:55:42 xhost erigon[1791797]: [INFO] [10-04|01:55:42.357] [1/18 Headers] Processed                 highest inserted=13349837 age=36s
Oct 04 01:55:42 xhost erigon[1791797]: [INFO] [10-04|01:55:42.357] [1/18 Headers] DONE                      in=1m15.687047895s
Oct 04 01:55:42 xhost erigon[1791797]: [INFO] [10-04|01:55:42.404] [4/18 Bodies] Processing bodies...       from=13097517 to=13349837
Oct 04 01:56:16 xhost erigon[1791797]: [EROR] [10-04|01:56:16.403] Staged Sync                              error="[4/18 Bodies] request more bodies: header not found: %!w(<nil>), blockNum=13107518, trace=[github.com/ledgerwatch/erigon/turbo/stages/bodydownload.(*BodyDownload).RequestMoreBodies github.com/ledgerwatch/erigon/eth/stagedsync.BodiesForward github.com/ledgerwatch/erigon/eth/stagedsync.DefaultStages.func10 github.com/ledgerwatch/erigon/eth/stagedsync.(*Sync).runStage github.com/ledgerwatch/erigon/eth/stagedsync.(*Sync).Run github.com/ledgerwatch/erigon/turbo/stages.StageLoopStep github.com/ledgerwatch/erigon/turbo/stages.StageLoop]"
Oct 04 01:56:16 xhost erigon[1791797]: [INFO] [10-04|01:56:16.851] [4/18 Bodies] Processing bodies...       from=13097517 to=13349837
Oct 04 01:56:21 xhost erigon[1791797]: [EROR] [10-04|01:56:21.287] Staged Sync                              error="[4/18 Bodies] request more bodies: header not found: %!w(<nil>), blockNum=13107518, trace=[github.com/ledgerwatch/erigon/turbo/stages/bodydownload.(*BodyDownload).RequestMoreBodies github.com/ledgerwatch/erigon/eth/stagedsync.BodiesForward github.com/ledgerwatch/erigon/eth/stagedsync.DefaultStages.func10 github.com/ledgerwatch/erigon/eth/stagedsync.(*Sync).runStage github.com/ledgerwatch/erigon/eth/stagedsync.(*Sync).Run github.com/ledgerwatch/erigon/turbo/stages.StageLoopStep github.com/ledgerwatch/erigon/turbo/stages.StageLoop]"
Oct 04 01:56:21 xhost erigon[1791797]: [INFO] [10-04|01:56:21.739] [4/18 Bodies] Processing bodies...       from=13097517 to=13349837
Oct 04 01:56:26 xhost erigon[1791797]: [INFO] [10-04|01:56:26.106] [p2p] GoodPeers                          eth66=6 eth65=3
Oct 04 01:56:26 xhost erigon[1791797]: [EROR] [10-04|01:56:26.165] Staged Sync                              error="[4/18 Bodies] request more bodies: header not found: %!w(<nil>), blockNum=13107518, trace=[github.com/ledgerwatch/erigon/turbo/stages/bodydownload.(*BodyDownload).RequestMoreBodies github.com/ledgerwatch/erigon/eth/stagedsync.BodiesForward github.com/ledgerwatch/erigon/eth/stagedsync.DefaultStages.func10 github.com/ledgerwatch/erigon/eth/stagedsync.(*Sync).runStage github.com/ledgerwatch/erigon/eth/stagedsync.(*Sync).Run github.com/ledgerwatch/erigon/turbo/stages.StageLoopStep github.com/ledgerwatch/erigon/turbo/stages.StageLoop]"
Oct 04 01:56:26 xhost erigon[1791797]: [INFO] [10-04|01:56:26.578] [4/18 Bodies] Processing bodies...       from=13097517 to=13349837
Oct 04 01:56:30 xhost erigon[1791797]: [EROR] [10-04|01:56:30.956] Staged Sync                              error="[4/18 Bodies] request more bodies: header not found: %!w(<nil>), blockNum=13107518, trace=[github.com/ledgerwatch/erigon/turbo/stages/bodydownload.(*BodyDownload).RequestMoreBodies github.com/ledgerwatch/erigon/eth/stagedsync.BodiesForward github.com/ledgerwatch/erigon/eth/stagedsync.DefaultStages.func10 github.com/ledgerwatch/erigon/eth/stagedsync.(*Sync).runStage github.com/ledgerwatch/erigon/eth/stagedsync.(*Sync).Run github.com/ledgerwatch/erigon/turbo/stages.StageLoopStep github.com/ledgerwatch/erigon/turbo/stages.StageLoop]"
Oct 04 01:56:31 xhost erigon[1791797]: [INFO] [10-04|01:56:31.354] [4/18 Bodies] Processing bodies...       from=13097517 to=13349837
Oct 04 01:56:35 xhost erigon[1791797]: [EROR] [10-04|01:56:35.793] Staged Sync                              error="[4/18 Bodies] request more bodies: header not found: %!w(<nil>), blockNum=13107518, trace=[github.com/ledgerwatch/erigon/turbo/stages/bodydownload.(*BodyDownload).RequestMoreBodies github.com/ledgerwatch/erigon/eth/stagedsync.BodiesForward github.com/ledgerwatch/erigon/eth/stagedsync.DefaultStages.func10 github.com/ledgerwatch/erigon/eth/stagedsync.(*Sync).runStage github.com/ledgerwatch/erigon/eth/stagedsync.(*Sync).Run github.com/ledgerwatch/erigon/turbo/stages.StageLoopStep github.com/ledgerwatch/erigon/turbo/stages.StageLoop]"
Oct 04 01:56:36 xhost erigon[1791797]: [INFO] [10-04|01:56:36.251] [4/18 Bodies] Processing bodies...       from=13097517 to=13349837

[AskAlexSharov]

Then try switch to branch logs_in_recover_from_db and run:

./build/bin/integration stage_headers --unwind=300_000 --datadir=/blockchain/erigon