PoA miner panic when importing blocks

[csokun]

I have set up a private PoA network with 4 validators running on Kubernetes (stateless). The network is up and running for weeks without any hiccup. I decided to restart one of a validator and it can't recover from this outage.

System information

Geth version: v1.9.9
OS & Version: Linux
Commit hash :

Expected behaviour

A validator run on Kubernetes (stateless) should be able to restart, import current blocks and join the network.

Actual behaviour

The validator node restarted and started syncing blocks and then it panic.

Steps to reproduce the behaviour

geth --nousb \
   --synmode 'fast' \
   --unlock '0x.......' \
   --password password_file.txt \
   --mine \
    --miner.gasprice 1 \
    --miner.threads 2 \
    --miner.gastarget 1000000000 \
    --miner.gaslimit 1000000000 \
   --networkid ...

Backtrace

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x530bce]

goroutine 70 [running]:
math/big.(*Int).Cmp(0xc001397958, 0x0, 0x49c6be13f88c3fb)
        /usr/local/go/src/math/big/int.go:326 +0x2e
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).handleMsg(0xc000236d20, 0xc0005ca000, 0x0, 0x0)
        /go-ethereum/eth/handler.go:722 +0xd35
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).handle(0xc000236d20, 0xc0005ca000, 0x0, 0x0)
        /go-ethereum/eth/handler.go:359 +0x864
github.com/ethereum/go-ethereum/eth.(*ProtocolManager).makeProtocol.func1(0xc0005bc060, 0x13ec3a0, 0xc000110dc0, 0x0, 0x0)
        /go-ethereum/eth/handler.go:211 +0x1ee
github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols.func1(0xc000110dc0, 0xc0005bc060, 0x13ec3a0, 0xc000110dc0)
        /go-ethereum/p2p/peer.go:367 +0x66
created by github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols
        /go-ethereum/p2p/peer.go:366 +0x203

[holiman]

This issue could happen after an unclean shutdown (crash), but should be fixed by #20287 - which makes sure that we write blockchain data in an atomic way, and should prevent these types of corruptions where we have e.g. the blocks and state but are missing the total difficulty. That PR is included in 1.9.10.

Note, though, updating to 1.9.10 will not solve the issue for you -- I would suspect that your best bet is to do a debug.setHead(...) to rewind the chaiin a bit, maybe a couple of hundred blocks, until you get to a point where you have the complete data. You might even have to go back a few thousand blocks -- I can't say really :(

[karalabe]

We've just figured out that this is actually a legit bug. Fixing now.

[karalabe]

#20770