[Node Operator Issue] Granite Release bricks node, full re-sync required

[moorow]

Discussed in #564

^{Originally posted by moorow August 30, 2024}

Are you running the most up to date node software?

• Yes

Did you check the documentation?

• Yes

Did you check for duplicate questions?

• Yes

Issue Description

Updating to the Granite release seems to break the state of a node.

Here are steps I followed and logs:

1. Graceful stop op-node

2. Graceful stop op-geth

3. Start op-geth (New version)

INFO [08-30|09:27:51.999] Post-Merge hard forks (timestamp based):
INFO [08-30|09:27:51.999]  - Shanghai:                    @1704992401 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md)
INFO [08-30|09:27:51.999]  - Cancun:                      @1710374401 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/cancun.md)
INFO [08-30|09:27:51.999]  - Regolith:                    @0
INFO [08-30|09:27:51.999]  - Canyon:                      @1704992401
INFO [08-30|09:27:51.999]  - Ecotone:                     @1710374401
INFO [08-30|09:27:51.999]  - Fjord:                       @1720627201
INFO [08-30|09:27:51.999]  - Granite:                     @1726070401
INFO [08-30|09:27:51.999]
INFO [08-30|09:27:51.999] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [08-30|09:27:51.999]
INFO [08-30|09:27:52.000] Loaded most recent local block           number=124,705,593 hash=953dd8..ba9ede td=210,470,125 age=1m49s
INFO [08-30|09:27:52.000] Loaded most recent local finalized block number=124,704,852 hash=2c608b..db1b52 td=210,470,125 age=26m31s
INFO [08-30|09:27:52.000] Loaded last snap-sync pivot marker       number=123,961,326
WARN [08-30|09:27:52.000] Head state missing, repairing            number=124,705,593 hash=953dd8..ba9ede snaproot=2a2e50..f5c381
INFO [08-30|09:28:03.101] Block state missing, rewinding further   number=124,703,029 hash=baeb46..e4fa7a elapsed=11.100s
INFO [08-30|09:28:11.101] Block state missing, rewinding further   number=124,612,195 hash=441943..78c159 elapsed=19.100s
INFO [08-30|09:28:19.101] Block state missing, rewinding further   number=124,466,580 hash=de38e2..34caa6 elapsed=27.100s
INFO [08-30|09:28:27.101] Block state missing, rewinding further   number=124,251,092 hash=62041b..f1a58b elapsed=35.101s
INFO [08-30|09:28:35.101] Block state missing, rewinding further   number=124,005,258 hash=798f45..1b1643 elapsed=43.101s
INFO [08-30|09:28:36.505] Pivot block reached, resetting to genesis number=123,961,326 hash=5678e6..fd8b23
INFO [08-30|09:28:36.507] Chain is stateless, wait state sync      number=0           hash=7ca38a..3fa48b
INFO [08-30|09:28:36.508] Loaded most recent local header          number=124,705,593 hash=953dd8..ba9ede td=210,470,125 age=2m33s
INFO [08-30|09:28:36.508] Loaded most recent local block           number=0           hash=7ca38a..3fa48b td=1           age=55y5mo2w
INFO [08-30|09:28:36.508] Loaded most recent local snap block      number=124,705,593 hash=953dd8..ba9ede td=210,470,125 age=2m33s
INFO [08-30|09:28:36.508] Loaded most recent local finalized block number=124,704,852 hash=2c608b..db1b52 td=210,470,125 age=27m15s
INFO [08-30|09:28:36.508] Loaded last snap-sync pivot marker       number=123,961,326
WARN [08-30|09:28:36.509] Enabling snapshot recovery               chainhead=0 diskbase=124,705,308
WARN [08-30|09:28:36.526] Snapshot is not continuous with chain    snaproot=1a96d0..894acc chainroot=eddb4c..a568eb
INFO [08-30|09:28:36.526] Initialized transaction indexer          range="last 2350000 blocks"
INFO [08-30|09:28:36.526] Initialising Ethereum protocol           network=10         dbversion=8
INFO [08-30|09:28:36.526] Loaded local transaction journal         transactions=0 dropped=0
INFO [08-30|09:28:36.526] Enabled snap sync                        head=0 hash=7ca38a..3fa48b
INFO [08-30|09:28:36.526] Gasprice oracle is ignoring threshold set threshold=2
WARN [08-30|09:28:36.527] Engine API enabled                       protocol=eth

3. Start op-node (New version)

t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"
t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"
t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"
t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"
t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"
t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"
t=2024-08-30T09:29:05+0000 lvl=warn msg="Deriver system is resetting" err="failed to find the L2 Heads to start from: failed to fetch current L2 forkchoice state: failed to find the L2 head block: failed to determine L2BlockRef of latest, could not get payload: post-merge block header requires EIP-1559 base fee field, but got <nil>"

Protocol Description

network: op-mainnet
chainid: 10
L1 chain: mainnet

Old Versions

op-geth: v1.101315.2
op-node: v1.7.7

New Versions

op-geth: v1.101408.0
op-node: v1.9.1

Node Logs

No response

Additional Information

No response

Feedback

No response

[moorow]

Based on some discussion on Discord it looks like this may be related to Path Based (PBSS)?

Someone said that this only affects Full nodes, not Archive.

INFO [08-30|09:27:51.974] State schema set to default              scheme=path

This line was printed in the logs

[sbvegan]

Thanks for the flag, can you share your runtime configuration you're using for op-mode and op-geth?

[moorow]

Sure thing:

op-geth

      - --ws
      - --ws.port=8550
      - --ws.addr=0.0.0.0
      - --ws.origins="*"
      - --http
      - --http.port=8549
      - --http.addr=0.0.0.0
      - --http.vhosts="*"
      - --http.corsdomain="*"
      - --authrpc.addr=0.0.0.0
      - --authrpc.jwtsecret=/data/jwt.txt
      - --authrpc.port=9333
      - --authrpc.vhosts="*"
      - --datadir=/data/geth
      - --verbosity=3
      - --rollup.disabletxpoolgossip=true
      - --rollup.sequencerhttp=https://mainnet-sequencer.optimism.io
      - --op-network=op-mainnet
      - --syncmode=snap
      - --rpc.gascap=0
      - --rpc.evmtimeout=10s
      - --rpc.batch-response-max-size=50000000
      - --rpc.batch-request-limit=2000
      - --nat=extip:${HOST_IP_ADDRESS}

op-node

      - --l1=${L1_EXECUTION_RPC_URL}
      - --l1.rpckind=alchemy
      - --l1.trustrpc
      - --l1.beacon=${L1_BEACON_RPC_URL}
      - --l2=ws://op-geth:9333
      - --network=op-mainnet
      - --rpc.addr=127.0.0.1
      - --rpc.port=9545
      - --l2.jwt-secret=/data/jwt.txt
      - --syncmode=execution-layer
      - --p2p.advertise.ip=${HOST_IP_ADDRESS}
      - --p2p.listen.ip=0.0.0.0
      - --p2p.listen.tcp=9003
      - --p2p.listen.udp=9003
      - --p2p.advertise.tcp=9003
      - --p2p.advertise.udp=9003
      - --metrics.enabled
      - --metrics.addr=0.0.0.0
      - --metrics.port=7300