Replica snap sync on new L2 network #447
-
|
Bug Description The replica syncing never "finishes" even though it's syncing side by side. Left side is sequencer op-geth on the right is replica op-geth. replica op-geth Note the block number is "number": "0x0",
Expected behavior Environment Information:
Configurations: Replica node: Sequencer op-geth Logs: Additional context
Please ensure all required sections are filled out accurately to expedite the debugging process and improve issue resolution efficiency. |
Beta Was this translation helpful? Give feedback.
Replies: 7 comments 24 replies
-
|
replica op-geth: |
Beta Was this translation helpful? Give feedback.
-
|
sequencer geth: |
Beta Was this translation helpful? Give feedback.
-
|
@opfocus I think you've seen this too, can you provide any context you have? Did this ever get resolved? |
Beta Was this translation helpful? Give feedback.
-
|
Using full mode solved the problem in my tests, but I'm not sure why snap mode doesn't return the correct block number |
Beta Was this translation helpful? Give feedback.
-
|
Hey @opfocus , did you get it to work with: on op-geth and: Or did you set: for both op-geth and the replica? |
Beta Was this translation helpful? Give feedback.
-
|
#17 (comment) This is my initial sharing, I am testing it further |
Beta Was this translation helpful? Give feedback.
-
|
I have done this: both op-geth-replica and op-geth: op-geth-replica logs: op-geth: If I query op-geth the blockNumber remains 0x0: I would expect thought that I would be able to have a network with: and have and op-node-replica: |
Beta Was this translation helpful? Give feedback.
-
|
op-geth op-node.yaml replicas:op-node: op-replica block number stays 0x0 |
Beta Was this translation helpful? Give feedback.
-
|
@opfocus it makes sense and would explain why @InoMurko is getting |
Beta Was this translation helpful? Give feedback.
-
Yes that is correct. Snap sync never completes. |
Beta Was this translation helpful? Give feedback.
-
|
I have some doubts about it, I will test it again later |
Beta Was this translation helpful? Give feedback.
-
|
cofig: op-geth: ./build/bin/geth op-node: Log before snap sync completes synchronization op-node log: JSON-RPC: op-geth: Log after snap sync completes synchronization op-node: JSON-RPC: op-geth : |
Beta Was this translation helpful? Give feedback.
-
|
@opfocus thank you for taking the time to reproduce this issue. The |
Beta Was this translation helpful? Give feedback.
-
|
@opfocus thanks for this. the logs and setup you posted are from the op-node and op-geth that act as replicas. can you post the setup of main op-node and op-geth? Is this a network that was recently deployed (a fresh L2)? what version? |
Beta Was this translation helpful? Give feedback.
-
|
https://docs.optimism.io/builders/chain-operators/tutorials/create-l2-rollup#start-op-geth sequencer op-node (tutorials/chain branch/v1.7.6): |
Beta Was this translation helpful? Give feedback.
-
|
We have run into similar issues running our own OP stack off of holesky L1 testnet. After some struggling we have managed to get some connectivity going between the sequencer and replica by specifying the sequencer's op-node and geth node respective bootnode enr in the replica. This additionally required manually passing the replica op-node multiaddr as a parameter to the Here is our configs derived from our kubernetes setup: Sequencerop-geth - name: geth
args:
- --datadir=/data/geth
- --http
- --http.corsdomain=*
- --http.vhosts=*
- --http.addr=0.0.0.0
- --http.api=web3,debug,eth,txpool,net,engine
- --ws
- --ws.addr=0.0.0.0
- --ws.port=8546
- --ws.origins=*
- --ws.api=debug,eth,txpool,net,engine
- --syncmode=full
- --gcmode=archive
- --networkid=7890785
- --authrpc.vhosts=*
- --authrpc.addr=0.0.0.0
- --authrpc.port=8551
- --authrpc.jwtsecret=/jwtsecret
- --rollup.disabletxpoolgossip=true
- --rollup.sequencerhttp=http://localhost:8547
- --nat=extip:<EXTERNAL_IP>
- --override.cancun=0
- --override.canyon=0
- --override.ecotone=1717852500
- --metrics
- --metrics.expensive=trueop-node - name: op-node
args:
- --l2=http://localhost:8551
- --l2.jwt-secret=/jwtsecret
- --sequencer.enabled=true
- --sequencer.l1-confs=5
- --verifier.l1-confs=4
- --rollup.config=/rollup/rollup.json
- --rpc.addr=0.0.0.0
- --rpc.port=8547
- --rpc.enable-admin
- --p2p.sequencer.key=$(SEQUENCER_PRIVATE_KEY)
- --p2p.advertise.ip=<EXTERNAL_IP>
- --p2p.advertise.tcp=9003
- --p2p.advertise.udp=9003
- --p2p.listen.tcp=9003
- --p2p.listen.udp=9003
- --l1=<L1_EL_RPC>
- --l1.trustrpc
- --l1.rpckind=debug_geth
- --l1.beacon=<L1_CL_RPC>
- --override.ecotone=1717852500 # Sat Jun 08 2024 13:15:00 GMT+0000
- --metrics.enabled=true
Replicaop-geth - name: geth
args:
- --datadir=/data/geth
- --http
- --http.corsdomain=*
- --http.vhosts=*
- --http.addr=0.0.0.0
- --http.api=web3,debug,eth,txpool,net,engine
- --ws
- --ws.addr=0.0.0.0
- --ws.port=8546
- --ws.origins=*
- --ws.api=debug,eth,txpool,net,engine
- --syncmode=snap
# - --gcmode=archive
- --verbosity=3
- --networkid=7890785
- --authrpc.vhosts=*
- --authrpc.addr=0.0.0.0
- --authrpc.port=8551
- --authrpc.jwtsecret=/jwtsecret
- --rollup.sequencerhttp=<SEQUENCER_RPC>
- --rollup.disabletxpoolgossip=true
- --metrics
- --metrics.expensive=true
- --override.cancun=0
- --override.canyon=0
- --override.ecotone=1717852500 # Sat Jun 08 2024 13:15:00 GMT+0000
- --bootnodes=<SEQUENCER_OP_GETH_BOOTNODE_ENR>
op-node - name: op-node
args:
- --l2=http://localhost:8551
- --l2.jwt-secret=/jwtsecret
- --sequencer.enabled=false
- --sequencer.l1-confs=5
- --verifier.l1-confs=4
- --rollup.config=/rollup/rollup.json
- --rpc.addr=0.0.0.0
- --rpc.port=8547
- --syncmode=execution-layer
- --rpc.enable-admin
- --p2p.listen.ip=0.0.0.0
- --p2p.advertise.tcp=9003
- --p2p.advertise.udp=9003
- --p2p.listen.tcp=9003
- --p2p.listen.udp=9003
- --p2p.bootnodes=<SEQUENCER_OP_NODE_BOOTNODE_ENR>
- --l1=<L1_EL_RPC>
- --l1.beacon=<L1_CL_RPC>
- --l1.rpckind=debug_geth
- --override.ecotone=1717852500 # Sat Jun 08 2024 13:15:00 GMT+0000
- --metrics.enabled=true
As stated above, currently running a snap sync on the replica: Replica op-node logs:Replica op-geth logs:One problem which still persists after a couple of attempts peering replicas is getting the op-geth nodes to identify each other. |
Beta Was this translation helpful? Give feedback.
-
|
@InoMurko if this doesn't end up being a snap sync issue, then there's some additional things I'd investigate: Gathering each node's own information with opp2p_self and its peer information with opp2p_peers. Specifically taking note of its Connectedness (I think its an enum): "NotConnected", "Connected", "CanConnect" (gracefully disconnected), or "CannotConnect" (tried but failed). And comparing the sequencer and replica's sync status |
Beta Was this translation helpful? Give feedback.
-
|
Noting we may need the op-geth (execution layer) peer and connectivity information too. It might be best to provide the full |
Beta Was this translation helpful? Give feedback.
-
|
This was already shared in the slack thread, fyi |
Beta Was this translation helpful? Give feedback.
-
|
As far as I can tell - nodes connect to each other fine and they exchange p2p traffic, but the "finalisation message" that would cut over snap sync never actually arrives. replica op-node sequencer op node op node op nodes can connect to each other and there’s traffic (log from replica op-node) |
Beta Was this translation helpful? Give feedback.
-
I've had no success adding op-geth peers (but occasionally it finds it on its own). It should be noted that op-node and op-geth p2p are two different things. |
Beta Was this translation helpful? Give feedback.
-
|
I think not using snap |
Beta Was this translation helpful? Give feedback.
cofig:
op-node:
op-geth:
./build/bin/geth
--http
--http.port=8545
--http.addr=localhost
--authrpc.addr=localhost
--authrpc.jwtsecret=./jwt.txt
--verbosity=3
--rollup.sequencerhttp=http://164.92.86.206:8545
--datadir=$DATADIR_PATH
op-node: