Slow Peer Discovery on Base Mainnet (op-reth) After Restart

[suntzu93]

Describe the bug

When running an op-reth node on Base mainnet, the node takes a significant amount of time to find peers and resume syncing after a restart.

There are no observed bottlenecks in disk IO, CPU, or memory.

op-reth config

op-reth node \
    --chain base-mainnet \
    --datadir /root/.local/share/base/reth \
    --authrpc.jwtsecret /etc/base/jwt.hex \
    --authrpc.addr 127.0.0.1 \
    --authrpc.port 7551 \
    --http.port 7659 \
    --port 20303 \
    --torrent.port 42059 \
    --discovery.port 40203 \
    --http \
    --ws \
    --ws.port 7546 \
    --http.api eth,debug,net,trace,web3,reth \
    --ws.api eth,debug,net,trace,web3,reth \
    --rollup.discovery.v4 \
    --rollup.sequencer-http https://mainnet-sequencer.base.org \
    --rollup.disable-tx-pool-gossip

op-node

op-node \
                --l1=xxxx \
                --l2=http://127.0.0.1:7551 \
                --network=base-mainnet \
                --rpc.addr=0.0.0.0 \
                --rpc.port=7545 \
                --l2.jwt-secret=/etc/base/jwt.hex \
                --l1.trustrpc \
                --l1.rpckind=reth \
                --metrics.enabled \
                --metrics.port 6300 \
                 --syncmode=execution-layer \
                --l1.beacon=xxx

Steps to reproduce

Steps to reproduce:

1. Run an op-reth node on Base mainnet.
2. Restart the node.
3. Notice that it takes a long time to find peers and continue syncing.

Node logs

2025-05-12T13:06:12.527989Z  INFO Status connected_peers=0 stage=MerkleExecute checkpoint=30054414 target=30059659
2025-05-12T13:06:37.527619Z  INFO Status connected_peers=0 stage=MerkleExecute checkpoint=30054414 target=30059659
2025-05-12T13:07:02.527691Z  INFO Status connected_peers=0 stage=MerkleExecute checkpoint=30054414 target=30059659
2025-05-12T13:07:27.528421Z  INFO Status connected_peers=0 stage=MerkleExecute checkpoint=30054414 target=30059659
2025-05-12T13:07:52.527621Z  INFO Status connected_peers=0 stage=MerkleExecute checkpoint=30054414 target=30059659
2025-05-12T13:08:17.528124Z  INFO Status connected_peers=0 stage=MerkleExecute checkpoint=30054414 target=30059659
2025-05-12T13:08:42.527893Z  INFO Status connected_peers=0 stage=MerkleExecute checkpoint=30054414 target=30059659
2025-05-12T13:09:07.528400Z  INFO Status connected_peers=0 stage=MerkleExecute checkpoint=30054414 target=30059659
2025-05-12T13:09:32.528213Z  INFO Status connected_peers=0 stage=MerkleExecute checkpoint=30054414 target=30059659
2025-05-12T13:09:57.528315Z  INFO Status connected_peers=0 stage=MerkleExecute checkpoint=30054414 target=30059659

Platform(s)

Linux (x86)

Container Type

Not running in a container

What version/commit are you on?

reth-optimism-cli Version: 1.3.12

What database version are you on?

2025-05-12T13:12:41.740760Z INFO Initialized tracing, debug log directory: /root/.cache/reth/logs/dev
Error: Datadir does not exist: "/root/.local/share/reth/dev"

Location:
/root/chains/base-reth/reth/crates/cli/commands/src/db/mod.rs:76:9

Which chain / network are you on?

base-mainnet

What type of node are you running?

Archive (default)

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

cargo build --release --package op-reth

Code of Conduct

• I agree to follow the Code of Conduct

[jenpaff]

Hey @emhane ! I saw you assigned yourself to the bug, could you share your assessment with us with a proposed solution before going ahead to submit a PR? would like to discuss proposed solutions with the team first.

[emhane]

absolutely ! @suntzu93 could you please share metrics after running the node for a while and getting all the peers back. I'm curious if your NAT setup allows incoming connections at all, i.e. allows you to be discovered by others, and this panel could help us figure that out https://reth.paradigm.xyz/d/d47d679c-c3b8-40b6-852d-cbfaa2dcdb37/reth-transaction-pool?orgId=1&refresh=30s&from=1747132555879&to=1747136155879&viewPanel=16

instructions of how to run the dashboard can be found here https://github.com/paradigmxyz/reth/blob/main/book/installation/docker.md#run-only-grafana-in-docker

additional panels that would be good to see screenshots of are those in the discovery dashboard
https://reth.paradigm.xyz/d/fd2d69b5-ca32-45d0-946e-c00ddcd7052c/reth---peer-discovery?orgId=1&refresh=30s

[suntzu93]

instructions of how to run the dashboard can be found here

absolutely ! @suntzu93 could you please share metrics after running the node for a while and getting all the peers back. I'm curious if your NAT setup allows incoming connections at all, i.e. allows you to be discovered by others, and this panel could help us figure that out https://reth.paradigm.xyz/d/d47d679c-c3b8-40b6-852d-cbfaa2dcdb37/reth-transaction-pool?orgId=1&refresh=30s&from=1747132555879&to=1747136155879&viewPanel=16

instructions of how to run the dashboard can be found here https://github.com/paradigmxyz/reth/blob/main/book/installation/docker.md#run-only-grafana-in-docker

additional panels that would be good to see screenshots of are those in the discovery dashboard https://reth.paradigm.xyz/d/fd2d69b5-ca32-45d0-946e-c00ddcd7052c/reth---peer-discovery?orgId=1&refresh=30s

It took 1 to 2 hours for my node to find peers to connect with, and now it has synced to the chain head. I believe the issue is not on my NAT side since I am now connected to the peers and syncing. I will share the metrics again after the next restart (since I haven't enabled it) because I don't want my node to wait another 1 to 2 hours to find peers

[emhane]

if you have peers now, you can share the metrics, it will tell us if you have both incoming and outgoing connections, or only outgoing (which would likely mean you are behind a NAT which doesn't allow you to be discovered based on your ENR + you are using a non-default discovery port so apps won't find you by port scanning either)

[suntzu93]

I didn’t enable metrics during the last restart. I’ve just enabled it now, but I don’t want to restart the node at the moment. I’ll share the metrics once we have a new version and I’ve restarted the node.