Syncing Base archive nodes - MERKLE_STAGE_DEFAULT_CLEAN_THRESHOLD potentially too low

[jamesstanleystewart]

Describe the bug

Over the last few days we've had issues with Base Archive nodes being unable to get back to head from snapshot. I've read a few threads similar. We're now running the new i7ie aws instances, which are fast enough to get to head, however I noticed one thing in my travels that may be impacting Base's sync and I thought it worthwhile to get an experts opinion.

During my investigation I found that the MerkleExecute stage has a constant config: MERKLE_STAGE_DEFAULT_CLEAN_THRESHOLD = 5000
https://github.com/paradigmxyz/reth/blob/main/crates/stages/stages/src/stages/merkle.rs#L43C11-L43C47

From what I can tell this causes Reth to rebuild the Merkle data if it's syncing more than 5000 blocks from head. In Base, this takes 2-3 hours on aws nvme ssd (MerkleExecute stage_progress=0.07% stage_eta=2h 44m 13s <- 6000 blocks from head), and on Base, 5000 blocks pass every ~2 hours (compared to 16 hours for ethereum). This causes nodes to get "stuck" just beyond the 5000 block threshold, and if they pass it, MerkleExecute finishes very quickly ( I just watched an i7ie take 13 minutes for 4000 blocks).

I don't know enough about the Merkle data to know for sure if it makes sense to double (or make configurable) that threshold for Base, but one of you will :) Let me know if I've read the situation incorrectly, but I thought it might be useful for others that hit this issue.

Steps to reproduce

Sync a Base archive node from snapshot on an i3en aws instance

Node logs

Platform(s)

No response

Container Type

Kubernetes

What version/commit are you on?

op-reth:v1.2.0

What database version are you on?

Unsure

Which chain / network are you on?

Base Mainnet

What type of node are you running?

Archive (default)

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• I agree to follow the Code of Conduct

[jamesstanleystewart]

Potentially relates to #11306 and #14515

[jenpaff]

@jamesstanleystewart could you specify which snapshot you've used?

[superkeka]

With the current implementation, it is almost impossible to reach the last block (RAID 1 Enterprise NVME SSD).

[exp0nge]

@jamesstanleystewart In addition to which snapshot, do you have timings per stage?

Would also be helpful to have all the flags, memory, cpu you are using on op-reth & op-node.

[jamesstanleystewart]

We've been running Reth for base for 6 months or so now, I think we most likely used a publicnode snapshot originally, but now use our own internal ones.

As for timings, when the node struggles to get in sync, MerkleExecute takes 90%+ of the ~3 hour sync time. Once it crosses the 5000 block threshold, MerkleExecute takes something like 50% of the total (much shorter - 15 mins or less) sync time.

To be clear, moving to the new AWS i7ie family has provided us enough extra disk performance to get into sync. I assume this is because it can do the Merkle rebuild faster and cross the 5000 block chasm.

CPU/Memory - previously they ran on i3en.(2 or 3 we tried both)xlarge machines, now they run on i7ie.2xlarge successfully.

op-reth:

      --datadir=/data                                                                                                                                                                                                                                
      --ipcpath=/data/reth.ipc                                                                                                                                                                                                                       
     <http and ws api args>                                                                                                                                                                                                                               
      --max-inbound-peers=50                                                                                                                                                                                                                         
      --max-outbound-peers=50                                                                                                                                                                                                                        
      --rollup.disable-tx-pool-gossip                                                                                                                                                                                                                
      --rollup.sequencer-http=https://mainnet-sequencer.base.org                                                                                                                                                                                     
      --chain=base                                                                                                                                                                                                                                   
      --authrpc.addr=127.0.0.1                                                                                                                                                                                                                       
      --authrpc.port=8551                                                                                                                                                                                                                            
      --authrpc.jwtsecret=/tmp/op/jwt-secret.txt                                                                                                                                                                                                     
      --rpc.gascap=500000000                                                                                                                                                                                                                         
      --rpc.max-response-size=500                                                                                                                                                                                                                    
      --metrics=0.0.0.0:9090  

op-node

      OP_NODE_L2_ENGINE_KIND:                 reth                                                                                                                                                                                                   
      OP_NODE_RPC_ADDR:                       0.0.0.0                                                                                                                                                                                                
      OP_NODE_RPC_PORT:                       9545                                                                                                                                                                                                   
      OP_NODE_P2P_BOOTNODES:                 <some bootnoodes>                                                                                             
      OP_NODE_P2P_LISTEN_IP:                  0.0.0.0                                                                                                                                                                                                
      OP_NODE_P2P_LISTEN_TCP_PORT:            30303                                                                                                                                                                                                  
      OP_NODE_P2P_LISTEN_UDP_PORT:            30303                                                                                                                                                                                                  
      OP_NODE_P2P_ADVERTISE_TCP:              30303                                                                                                                                                                                                  
      OP_NODE_P2P_ADVERTISE_UDP:              30303                                                                                                                                                                                                  
      OP_NODE_P2P_PEERSTORE_PATH:             /p2p/peers                                                                                                                                                                                             
      OP_NODE_P2P_DISCOVERY_PATH:             /p2p/discovery                                                                                                                                                                                         
      OP_NODE_P2P_DISABLE:                    false                                                                                                                                                                                                  
      OP_NODE_P2P_NO_DISCOVERY:               false                                                                                                                                                                                                  
      OP_NODE_METRICS_ENABLED:                true                                                                                                                                                                                                   
      OP_NODE_METRICS_ADDR:                   0.0.0.0                                                                                                                                                                                                
      OP_NODE_METRICS_PORT:                   9091                                                                                                                                                                                                   
      OP_NODE_L2_ENGINE_AUTH:                 /tmp/op/jwt-secret.txt                                                                                                                                                                                 
      OP_NODE_VERIFIER_L1_CONFS:              0                                                                                                                                                                                                      
      OP_NODE_LOG_FORMAT:                     json                                                                                                                                                                                                   
      OP_NODE_LOG_LEVEL:                      info                                                                                                                                                                                                   
      OP_NODE_PPROF_ENABLED:                  false                                                                                                                                                                                                  
      OP_NODE_PPROF_PORT:                     6666                                                                                                                                                                                                   
      OP_NODE_PPROF_ADDR:                     0.0.0.0                                                                                                                                                                                                
      OP_NODE_L2_ENGINE_RPC:                  http://localhost:8551                                                                                                                                                                                  
      OP_NODE_NETWORK:                        base-mainnet                                                                                                                                                                                           
      OP_NODE_L1_ETH_RPC:                     <our l1 node>                                                                                                                           
      OP_NODE_L1_TRUST_RPC:                   false                                                                                                                                                                                                  
      OP_NODE_L1_BEACON:                      <our l1 node>                                                                                                                           
      OP_NODE_L1_BEACON_IGNORE:               false                                                                                                                                                                                                  
      OP_NODE_ROLLUP_LOAD_PROTOCOL_VERSIONS:  true                                                                                                                                                                                                   
      OP_NODE_SYNCMODE:                       execution-layer 

[jamesstanleystewart]

here's a chart showing the block number and sync improvement when we moved to i7ie