node seems switched to `chain-tip mode` too early

[AskAlexSharov]

Commit every block buf=3.4GB/512.0MB - this log looks very weird for me. commit every block means buf must be very small (because chain-tip mode does flush to db every block). Maybe we somehow switched to "chain tip mode" too early and it make things slow.

Logs:

erigon[1056464]: [INFO] [04-02|06:42:21.102] [4/6 Execution] Commit every block       blk=69784453 blks=179 blk/s=0.5 txs=14177 tx/s=37 gas/s=6.49M buf=3.4GB/512.0MB stepsInDB=1.00 step=3329.1 inMem=false alloc=24.6GB sys=44.8GB
erigon[1056464]: [INFO] [04-02|06:43:04.515] [4/6 Execution] Commit every block       blk=69784454 blks=2 blk/s=0.0 txs=78 tx/s=1 gas/s=232.35k buf=5.4MB/512.0MB stepsInDB=1.11 step=3329.1 inMem=false alloc=21.3GB sys=44.8GB
erigon[1056464]: [INFO] [04-02|06:43:30.119] [4/6 Execution] Commit every block       blk=69784475 blks=20 blk/s=0.8 txs=1459 tx/s=61 gas/s=10.74M buf=83.8MB/512.0MB stepsInDB=1.11 step=3329.1 inMem=false alloc=22.1GB sys=44.8GB
erigon[1056464]: [INFO] [04-02|06:43:45.763] [4/6 Execution] Commit every block       blk=69784485 blks=11 blk/s=0.7 txs=723 tx/s=46 gas/s=11.32M buf=140.2MB/512.0MB stepsInDB=1.11 step=3329.1 inMem=false alloc=22.6GB sys=44.8GB
erigon[1056464]: [INFO] [04-02|07:23:00.469] [4/6 Execution] Commit every block       blk=69790597 blks=145 blk/s=0.1 txs=9845 tx/s=6 gas/s=1.09M buf=8.4GB/512.0MB stepsInDB=1.11 step=3329.4 inMem=false alloc=22.7GB sys=46.2GB
erigon[1056464]: [INFO] [04-02|07:25:13.286] [4/6 Execution] Commit every block       blk=69790598 blks=2 blk/s=0.0 txs=50 tx/s=0 gas/s=90.38k buf=6.2MB/512.0MB stepsInDB=1.44 step=3329.4 inMem=false alloc=21.1GB sys=46.2GB
erigon[1056464]: [INFO] [04-02|07:25:28.712] [4/6 Execution] Commit every block       blk=69790604 blks=7 blk/s=0.5 txs=431 tx/s=27 gas/s=5.95M buf=40.4MB/512.0MB stepsInDB=1.44 step=3329.4 inMem=false alloc=21.4GB sys=46.2GB
erigon[1056464]: [INFO] [04-02|07:25:48.652] [4/6 Execution] Commit every block       blk=69790621 blks=18 blk/s=0.9 txs=1198 tx/s=60 gas/s=11.65M buf=99.8MB/512.0MB stepsInDB=1.44 step=3329.4 inMem=false alloc=22.1GB sys=46.2GB
erigon[1058006]: [INFO] [04-02|07:56:10.625] [4/6 Execution] Commit every block       blk=69791877 blks=29 blk/s=0.1 txs=2026 tx/s=4 gas/s=685.78k buf=2.6GB/512.0MB stepsInDB=0.44 step=3329.5 inMem=false alloc=29.8GB sys=40.6GB
erigon[1058006]: [INFO] [04-02|08:03:13.894] [4/6 Execution] Commit every block       blk=69792645 blks=50 blk/s=0.2 txs=3286 tx/s=11 gas/s=2.02M buf=1.6GB/512.0MB stepsInDB=0.50 step=3329.5 inMem=false alloc=29.4GB sys=40.6GB

[taratorio]

@AskAlexSharov hey, could this be related to limitedBigJump in update fork choice?

limitedBigJump := e.syncCfg.LoopBlockLimit > 0 && finishProgressBefore > 0 && fcuHeader.Number.Uint64()-finishProgressBefore > uint64(e.syncCfg.LoopBlockLimit-2)
...
initialSync := limitedBigJump
...

Should we have it be fcuHeader.Number.Uint64()-finishProgressBefore > 8 (or some other smaller number) instead of fcuHeader.Number.Uint64()-finishProgressBefore > uint64(e.syncCfg.LoopBlockLimit-2) (LoopBlockLimit=5,000)?

One scenario:

• LoopBlockLimit=5,000 blocks
• a checkpoint is ~1024 blocks (note it varies, can go up to 1024 can see charts on our monitoring)
• a milestone is ~30 block (note it varies, can be less or more and goes up to 160ish sometimes, check charts)
• say Erigon was down for 2 hours
• 2 hours is ~3600 blocks (30 blocks per min x 60 mins x 2 hours)
• after starting Erigon Astrid will first fetch all checkpoints and their blocks since last time and then the same for milestones
• so first it will fetch 3x1024=3,072 blocks for the first 3 checkpoints -> call InsertBlocks -> call UpdateForkChoice (limitedBigJump should be true here, but it is not going to be I think because of the logic)
• note if Erigon was down for longer and there were 10,000 blocks Astrid would try to insert them in 2 batches of roughly 5,000 blocks (based on LoopBlockLimit)
• then after that it will proceed with milestones - so it will fetch remaining e.g. 16 milestones x 30 blocks = 480 blocks -> call InsertBlocks -> call UpdateForkChoice
• after that Astrid will switch to listening for new blocks from devp2p and be in "chain tip mode"

[taratorio]

Or might be because of this? It looks like shouldGenerateChangesetsForLastBlocks only applies to Bor and it looks like it results in calculating commitment and change sets for the last 32 blocks of every batch of blocks (regardless of whether it is close to the tip or not). Not clear why this is the case. In Ethereum we do it once at the end of the batch or when we've reached the commit threshold or for every block when the batch of blocks is <= 32.

[AskAlexSharov]

90% chance it related to shouldGenerateChangesetsForLastBlocks and not related to limitedBigJump. (Because “commit every block” or not - ExecV3 func deciding).

[AskAlexSharov]

“ Not clear why this is the case.” - reason is: if we save diff about jump from N to N+10, then we cannot unwind to N+9.
So, we have to calculate commitment for every block - to produce diff - to have ability unwind to any block.

[taratorio]

Yes. I suspect we can remove this requirement now with Astrid - looks like this change was added in September 2024 when we were still using the previous bor heimdall stage flow and Astrid was enabled by default in January 2025.

My thought process behind this is:

• we only need to calculate the commitment for the last block in the batch with Astrid (like we do for Eth/Gnosis)
• because in "initial sync-to-tip" mode we insert blocks in batches of "checkpoints/milestones" which are finalised and then call execute on them
• so, if there is an unwind, on batch B with checkpoints D, E, F - then we will unwind back to the last block of checkpoint C - checkpoint C is the last checkpoint of batch B-1 (with checkpoints A, B, C as example)
• you can apply the same concept for milestones (they are smaller checkpoints in terms of block range) - we first sync all latest checkpoints, then sync all latest milestones
• because checkpoints and milestones are finalised blocks then this means if we need to unwind - it is because of either bad data (missing state sync events) from heimdall or because of a bug in our execution
• when we finish sync-ing the latest milestone - we are guaranteed to be ~32 blocks away from the tip on average and ~160 blocks away from the tip in the worst case - this is based on these metrics in our monitoring of milestone lengths

so to summarise:

• in "sync-to-tip mode" I suspect we only need commitment & change sets for the last block of the batch only (this will make sync-to-tip quicker and align it with Eth/Gnosis)
• in "tip mode" we need to calculate commitment & changesets for the last 32 blocks on average and last ~160 blocks in worst case (maybe we can have config for this for Bor - ie have changesetSafeRange=256 or 192 for Bor to cover the worst case and keep it 32 for ethereum/gnosis)

[taratorio]

Oh, but I see we have this in exec3 root hash mismatch unwind logic - handleIncorrectRootHashErr:

maxUnwindJumpAllowance = 1000
...
// Binary search, but not too deep
jump := cmp.InRange(1, maxUnwindJumpAllowance, (maxBlockNum-minBlockNum)/2)
unwindTo := maxBlockNum - jump

unwind is limited to last 1000 and also takes some binary search approach

also in serialExecutor.execute we have unwind to prev block only here:

se.u.UnwindTo(txTask.BlockNum-1

so seems like we dont unwind back to prev exec stage progress (last block of prev batch) in reality, so my thought about only needing last block of checkpoint C in batch B-1 might not work as things are currently... I see, it's a tricky one

[VBulikov]

VBulikov added Imp2 as Importance