Incorrect state root at height 22503505

[0xc0ffeebabeeth]

Describe the bug

Ran into an incorrect state root at height 22503505. Error seems to have happened to ballpark 50% of nodes (which all run the same setup).

Reth was run with the following args:

reth node --ws --ws.port 8546 --http --http.port 8545 --engine.persistence-threshold 0 --engine.memory-block-buffer-target 0

unwinding by 10 blocks is enough to get reth to sync past the bad block

Steps to reproduce

1. "reth node --ws --ws.port 8546 --http --http.port 8545 --engine.persistence-threshold 0 --engine.memory-block-buffer-target 0"
2. Sync past 22503505
3. Get (un)lucky

Invalid hook logs:

22503505_0x389d9a7854df5be972cd49a3c149a0c4c4c2397a41f8547613b71f02b3ded61e.witness.re_executed.json
22503505_0x389d9a7854df5be972cd49a3c149a0c4c4c2397a41f8547613b71f02b3ded61e.trie_updates.re_executed.json
22503505_0x389d9a7854df5be972cd49a3c149a0c4c4c2397a41f8547613b71f02b3ded61e.trie_updates.original.json

22503505_0x389d9a7854df5be972cd49a3c149a0c4c4c2397a41f8547613b71f02b3ded61e.header_state_root.diff.(actually_.diff_but_cant_upload_to_github).json

Node logs

2025-05-18T16:45:07.800103Z  INFO State root task finished state_root=0x36d1f0f43c3c1bd769d1a6798b3e7698def33fb5e3ca49f87628a68f75932ea9 elapsed=8.464142ms
2025-05-18T16:45:07.800122Z  WARN State root task returned incorrect state root state_root=0x36d1f0f43c3c1bd769d1a6798b3e7698def33fb5e3ca49f87628a68f75932ea9 block_state_root=0xb85e3d09a126835ab58b733ee03ea78c6d695a26b680f2da2c469ffbb3a08037
2025-05-18T16:45:07.800451Z  WARN Failed to compute state root in parallel block=NumHash { number: 22503505, hash: 0x389d9a7854df5be972cd49a3c149a0c4c4c2397a41f8547613b71f02b3ded61e } persisting_kind=NotPersisting
2025-05-18T16:45:07.993407Z  WARN Re-executed state root does not match block state root header_state_root=0xb85e3d09a126835ab58b733ee03ea78c6d695a26b680f2da2c469ffbb3a08037 re_executed_root=0x36d1f0f43c3c1bd769d1a6798b3e7698def33fb5e3ca49f87628a68f75932ea9 diff_path=/root/.local/share/reth/mainnet/invalid_block_hooks/witness/22503505_0x389d9a7854df5be972cd49a3c149a0c4c4c2397a41f8547613b71f02b3ded61e.header_state_root.diff
2025-05-18T16:45:08.010508Z  WARN Trie updates mismatch after re-execution original_path=/root/.local/share/reth/mainnet/invalid_block_hooks/witness/22503505_0x389d9a7854df5be972cd49a3c149a0c4c4c2397a41f8547613b71f02b3ded61e.trie_updates.original.json re_executed_path=/root/.local/share/reth/mainnet/invalid_block_hooks/witness/22503505_0x389d9a7854df5be972cd49a3c149a0c4c4c2397a41f8547613b71f02b3ded61e.trie_updates.re_executed.json
2025-05-18T16:45:08.011494Z  WARN Invalid block error on new payload invalid_hash=0x389d9a7854df5be972cd49a3c149a0c4c4c2397a41f8547613b71f02b3ded61e invalid_number=22503505 validation_err=mismatched block state root: got 0x36d1f0f43c3c1bd769d1a6798b3e7698def33fb5e3ca49f87628a68f75932ea9, expected 0xb85e3d09a126835ab58b733ee03ea78c6d695a26b680f2da2c469ffbb3a08037
2025-05-18T16:45:08.011773Z  WARN Bad block with hash invalid_ancestor=BlockWithParent { parent: 0x97549a490e7a36601b392f98fa36998c31b81d461fa1a5895f7215a414996a54, block: NumHash { number: 22503505, hash: 0x389d9a7854df5be972cd49a3c149a0c4c4c2397a41f8547613b71f02b3ded61e } }

Platform(s)

Linux (x86)

Container Type

Not running in a container

What version/commit are you on?

reth Version: 1.3.12
Commit SHA: 6f8e725
Build Timestamp: 2025-04-17T19:30:25.118100950Z
Build Features: asm_keccak,jemalloc
Build Profile: maxperf

What database version are you on?

Current database version: 2
Local database version: 2

Which chain / network are you on?

mainnet

What type of node are you running?

Archive (default)

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• I agree to follow the Code of Conduct

[alextes]

from recent logs in our block validation on the ultra sound relay it looks like i see five other instances across builders in recent week or two although i can't be sure their state root issues are due to the same bug of course.

[jenpaff]

fixed with https://github.com/paradigmxyz/reth/releases/tag/v1.4.8, pls reopen a new issue if you're still experiencing issues