fix(flat): stop the trie warmer poisoning reads with a cached Unknown - #12793
Conversation
#12429 restricted the trie warmer to persistence + the trie node cache and stopped it reading the recyclable in-memory `_snapshots` tier, which closed a recycle-under-reader race. But the warmer still wrote the result of its now narrower lookup into the shared `_transientResource` via `GetOrAdd`, including the miss: a node that lives only in `_snapshots` (recent, not-yet-persisted blocks) resolved to `new TrieNode(Unknown, hash)` and was cached under its path. A normal read consults `_transientResource` before `_snapshots`, and the cache matches on `Keccak == hash`, so the cached Unknown was served instead of the real node. The affected subtree read as empty and the block's state root diverged (observed on Flat sync, block 25681470: InvalidStateRoot). Cache only a real persistence hit into the transient; on a miss return an unshared Unknown so the normal read falls through to `_snapshots`. The recycle race stays fixed - the warmer still never reads `_snapshots`. In-memory nodes are already hot and do not need warming. Regression tests warm a committed (in-memory-only) node's path, then assert a subsequent normal read of the same path/hash still resolves the committed node rather than the warmer's Unknown, for both state and storage.
|
Claude finished @AnkushinDaniil's task in 4m 23s —— View job Review: fix(flat) — stop the trie warmer poisoning reads with a cached
|
|
|
…ve test Reorder Trie_warmer_reads_persistence_only... so the warmer reads run before the normal reads: on the pre-fix code the warmer poisons the transient and the trailing normal reads observe the cached Unknown, so the existing test now covers the regression in place. Drop the two dedicated tests it subsumes. Annotate TryFindStorageNodeInPersistence with [NotNullWhen(true)], matching its state counterpart, and drop the redundant node is not null checks at both call sites.
|
@claude review |
|
Claude finished @AnkushinDaniil's task in 3m 14s —— View job Re-review: fix(flat) — stop the trie warmer poisoning reads with a cached
|
| Previous finding | Status |
|---|---|
Medium — stale comment at SnapshotBundleWarmerTests.cs:76-77 (also raised by @benaadams) |
Fixed: comment deleted, normalStateRead/normalStorageRead moved below the four warmer reads |
| Low — the two new tests duplicated existing setup | Fixed: both removed; the existing test now carries the regression (net −3 lines in the test file) |
Low — redundant && node is not null |
Fixed: [NotNullWhen(true)] on TryFindStorageNodeInPersistence (SnapshotBundle.cs:344), both call sites (:316, :339) simplified |
Re-verification of the fix
- Poisoning channel is closed.
ChildCache.TryGetmatches onentry.node.Keccak == hash(TrieNodeCache.cs:255), so a cachedTrieNode(Unknown, hash)is the only entry that can shadow a real node at that path; every other entry either genuinely matches the requested hash (content-addressed → correct node) or is skipped and the read falls through to_snapshots. Caching only real persistence hits removes it. - The follow-on corruption is removed too.
TryLoadStateRlpis path-keyed and ignoreshash(ReadOnlySnapshotBundle.cs:193-206), so onmasterthe sharedUnknownwas resolved from the persisted RLP at that path while keepingKeccak == hash— a cached entry holding stale content under a matching hash. An unsharedUnknowncannot be observed by a normal read at all. - Adapter contract unchanged.
FindCachedOrUnknownthrowsNodeHashMismatchExceptionwhennode.Keccak != hash(StateTrieStoreAdapter.cs:50-51); the miss path returnsUnknown(hash), exactly whatGetOrAddreturned before (it overwrote and returned the passed node). No new throw path, and the hit path still routes throughGetOrAdd*with identical semantics — the only behavioural deltas are (a) misses not cached, (b) misses unshared. - fix(flat): guard the trie-warmer against a TransientResource recycle race (storage reads as 0x00) #12429 invariant preserved. Neither
WarmUpStateNodenorWarmUpStorageNodetouches_snapshots;TryLeaseTransientResource(lease + ABA + disposed re-check) is untouched. Returning an unsharedUnknownis also strictly safer for concurrency: warmer and reader can no longer race toResolveNodethe same instance. - Test is still a genuine positive control.
CollectAndApplySnapshotsets_trieChanged = falseand callsSwapTransientResource(SnapshotBundle.cs:600-601), so the post-commit normal read skips_changedStateNodesand consults the same transient the warmer just wrote to. OnmasternormalStateReadtherefore returns the poisonedUnknownandIs.SameAs(committedNode)fails; here it falls through to_snapshots. The reorder covers both state and storage in one test, so the coverage is not weaker than the two deleted tests.
Remaining minor items (all optional)
- Low — negative results are no longer memoised (carried over, unchanged by
f013271). A repeat warm of a path absent from persistence now re-queries_trieNodeCache+ the persistence reader and allocates a freshUnknowneach time. This is the common case for the top of the trie during Flat sync (upper branches change every block and live only in_snapshots), so it is the one place worth a number: an expbflatrun attached to the PR would settle it. Correctness clearly wins regardless. - Nit — the sibling annotation is still missing.
DoTryFindStorageNodeExternal(SnapshotBundle.cs:359) has the same shape as the method you just annotated but still lacks[NotNullWhen(true)], which keeps the redundant&& node is not nullat:296. Untouched by this diff, so leaving it is the right call per AGENTS.md ("don't touch unrelated code") — noting it only so it isn't read as an inconsistency. Fix this → - Nit — two still-accurate comments went with the reorder (
// The warmer still returns nodes that are in persistence (state and storage).and thenormalXxxReadrationale). The header comment atSnapshotBundleWarmerTests.cs:46-49already states both facts, so nothing is lost; no action needed.
Notes
dotnet testis not in my allowed tools (the call was blocked again), so this remains a static review — I did not executeNethermind.State.Flat.Test. CI: 4 checks green, 10 still in progress at the time of writing, includingNethermind tests (Flat DB), which is the one that matters here; 0 failures so far.- Two pre-existing properties I re-confirmed and am not raising as findings, listed so they aren't mistaken for gaps in this fix: (1) both
TryFind*InPersistenceand_changedStateNodesare path-keyed and unverified, so a warm hit can be a stale-hash node that the adapter then rejects withNodeHashMismatchException; (2)GetOrAddoverwrites onKeccakmismatch, so such a node can evict a fresher transient entry — harmless, since every transient read is hash-checked and in-block reads are served from_changedStateNodes.
Looks good to merge from my side once the Flat DB suite reports green.
· branch daniil/fix-warmer-unknown-poison
|
Sync verification passes properly |
…ive reads The trie warmer re-warms high-fanout upper nodes on every prewarm traversal. Those nodes live in the recyclable _snapshots, not persistence, so #12793 (which stopped caching the warmer's persistence miss) made every visit repeat the full TryFind...InPersistence lookup, costing ~+3-5% AVG block processing on FlatDB. Restore the negative cache: the warmer caches the miss as a bare Unknown sentinel again, so repeat visits short-circuit. Live reads (FindStateNodeOrUnknown / FindStorageNodeOrUnknown) now skip that sentinel and fall through to the authoritative _snapshots/persistence lookup, so the cached miss can no longer poison a real read (the InvalidStateRoot #12793 fixed). The transient is a pure cache - authoritative nodes live in _changedStateNodes/_snapshots/persistence - so evicting a slot to a sentinel is safe.
… shared TrieNodeCache Review follow-up. The negative-cache sentinel is written into the recyclable transient, which is later handed wholesale to the process-wide TrieNodeCache at commit (FlatDbManager -> TrieNodeCache.Add). Add copied every non-null node with no sentinel filter, so an unchanged node's sentinel was promoted and then returned by the unguarded _trieNodeCache read in DoFindStateNodeExternal / DoTryFindStorageNodeExternal one block later - the same InvalidStateRoot class #12793 fixed, one indirection removed. Filter the sentinel in TrieNodeCache.Add so it never enters the shared cache (this also keeps sentinels from consuming the cache budget). The sentinel discriminator is hoisted to a single TrieNodeCache.IsWarmerMiss used by both the Add filter and the SnapshotBundle live-read guards. Regression test uses a real TrieNodeCache and asserts a promoted miss no longer masks the real node.
…dicate Address the re-review: add a regression test that a repeated warmer visit to a missing path is served from the transient negative cache instead of re-probing persistence (the +3-5% AVG behaviour #12793 dropped). Rename IsWarmerMiss to the source-neutral IsPlaceholder since the TrieNodeCache.Add filter also gates placeholders arriving from the commit path, and document both sources.
…ive reads (#12877) * fix(flat): restore the trie warmer negative cache without poisoning live reads The trie warmer re-warms high-fanout upper nodes on every prewarm traversal. Those nodes live in the recyclable _snapshots, not persistence, so #12793 (which stopped caching the warmer's persistence miss) made every visit repeat the full TryFind...InPersistence lookup, costing ~+3-5% AVG block processing on FlatDB. Restore the negative cache: the warmer caches the miss as a bare Unknown sentinel again, so repeat visits short-circuit. Live reads (FindStateNodeOrUnknown / FindStorageNodeOrUnknown) now skip that sentinel and fall through to the authoritative _snapshots/persistence lookup, so the cached miss can no longer poison a real read (the InvalidStateRoot #12793 fixed). The transient is a pure cache - authoritative nodes live in _changedStateNodes/_snapshots/persistence - so evicting a slot to a sentinel is safe. * fix(flat): stop the warmer-miss sentinel from being promoted into the shared TrieNodeCache Review follow-up. The negative-cache sentinel is written into the recyclable transient, which is later handed wholesale to the process-wide TrieNodeCache at commit (FlatDbManager -> TrieNodeCache.Add). Add copied every non-null node with no sentinel filter, so an unchanged node's sentinel was promoted and then returned by the unguarded _trieNodeCache read in DoFindStateNodeExternal / DoTryFindStorageNodeExternal one block later - the same InvalidStateRoot class #12793 fixed, one indirection removed. Filter the sentinel in TrieNodeCache.Add so it never enters the shared cache (this also keeps sentinels from consuming the cache budget). The sentinel discriminator is hoisted to a single TrieNodeCache.IsWarmerMiss used by both the Add filter and the SnapshotBundle live-read guards. Regression test uses a real TrieNodeCache and asserts a promoted miss no longer masks the real node. * test(flat): pin the warmer negative cache; rename the placeholder predicate Address the re-review: add a regression test that a repeated warmer visit to a missing path is served from the transient negative cache instead of re-probing persistence (the +3-5% AVG behaviour #12793 dropped). Rename IsWarmerMiss to the source-neutral IsPlaceholder since the TrieNodeCache.Add filter also gates placeholders arriving from the commit path, and document both sources. * test(flat): fold the counting cache into NullTrieNodeCache
…cture #12877 restored the warmer's negative cache by writing the miss placeholder (NodeType.Unknown, empty RLP) into the shared _transientResource.Nodes and tried to keep it safe with an IsPlaceholder predicate at every live-read site and at the shared-cache promotion. That coexistence is the defect: the same placeholder instance sits in the cache that live reads consult, so a warmer resolve racing a live read reopens the InvalidStateRoot window #12793 closed (reproduced on Flat sync at live head, mainnet and gnosis; HalfPath unaffected). Remove the coexistence instead of guarding it. Warmer misses now go to a dedicated TransientResource.MissNodes cache that no live read ever consults and that TrieNodeCache.Add never promotes. Nodes therefore holds only real nodes, so the IsPlaceholder guards and the predicate are dead code and are dropped. The negative cache is preserved (a repeated warmer miss is still served without re-probing persistence), so the #12793 perf recovery stands.
…iant Reword the XML doc to claim only what is verified — no live read consults MissNodes and TrieNodeCache.Add never promotes it — instead of the stronger 'a stale miss can never reach the state-root computation', which does not hold: the placeholder also escapes as the warmer return value and is later resolved by a live read through persistence (by hash, correct by construction, matching #12793 and pre-dating this PR).
…12951) * fix(flat): isolate the trie warmer negative cache in a dedicated structure #12877 restored the warmer's negative cache by writing the miss placeholder (NodeType.Unknown, empty RLP) into the shared _transientResource.Nodes and tried to keep it safe with an IsPlaceholder predicate at every live-read site and at the shared-cache promotion. That coexistence is the defect: the same placeholder instance sits in the cache that live reads consult, so a warmer resolve racing a live read reopens the InvalidStateRoot window #12793 closed (reproduced on Flat sync at live head, mainnet and gnosis; HalfPath unaffected). Remove the coexistence instead of guarding it. Warmer misses now go to a dedicated TransientResource.MissNodes cache that no live read ever consults and that TrieNodeCache.Add never promotes. Nodes therefore holds only real nodes, so the IsPlaceholder guards and the predicate are dead code and are dropped. The negative cache is preserved (a repeated warmer miss is still served without re-probing persistence), so the #12793 perf recovery stands. * docs(flat): scope the MissNodes summary to the proven isolation invariant Reword the XML doc to claim only what is verified — no live read consults MissNodes and TrieNodeCache.Add never promotes it — instead of the stronger 'a stale miss can never reach the state-root computation', which does not hold: the placeholder also escapes as the warmer return value and is later resolved by a live read through persistence (by hash, correct by construction, matching #12793 and pre-dating this PR). * perf(flat): publish the trie warmer's persistence reads into the transient Isolating the warmer's negative cache in `MissNodes` is what makes #12924 correct, but it also strands the warmer's work. A warmer miss means the node is absent from the in-memory structures, not from persistence: the warmer returns a placeholder and the traversal resolves it through `ReadOnlySnapshotBundle.TryLoadStateRlp`, i.e. a real persistence read. That placeholder lives in `MissNodes`, which neither live reads nor `TrieNodeCache.Add` consult, so every node the warmer pulls off disk is fetched, decoded and then discarded at block end while the live path repeats the same read. Publish what the warmer read: a private `TrieNode` built from the RLP, decoded before it is stored in `TransientResource.Nodes`, never handed to the warmer. The block's live reads and the cross-block `TrieNodeCache` reuse the read; no reader can observe a node mid-resolution, which is the hazard that made publishing the warmer's own instance unsafe (`IsPlaceholder` was a predicate over `NodeType`/`FullRlp` while another thread wrote both). Persistence read count is unchanged: the publish is a side effect of the read the warmer already performs, not an extra one. * fix(flat): verify the warmer's persistence RLP hashes to the requested node Review catch on the previous commit, and a real defect in it. `ReadOnlySnapshotBundle.TryLoadStateRlp`/`TryLoadStorageRlp` take a `hash` but never use it: the read is keyed by path alone. Publishing those bytes as `new TrieNode(NodeType.Unknown, hash, rlp)` therefore *stamps* the requested hash onto whatever node currently sits at that path, and the warmer can ask for a (path, hash) pair the pinned persistence view does not hold - it reaches newer pairs through the cross-block `TrieNodeCache`, and on a multi-block branch through a post-commit storage root. Every reader-side guard (`ChildCache.TryGet`, `TrieNodeCache.TryGet`, `FindCachedOrUnknown`'s `NodeHashMismatchException`) compares against the node's claimed `Keccak`, so all of them would be satisfied by construction and a stale node would be served to live reads and promoted cross-block: `InvalidStateRoot`, the symptom this whole chain keeps producing. Verify the binding instead of asserting it - publish only when the RLP hashes to the requested hash - and state the invariant in the `<remarks>`. Also from the review: - `TryResolveNode` instead of `ResolveNode`, so a publish-side decode failure cannot change the outcome or exception type of the read it piggybacks on. - Pick the `ReadOnlySpan<byte>` ctor explicitly so overload resolution cannot silently flip between copying and aliasing the caller's array. - Document the costs honestly: a Keccak per published node, the RLP copy, the dual `MissNodes`/`Nodes` slot and its effect on the regrow decision, and the `GetOrAdd`-replaces-on-mismatch interaction with a racing trie commit. - Tests now drive the real warmer adapter path rather than calling the bundle method directly, dropping an assertion that held trivially, and cover the mismatch case: RLP for node A returned at the path where the warmer asks for node B must publish nothing. Verified non-vacuous - removing the hash guard fails both cases. * test(flat): cover same-transient warmer miss isolation * test(flat): cover resolved warmer miss isolation * test(flat): cover storage warmer miss reuse * Optimize safe trie warmer cache promotion * Avoid copying published warmer node RLP * Synchronize shared trie warmer resolution * fix(flat): treat a stale warmer read as a miss instead of an error A path-keyed persistence read can answer a warmer read with another version of the node at that path, so the RLP hash check is expected to fail while the warmer runs ahead of or behind the live reads. Resolve warm-up traversal with TryResolveNode so such a node just ends the traversal unresolved and unpublished, instead of throwing out to the warmer processor as an ERROR. * fix(flat): drop the unused Trie.Pruning using directive Left over from the removed PublishWarmedNode; every Metrics reference in the file is fully qualified, so it fails Code Lint with IDE0005. Co-authored-by: Kamil Chodoła <43241881+kamilchodola@users.noreply.github.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(flat): assert a live read stays Unknown while a warmer node resolves Also drop the test subsumed by its parameterized successor and trim the comments this PR added to one-liners. * fix(flat): never Sleep(1) on a block thread waiting for a warmer resolution A live reader that reaches a warmer-owned node through a shared parent's child slot waits for the warmer's in-flight persistence read; SpinWait's Sleep(1) escalation would add up to a millisecond of tail latency there, so spin and yield only. Also cover the retirement drain with a regression test and drop a vacuous assertion. --------- Co-authored-by: AnkushinDaniil <ankushin.daniil42@gmail.com> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Changes
Regression from #12429. That PR correctly stopped the trie warmer reading the recyclable in-memory
_snapshotstier (closing a recycle-under-reader race), but the warmer still wrote the result of its now-narrower lookup — including a miss — into the shared_transientResourceviaGetOrAdd.A node that lives only in
_snapshots(recent, not-yet-persisted blocks) is not in persistence or the trie node cache, so the warmer resolved it tonew TrieNode(Unknown, hash)and cached that Unknown under its path. A normal read consults_transientResourcebefore_snapshots, and the transient cache matches onKeccak == hash, so the poisoned Unknown was served instead of the real node. The subtree read as empty and the state root diverged.Observed on Flat sync, block
25681470:InvalidStateRoot: Expected 0xe6a4b280…ba748, got 0xd3a9f4b3…fd50.WarmUpStateNode/WarmUpStorageNode: cache only a real persistence hit into the transient; on a miss return an unsharedUnknownso a normal read falls through to_snapshots._snapshots. In-memory nodes are already hot and do not need warming.Unknown. Both fail on currentmaster(returnUnknown) and pass with this fix.Types of changes
Testing
Requires testing
If yes, did you write tests?
Notes on testing
Nethermind.State.Flat.Testpasses locally. Positive control: both new tests fail onmaster(normalReadreturnsUnknown) and pass with the fix. The one unrelated local failure (ArenaWriter_…FrontierDelta, a global static metric gauge) reproduces on a cleanmastercheckout with none of these changes.Documentation
Requires documentation update
Requires explanation in Release Notes