Bug A: off-trie consensus state drift (pending_rewards, total_minted, epoch_state) — silent fork-trigger on any consuming tx

[satyakwok]

Discovered

2026-05-31 testnet h=5817131 stall. val3's StakingOp::ClaimRewards tx applied to a cluster where pending_rewards had drifted across vals → 3-way state fork (val1, val2, val4 each computed different state_root post-claim). Cluster wedged + cascaded into bug B/C (block.hash↔justification mismatch, missing add_block check). Full investigation in handoff notes.

STATE-FP fingerprints at h=5817131:

val1: acc=2f08b0b5...
val2: acc=b9f4202f...
val4: acc=619d9ba0...

All three valid pre-claim (h=5817130: acc=f877a9... matched). Diverged immediately after claim.

Root cause

distribute_reward (mutates stake_registry.pending_rewards), record_block_signatures (mutates liveness tracker), and epoch_manager.record_block are called AFTER bc.add_block(blk) completes in main.rs and libp2p_node.rs apply paths. add_block computes state_root from transactions only — post-block mutations are NOT in the commitment.

Consequence: pending_rewards, liveness stats, total_minted, epoch state all live OUTSIDE state_root. Drift between vals accumulates silently and is undetectable until a tx tries to USE the state (ClaimRewards drain, Unjail self_stake check, AddSelfStake authorization).

Per memory project_total_minted_drift_real_v2_2_17 (2026-05-26):

v2.2.17 boot proved 4 vals all had divergent total_minted (7 SRX delta val1↔val2). B3b heals it.

Same bug class. B3b is a per-epoch reconciliation patch but it's a band-aid — the drift surface is large.

Fix options

Fix	Scope	Risk
F2 (architectural)	Move pending_rewards, total_minted, liveness counters INTO state_trie. Every mutation reflected in state_root. Drift detected immediately at apply.	High — touches consensus state layout, migration required for existing chains
F3 (B3b extension)	Periodic reconciliation: at epoch boundary, all vals must agree on off-trie state via consensus message (system tx). Force-sync to consensus value if divergent.	High — new consensus messages + verification rules
F1 (gate)	Disable mutating staking ops (ClaimRewards/Unjail/AddSelfStake) via fork gate until F2 or F3 ships	Low — operator opt-out but blocks legit users

F2 is the proper architectural fix. F3 is the operational band-aid that ships faster. F1 is the immediate safety valve.

Related immediate fix

• #(this PR) v2.2.23 adds block.hash == j.block_hash check in add_block_impl (gated by strict_justification fork) — closes one downstream consequence (Bug C in 2026-05-31 investigation taxonomy) but does NOT address the root cause (Bug A here).

Acceptance criteria

• Design SIP for F2 (state trie expansion) or F3 (reconciliation system tx)
• Implement chosen fix
• Testnet bake ≥1 week
• Mainnet activation via fork height
• Document upgrade path for chains with existing drift (likely requires snapshot + rebuild)