Skip to content

eth_call - perf improvements - #11965

Merged
svlachakis merged 130 commits into
masterfrom
evm-stream
Jun 30, 2026
Merged

eth_call - perf improvements#11965
svlachakis merged 130 commits into
masterfrom
evm-stream

Conversation

@svlachakis

@svlachakis svlachakis commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Summary

A set of EVM optimizations targeting eth_call and the rest of the cancelable RPC call/simulation surface (estimateGas, createAccessList, simulate).

Behaviour-preserving — differential-tested to produce identical gas and output — and scoped to cancelable frames, so block processing is unaffected. No consensus change.

1. Instruction-stream interpreter — on by default, scoped to cancelable RPC frames

Runs hot, repeatedly-executed bytecode over a preprocessed stream instead of the raw loop:

  • Per-block static gas charged once.
  • Pre-decoded PUSH immediates.
  • Fused PUSH+op pairs.
  • Static PUSH2+JUMP/JUMPI resolved to a target entry.

Engages only on cancelable (eth_call / estimateGas / createAccessList / simulate), non-tracing, tip-fork frames.

Block processing runs the unchanged dispatch and never builds or runs a stream.

Built async on the thread pool (a dedicated IThreadPoolWorkItem) after a fixed threshold of 4 executions; lock-free publish; falls back to a metered raw-code loop so gas and failure semantics stay exact.

~7–8% over the bytecode loop on the heavy eth_call corpus (more under concurrent load).

2. Per-fork specialized dispatch

The fork's opcode-gating flags (EIP-145 shift opcodes, EIP-3855 PUSH0) are lifted to compile-time IFlag generic type args, so the hot dispatch loop (RunByteCodeCore) is specialized per tip fork and the JIT folds away the per-op spec branches.

The direct-dispatch switch over hot opcodes runs on the cancelable path; block processing takes the plain function-pointer table (master-parity), avoiding the switch's I-cache cost on its diverse opcode mix.

  • Program counter kept register-resident across the dispatch loop: the loop reads/advances a loop-local int pc and handlers take ref pc instead of ref programCounter.
  • Taking the address of the field forced the JIT to stack-spill and reload it every opcode; the temp keeps it in a register on the fetch/bounds/increment critical path (written back to the field only at frame exit).
  • Applied to both RunByteCodeCore and RunStream.

Isolated arithmetic microbenchmark:

515.7 ns → 450.8 ns (-12.6%)

3. Per-thread EVM memory pool EvmPooledMemory rents/returns its backing buffer from a per-thread pool, removing cross-thread ArrayPool contention on the allocation-heavy memory path.

#11991

4. Flat-state read caching

#12035

5. Keccak cache tweak

Adjustment to the set-associative KeccakCache on the hashing path.
#12042

6. Per-contract storage memo & Skip cell hashing on read-mostly executions

(PersistentStorageProvider).** Consecutive SLOADs overwhelmingly hit the same contract; a one-entry memo (_lastStorageAddress_lastStorage) removes the _storages dictionary lookup from the per-SLOAD path. Invalidated wherever _storages is cleared, so the pooled per-contract state is never reachable after return.

The intra-block cache only holds journaled writes. On read-mostly executions (e.g. eth_call) it is empty, so a _intraBlockCache.Count != 0 guard skips hashing the 52-byte storage cell on every SLOAD before the lookup that would miss anyway.

#12043

ID PR med Δ vs master Δ vs Other Client PR p99 Δ p99 vs master
19149 128.5 −62.6 (−33%) −10.7 154.5 −92.2
19143 143.1 −53.2 (−27%) +6.3 171.4 −90.9
19146 123.2 −71.9 (−37%) −14.0 179.9 −69.6
19145 132.5 −64.0 (−33%) −4.1 158.7 −107.4
19142 139.4 −54.1 (−28%) +3.9 158.5 −89.0
25754 142.5 −60.4 (−30%) +2.7 183.8 −68.2
19148 117.3 −78.3 (−40%) −13.5 137.6 −95.2
25755 137.7 −62.4 (−31%) −0.3 175.7 −64.9
39853 149.3 −65.2 (−30%) +2.2 196.4 −52.4
39854 147.8 −70.4 (−32%) −1.5 191.6 −56.7
19144 120.9 −81.7 (−40%) −15.8 156.7 −94.8
39856 133.8 −70.7 (−35%) −6.2 158.4 −93.6
19147 129.3 −60.5 (−32%) −2.4 172.1 −70.3
25756 125.8 −71.2 (−36%) −12.6 161.8 −82.2
33042 126.3 −68.6 (−35%) −13.2 154.4 −99.9
39862 141.7 −50.1 (−26%) +2.1 179.3 −58.8
11153 85.1 −48.3 (−36%) −10.4 133.1 −49.5
11152 75.3 −31.1 (−29%) +0.4 90.6 −34.7
33045 124.5 −65.0 (−34%) −18.9 175.9 −53.5
39860 123.9 −73.6 (−37%) −18.1 150.1 −102.7
33046 142.1 −44.9 (−24%) +2.0 176.8 −53.5
25761 133.4 −48.1 (−27%) −7.7 705.9 ⚠ (jitter)
metric master → PR Δ vs Other Client
median 190.0 → 128.3 −61.7ms (−32%) −5.9ms (−4.4%, PR wins)
p99 237.7 → ~163 −74.7ms (−31%) +10ms (+7%, other client tighter tail)

At 60 RPS

Metric stream-on stream-off Result
ActualRPS 59.9 59.7 both hold
Median 171.7 296.6 42% lower
Mean 205.6 349.3 41% lower
P95 388 718 46% lower
P99 494 851 42% lower

At 65 RPS

Metric stream-on stream-off Result
Median 228.4 336.1 32% lower
Mean 295.8 378.9 22% lower
P99 768 919 17% lower

Types of changes

What types of changes does your code introduce?

  • Bugfix (a non-breaking change that fixes an issue)
  • New feature (a non-breaking change that adds functionality)
  • Breaking change (a change that causes existing functionality not to work as expected)
  • Optimization
  • Refactoring
  • Documentation update
  • Build-related changes
  • Other: Description

Testing

Requires testing

  • Yes
  • No

If yes, did you write tests?

  • Yes
  • No

@claude

claude Bot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Claude finished @svlachakis's task in 8m 33s —— View job


PR Review: EVM Stream

Tasks

  • Gather context and understand the PR
  • Read changed files and rule files
  • Review correctness, edge cases, regressions
  • Review security implications
  • Review performance (hot-path EVM)
  • Adherence to repo rules
  • Post inline comments for specific issues
  • Write final summary

Summary

This PR introduces a preprocessed instruction stream interpreter for the EVM: a first pass analyzes bytecode into basic blocks with summed static gas and pre-decoded PUSH immediates; a second pass executes the stream with block-level gas charging, gas-free in-block cores, fused PUSH+op pairs, and a metered fallback for low-gas or mid-block landings. It also adds a compile-time spec fingerprinting system (IEvmSpec / EvmSpecFingerprint) to select specialized dispatch for Cancun/Prague/Osaka on tip forks.

The architecture is sound and the safety invariants (JUMPDEST as solo block, pcToEntry invalidation inside fused pairs, fingerprint-guarded engagement) are well-reasoned. The differential tests and EvmSpecGuardTests provide a good automated safety net. However there are issues that must be resolved before merge.


Findings

High

1. Diagnostic code must be removed before mergeVirtualMachine.Stream.cs:31-36 and VirtualMachine.cs:1325-1326

StreamInterpreter.Diagnose / StreamInterpreter.Log calls File.AppendAllText on every instruction when enabled, and the code explicitly reads "TEMPORARY divergence diagnostics — remove before merge." The four if (StreamInterpreter.Diagnose) guards in the hot dispatch loop are conditional branches that add overhead even when disabled. The hardcoded /tmp/ paths also break on Windows. This entire block — Diagnose, Log, all four call sites, the NETHERMIND_STREAM_DIAG env var — needs to be deleted before merge.

2. ReadOnlySnapshotBundle memoization belongs in a separate PRReadOnlySnapshotBundle.cs:28-46

The addition of _accountPersistenceMemo, _slotPersistenceMemo, PersistenceMemoMaxEntries, and their associated read/write paths is an independent feature modifying Nethermind.State.Flat. It has nothing to do with the EVM stream, is not reflected in the PR title or description, and forces reviewers of one subsystem to also review the other. [Inline comment posted.]


Medium

3. Magic number 10 in fused-constant thresholdVirtualMachine.Stream.cs:139

entry.Advance > 10 encodes "PUSH8+op = 10 bytes → inline ulong; PUSH9+op = 11 bytes → pool index." If this threshold were wrong by 1 it would silently read the wrong source. A named constant InlinePushMaxAdvance = 1 + 8 + 1 with a derivation comment eliminates the silent-failure risk. [Inline comment posted.]

4. StreamInterpreter.Enabled should be volatileVirtualMachine.Stream.cs:22

A test thread flipping Enabled via try/finally (as the differential tests do) may not be immediately visible to a frame-executing thread without a memory barrier. volatile costs nothing and makes the cross-thread contract explicit. [Inline comment posted.]


Low

5. FramesExecuted++ is non-atomicVirtualMachine.Stream.cs:65

long increment is non-atomic on 32-bit platforms. Interlocked.Increment costs nothing relative to a frame and removes the platform caveat. [Inline comment posted.]

6. PragueEvmSpec dispatch registration is unexplainedIEvmSpec.cs:204

Prague has the same IEvmSpec fingerprint as Cancun (Prague EIPs add no new opcode gates), so Prague frames correctly take the Cancun dispatch path. The guard test validates this. However, the struct has no comment explaining why it is absent from the dispatch switch, which will confuse the next developer. [Inline comment posted.]

7. PR description is empty

No changes listed, no type-of-change checkboxes ticked, no testing notes. This is required by the PR template (it drives automatic labeling and release notes) and should be filled in before merge.

8. No EF consensus test run with stream enabled

The differential unit tests are synthetic. Running the Ethereum Foundation state test suite with NETHERMIND_EVM_STREAM=1 before merge would provide much higher confidence that no consensus edge case was missed.


Correctness Assessment

The core gas accounting is correct: block gas is the exact sum of individual in-block instruction costs, so whether charged once at BlockFirst or per-instruction in RunMeteredSegment, totals are identical. The operand order in FusedConstBinaryCore and FusedConstShiftCore matches Math2ParamCore / ShiftCore exactly. The pcToEntry InvalidEntry sentinels for fused-pair interiors prevent any jump from landing inside a pair. The metered fallback correctly handles mid-block landings from fused table handlers (e.g., PUSH2+JUMP landing one instruction into a block). No consensus bugs found by inspection.

Comment thread src/Nethermind/Nethermind.Evm/VirtualMachine.Stream.cs Outdated
Comment thread src/Nethermind/Nethermind.Evm/VirtualMachine.Stream.cs Outdated
Comment thread src/Nethermind/Nethermind.Evm/VirtualMachine.Stream.cs Outdated
Comment thread src/Nethermind/Nethermind.Evm/VirtualMachine.Stream.cs Outdated
Comment thread src/Nethermind/Nethermind.Evm/Instructions/IEvmSpec.cs Outdated
Comment thread src/Nethermind/Nethermind.State.Flat/ReadOnlySnapshotBundle.cs Outdated
@claude

claude Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Claude finished @svlachakis's task in 2m 31s —— View job


PR Review: EVM Stream (Tenth Pass — focused on benaadams' findings)

Re-reviewed against benaadams' comment. All of his findings have been addressed in the current branch HEAD (cfc8981).


benaadams findings — status

Finding Severity Status
Out-of-gas never detected on stream paths (ulong < 0 always false) High Fixed
Fork-gating comment contradicts actual gate Medium ✅ Fixed (one trailing stale comment remained → inline)
PUSH9+ excluded comment inaccurate Low ✅ Fixed
InstructionStreamCache memory footprint note Low ✅ Added
StaticJump transiently-stale programCounter note Low ✅ Added

High — out-of-gas detection (verified resolved)

All five sites benaadams listed now correctly use TGasPolicy.IsOutOfGas(in gas) instead of the dead GetRemainingGas(in gas) < 0 (which is always false for a ulong):

Site VirtualMachine.Stream.cs line
fused StaticJump 253
fused StaticJumpI 266
MSTORE / MLOAD / MCOPY fast path 317
general boundary-op epilogue 347
RunMeteredSegment per-op 485

This matches the canonical pattern: EthereumGasPolicy.Consume clamps Value to 0 and sets the separate OutOfGas flag (EthereumGasPolicy.cs:110-121), which IsOutOfGas reads (:211) — the same flag VirtualMachine.DispatchSpecialized.cs:206 checks. The precharge guard at :79 (GetRemainingGas(in gas) >= cost, both unsigned) is correct and was never affected.

Note: the git history shows 66cb289 "revert to GetRemainingGas < 0" did briefly reintroduce the broken form, but the later 94d3a83 reverted it back to IsOutOfGas, and HEAD is correct. Worth being careful no future rebase resurrects 66cb289.

Regression test added (InstructionStreamTests.cs:376-406) — exactly the coverage benaadams asked for:

  • OutOfGasOnBoundarySLoad — boundary op (cold SLOAD) exhausts gas.
  • OutOfGasInMeteredFallback — 500×PUSH0 block too large to precharge, dies mid-block on the metered path.

Both assert the stream engaged (FramesExecuted increased) and that StatusCode + GasSpent match the bytecode loop. Good.

Medium — fork-gating comment

InstructionStream.cs:102-109 and InstructionStreamCache.cs:9-16 now accurately document the spec.IncludePush0Instruction (Shanghai+) gate, the absence of an upper-bound fork check, and the requirement to revalidate precharged costs per new fork. ✅

One last stale sentence remains — the FusedOpcode doc-comment at InstructionStream.cs:28-29 still claims a "fingerprint gate ... keeps new forks off the stream until reviewed," which is now false. Posted an inline suggestion. Documentation-only; does not block merge.

Low nits — all resolved

  • TryGetInBlockCost doc (:269-273) now correctly says "PUSH2 excluded; PUSH1 and PUSH3..PUSH32 included; DUP9+/SWAP9+ excluded."
  • InstructionStreamCache carries the memory-footprint caveat (:14-16).
  • StaticJump carries the transiently-stale-programCounter note (:260-261).

Verdict: Mergeable ✓

benaadams' High consensus-divergence finding is fixed at all five sites and covered by a new regression test; his Medium and Low items are resolved. The only outstanding item is a one-line stale doc-comment (Low) which does not block merge.
• branch evm-stream

Comment thread src/Nethermind/Nethermind.Evm/CodeAnalysis/InstructionStream.cs
@benaadams

Copy link
Copy Markdown
Member

Findings

  1. Critical | EVM correctness/security: src/Nethermind/Nethermind.Evm/VirtualMachine.Stream.cs:250
    StaticJump / StaticJumpI skip the PUSH2 stack-overflow check. The existing bytecode path explicitly avoids the
    PUSH2; JUMP/JUMPI fast path when stack.Head == EvmStack.MaxStackSize - 1, so a full stack fails with StackOverflow
    before the jump. The stream path consumes jump gas and executes the jump anyway.
    Impact: post-Shanghai eth_call / estimateGas / simulation can return success or different output for user-driven
    bytecode that should fail with stack overflow. Block processing is gated out by default, but RPC correctness is
    affected.
    Fix: mirror InstructionPush2: consume/check only PUSH2 gas first, fail with StackOverflow when stack.Head >=
    EvmStack.MaxStackSize - 1, then consume JUMP/JUMPI gas and continue. Add differential tests for 1024 x PUSH0; PUSH2
    validDest; JUMP and the same for JUMPI.

  2. Medium | API/in-process integrity: src/Nethermind/Nethermind.Evm/CodeAnalysis/InstructionStream.cs:111, src/
    Nethermind/Nethermind.Evm/CodeAnalysis/CodeInfo.cs:73, src/Nethermind/Nethermind.Evm/VirtualMachine.Stream.cs:14
    The new stream implementation is exposed as public mutable API: CodeInfo.GetOrBuildStream() returns
    InstructionStream, whose arrays are public, and StreamInterpreter exposes public mutable process-wide switches.
    Impact: external in-process consumers/plugins can mutate stream arrays or force stream behavior globally,
    corrupting later EVM execution for cached code and locking an implementation detail into the public Nethermind.Evm
    API.
    Fix: make stream implementation types and switches internal; use InternalsVisibleTo or benchmark-only hooks where
    needed. If any surface must remain public, expose immutable/read-only views and document it as deliberate public
    API.

  3. Low | Robustness / benchmark leak: src/Nethermind/Nethermind.Evm.Benchmark/MultipleUnsignedOperations.cs:73
    _stateProvider.BeginScope(IWorldState.PreGenesis) returns an IDisposable, but the changed benchmark discards it and
    GlobalCleanup does not dispose it.
    Impact: benchmark runs leak the world-state scope and diverge from the pattern used in the new benchmarks.
    Fix: store it in a field, e.g. _stateScope, and dispose it in GlobalCleanup.

Optimization Notes

  • src/Nethermind/Nethermind.Evm/VirtualMachine.Stream.cs:50: Interlocked.Increment(ref
    StreamInterpreter.FramesExecuted) is paid on every streamed frame. If this is only for tests/diagnostics, consider
    compiling it out or using a cheaper diagnostics path.

  • src/Nethermind/Nethermind.Evm/CodeAnalysis/InstructionStream.cs:132: ConstantBytes is allocated for every constant
    even when no fused bitwise op needs it. Track whether bitwise/EQ fusion exists and allocate it only when required.

@claude verify these points

@claude

claude Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Claude finished @benaadams's task in 2m 15s —— View job


Verification of @benaadams' findings

I read the current branch HEAD and checked each point against the code.

# Finding Severity Verdict
1 StaticJump/StaticJumpI skip the PUSH2 stack-overflow check Critical Confirmed — real RPC correctness divergence
2 Stream impl exposed as public mutable API Medium ✅ Confirmed (design call)
3 Benchmark leaks the world-state scope Low ✅ Confirmed
Opt A FramesExecuted increment on every frame ✅ Confirmed (test-only)
Opt B ConstantBytes always allocated ✅ Confirmed

1. Critical — StaticJump/StaticJumpI skip the PUSH2 overflow check ✅ Confirmed

The unfused/dispatch path deliberately preserves the overflow a real PUSH2 raises at a full stack. InstructionPush2 (EvmInstructions.Stack.cs:131-137) only takes the PUSH2;JUMP/JUMPI fast path when:

// Head < MaxStackSize - 1 preserves the StackOverflow a non-fused PUSH2 would raise
// at head == 1024 (even though the following JUMP/JUMPI would immediately pop it).
stack.Head < EvmStack.MaxStackSize - 1 && ...

When stack.Head == 1024 (MaxStackSize - 1, since MaxStackSize == 1025) it falls through to the normal push, which overflows (newOffset >= MaxStackSize1025 >= 1025) and returns StackOverflow.

The stream's fused handlers (VirtualMachine.Stream.cs:250-281) consume gas and execute the jump with no stack.Head check at all:

case (Instruction)FusedOpcode.StaticJump:
    TGasPolicy.Consume(ref gas, GasCostOf.VeryLow + GasCostOf.Jump);
    if (TGasPolicy.IsOutOfGas(in gas)) { ... }
    opCodeCount++;
    entryIndex = (int)entry.Operand - 1;   // jumps — never checks Head == 1024
    break;
case (Instruction)FusedOpcode.StaticJumpI:
    TGasPolicy.Consume(ref gas, GasCostOf.VeryLow + GasCostOf.JumpI);
    ...

So bytecode that fills the stack to 1024 items and then runs PUSH2 dest; JUMP (or ; JUMPI) — which the analyzer fuses into StaticJump/StaticJumpIsucceeds on the stream path but fails with StackOverflow on the bytecode path. This is a real divergence on eth_call/estimateGas/simulate for post-Shanghai user bytecode. (Note: the in-block fused PUSH cases at lines 233-240 are safe — they genuinely call stack.PushUInt64/PushUInt256, which check overflow. Only the two static-jump cases bypass the push entirely.)

Fix — before consuming jump gas, mirror the InstructionPush2 guard at both sites:

case (Instruction)FusedOpcode.StaticJump:
    if (stack.Head >= EvmStack.MaxStackSize - 1) { exceptionType = EvmExceptionType.StackOverflow; break; }
    TGasPolicy.Consume(ref gas, GasCostOf.VeryLow + GasCostOf.Jump);
    ...

The order matters: the real PUSH2 charges its VeryLow then the push overflows, but gas-vs-overflow ordering produces the same terminal failure here — verify against the differential harness. Add differential tests for 1024×PUSH0; PUSH2 validDest; JUMP and the JUMPI variant (both jump-taken and not-taken). Fix this →


2. Medium — public mutable stream API ✅ Confirmed

InstructionStream is public sealed with public readonly array fields (Ops, BlockGas, Constants, ConstantBytes, PcToEntry) — the references are read-only but the array contents are mutable, and StreamOp/StreamOpKind/FusedOpcode are all public. StreamInterpreter.Enabled/ForceAllContexts/BuildThreshold are public static mutable process-wide switches, and CodeInfo.GetOrBuildStream() returns the stream publicly. An in-process plugin could mutate a cached stream's arrays (corrupting later execution of that code) or flip the global switches.

The cleaner posture is internal + InternalsVisibleTo for the test/benchmark assemblies. Caveat: much of Nethermind.Evm is already public (CodeInfo, EvmStack, Instruction), so this is a consistency/design judgment for the maintainers rather than a hard bug — but the mutable globals in particular are worth locking down.


3. Low — benchmark leaks the world-state scope ✅ Confirmed

MultipleUnsignedOperations.cs:73 calls _stateProvider.BeginScope(IWorldState.PreGenesis) and discards the returned IDisposable; GlobalCleanup (lines 96-101) disposes _evmState and _environment but not the scope. The new benchmarks (StreamInterpreterBenchmarks, InterpreterLoopBenchmarks) store it in a field and dispose it. Fix: _stateScope = _stateProvider.BeginScope(...) and dispose in GlobalCleanup.


Optimization notes

  • A — FramesExecuted (VirtualMachine.Stream.cs:50): confirmed — the only readers are tests (InstructionStreamTests.cs:329/357/396), which assert "the stream engaged". Nothing in production reads it, so the Interlocked.Increment is pure per-frame diagnostic cost. A [Conditional]/test-only hook or a debug-gated counter would remove it from the hot path.
  • B — ConstantBytes (InstructionStream.cs:132): confirmed — the constructor unconditionally allocates new byte[constants.Length * 32] and fills it, but it is only consumed by the fused bitwise cases (Eq/And/Or/Xor). A stream whose constants feed only arithmetic/shift fusion (which read the UInt256 Constants form) carries the big-endian copy for nothing. Tracking whether any bitwise/EQ fusion exists and allocating only then would save it.

Summary: All three findings and both optimization notes are valid. Finding #1 is a genuine consensus-style divergence on the cancelable RPC path (success where the spec requires StackOverflow) and should block merge until fixed + covered by a differential test. I verified but did not modify code — let me know if you'd like me to implement the #1 fix and tests.

@svlachakis

Copy link
Copy Markdown
Contributor Author

@claude review again all @benaadams 's comments

@claude

claude Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Claude finished @svlachakis's task in 1m 37s —— View job


Re-review of all @benaadams' findings — current HEAD 913eb6b

  • Read current branch HEAD
  • Verify finding Feature/jsonrpc #1 — Critical: StaticJump/StaticJumpI skip PUSH2 stack-overflow check
  • Verify finding Feature/networking #2 — Medium: public mutable stream API
  • Verify finding Netcore #3 — Low: benchmark leaks world-state scope
  • Verify Opt A — FramesExecuted per-frame increment
  • Verify Opt B — ConstantBytes always allocated
  • Verify prior OOG finding still fixed

@benaadams left two batches of comments. Here is the status of every item.

Batch 2 (latest — 2026-06-26 10:18)

# Finding Severity Status
1 StaticJump/StaticJumpI skip the PUSH2 stack-overflow check Critical Fixed
2 Stream impl exposed as public mutable API Medium ✅ Fixed
3 Benchmark leaks the world-state scope Low ✅ Fixed
A FramesExecuted Interlocked.Increment per frame opt ✅ Addressed
B ConstantBytes always allocated opt ✅ Fixed

#1 — Critical (fixed + tested). Both fused cases now mirror InstructionPush2 and fail with StackOverflow at a full stack before consuming jump gas:

  • VirtualMachine.Stream.cs:256StaticJump: if (stack.Head >= EvmStack.MaxStackSize - 1) { exceptionType = StackOverflow; break; }
  • VirtualMachine.Stream.cs:276StaticJumpI: same guard.

The exact differential tests benaadams asked for are present: FullStackStaticJumpOverflows and FullStackStaticJumpIOverflows (InstructionStreamTests.cs:338-339) build 1024×PUSH0; PUSH2 dest; JUMP/JUMPI (BuildFullStackJump, :282-287) and assert the stream result matches the bytecode loop. Confirmed real consensus-path divergence is closed.

#2 — Medium (fixed). The stream types are no longer public surface:

  • StreamInterpreter is now internal static class (:13) — the Enabled/ForceAllContexts/BuildThreshold process-wide switches are no longer externally reachable.
  • InstructionStream is internal sealed (InstructionStream.cs:111); StreamOp/StreamOpKind/FusedOpcode are all internal.
  • CodeInfo.GetOrBuildStream() is now internal (CodeInfo.cs:73).

In-process plugins can no longer mutate cached stream arrays or flip the global switches.

#3 — Low (fixed). MultipleUnsignedOperations now stores the scope in _stateScope (:34, assigned :74) and disposes it in GlobalCleanup (:102), matching the new benchmarks.

Opt A (addressed). FramesExecuted is now [ThreadStatic] public static long with a plain ++ (:32, :52) — no Interlocked on the hot path, and thread-local so there is no race. Cheaper than the per-frame atomic and resolves the original non-atomicity caveat too.

Opt B (fixed). ConstantBytes is now built only when bitwise/EQ fusion was emitted — the constructor takes buildConstantBytes and assigns ConstantBytes = [] otherwise (InstructionStream.cs:134-145). Arithmetic/shift-only streams no longer carry the big-endian copy.

Batch 1 (2026-06-26 03:14) — re-confirmed still fixed

Finding Severity Status
Out-of-gas never detected (ulong < 0 always false) High ✅ All five sites use TGasPolicy.IsOutOfGas (Stream.cs:262, 282, 333, 347, 485)
Fork-gating comment contradicts gate Medium InstructionStream.cs:102-109 remarks corrected
PUSH9+ excluded comment inaccurate Low ✅ Fixed
InstructionStreamCache memory note Low ✅ Added
StaticJump transiently-stale programCounter note Low ✅ Added (Stream.cs:270)

OOG regression tests OutOfGasOnBoundarySLoad and OutOfGasInMeteredFallback are present (InstructionStreamTests.cs:416, 423).

One remaining nit (Low, doc-only — does not block)

The FusedOpcode doc-comment still references the removed system: "The fingerprint gate keeps new forks ... off the stream until reviewed" (InstructionStream.cs:28-29). There is no fingerprint gate — the real gate is spec.IncludePush0Instruction (any fork ≥ Shanghai), already correctly documented on the InstructionStream remarks just below. This is the last stale reference; I flagged it inline previously. Documentation-only.

Verdict

All of @benaadams' Critical / High / Medium findings (and both optimization notes) are resolved on 913eb6b, the Critical #1 backed by the exact differential tests requested. The only outstanding item is a one-line stale doc-comment (Low). Mergeable from the standpoint of his review.
• branch evm-stream

negative-overflow trick for OOG detection — so every Consume now
guards the subtraction with a bounds branch. The stream block
precharge already proves remaining >= cost one line above, making
that branch redundant on the hottest per-block charge.

ConsumeUnchecked does the branchless `Value -= cost` only where the
caller has verified sufficiency; checked Consume stays everywhere
else. ulong API and OOG semantics unchanged (differential suite
24/24, incl. out-of-gas cases).
@svlachakis

Copy link
Copy Markdown
Contributor Author

@claude review again

@claude

claude Bot commented Jun 27, 2026

Copy link
Copy Markdown
Contributor

Claude finished @svlachakis's task in 2m 28s —— View job


PR Review: EVM Stream (Eleventh Pass)

Focused on the commits added since @benaadams' approval (913eb6b25a5d4b): the gas→ulong migration fallout and the new ConsumeUnchecked precharge tricks.

  • Read current branch HEAD & key files
  • Review new ConsumeUnchecked / OOG handling
  • Verify all prior High/Medium findings still resolved
  • Deep review of stream hot loop, fused jumps, metered fallback
  • Post findings

New commits since approval — ConsumeUnchecked tricks

The two latest commits replace the bounds-checked Consume with ConsumeUnchecked on the three hottest stream gas charges. I verified each call site is provably safe:

Site VirtualMachine.Stream.cs Guard
Per-block precharge :85 if (GetRemainingGas >= cost) one line above (:81)
Fused StaticJump :270 if (GetRemainingGas < jumpCost) → SetOutOfGas; goto OutOfGas (:264)
Fused StaticJumpI :292 if (GetRemainingGas < jumpiCost) → SetOutOfGas; goto OutOfGas (:286)

EthereumGasPolicy.ConsumeUnchecked (:124) is the raw gas.Value -= cost with no clamp and no OutOfGas flag. In all three cases sufficiency (Value >= cost) is proven immediately before, so the unsigned subtraction cannot wrap. The block-precharge set is arithmetic/stack/push/comparison/bitwise/shift only — none carry EIP-8037 state gas — so charging Value alone is complete and correct. The interface default (IGasPolicy.cs:49) routes ConsumeUnchecked back to the safe Consume for any non-EthereumGasPolicy implementation, so the optimization is opt-in per policy. ✅

The < cost → OutOfGas branches at :264/:286 correctly preserve benaadams' batch-1 finding (no ulong < 0 dead checks remain): the metered fallback (:94) and the boundary/MSTORE paths all gate on IsOutOfGas, not a signed-negative trick.

Prior High/Medium findings — all still resolved

  • StaticJump/StaticJumpI PUSH2 stack-overflow (benaadams Critical) — both fused cases guard stack.Head >= MaxStackSize - 1 → StackOverflow before charging jump gas (:258, :280). ✅
  • OOG never detected (ulong < 0) — all sites use IsOutOfGas. ✅
  • Public mutable stream APIStreamInterpreter, InstructionStream, StreamOp/FusedOpkind/FusedOpcode, and CodeInfo.GetOrBuildStream() are internal. ✅
  • State.Flat bundling (asdacap) — split to Flat - Persistence Cache #12035/SLOAD Improvements #12043; no State.Flat files in this PR. ✅
  • PUSH-immediate 0x5B jump targetTryBuild returns null; regression test present. ✅
  • Benchmark scope leak, FramesExecuted ([ThreadStatic]), ConstantBytes lazy alloc, stale fork-gate comments — all addressed.

One informational note (non-blocking)

Gas-vs-overflow ordering in fused static jumpsVirtualMachine.Stream.cs:258 / :280

The fused path checks stack overflow then gas; the real InstructionPush2 (EvmInstructions.Stack.cs:126) charges VeryLow first, then overflows on the fall-through push. So in the corner case of a full stack (1024) AND insufficient gas for the push, the unfused path terminates as OutOfGas (the loop's IsOutOfGas check wins over the returned StackOverflow) while the fused path returns StackOverflow.

This is not an observable divergence: the stream runs only on non-tracing cancelable RPC frames, both outcomes are exceptional halts that consume all remaining gas, and both yield StatusCode = failure with identical GasSpent. eth_call/estimateGas cannot distinguish them. Noting only for completeness — does not block, and the differential tests (which compare status + gas) correctly pass.


Verdict: Mergeable ✓

The new ConsumeUnchecked commits are correct and safely guarded. All Critical/High/Medium findings from all prior passes (including both of benaadams' batches) remain resolved, and benaadams has approved. The single new observation is informational and not observable on the gated RPC path.
• branch evm-stream

Comment thread src/Nethermind/Nethermind.Evm/CodeAnalysis/CodeInfo.cs
Comment thread src/Nethermind/Nethermind.Evm/CodeAnalysis/CodeInfo.cs
Comment thread src/Nethermind/Nethermind.Evm/CodeAnalysis/InstructionStream.cs Outdated
Comment thread src/Nethermind/Nethermind.Evm/CodeAnalysis/InstructionStream.cs Outdated
Comment thread src/Nethermind/Nethermind.Evm/VirtualMachine.Stream.cs Outdated
Comment thread src/Nethermind/Nethermind.Evm/CacheCodeInfoRepository.cs
Comment thread src/Nethermind/Nethermind.Evm/Instructions/EvmInstructions.FusedConst.cs Outdated
Comment thread src/Nethermind/Nethermind.Evm/Instructions/EvmInstructions.FusedConst.cs Outdated
@LukaszRozmej

Copy link
Copy Markdown
Member

I still don't like the added maintenance cost

@asdacap

asdacap commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Running era mainnet replay, just in case...

@svlachakis

svlachakis commented Jun 29, 2026

Copy link
Copy Markdown
Contributor Author

Looks like it has regressed in terms of eth_call time a bit, not sure where, investigating

@svlachakis

Copy link
Copy Markdown
Contributor Author

Nevermind, not apples to apples because machine is different. Let's get it merged. @asdacap did it passed your test?

@svlachakis

Copy link
Copy Markdown
Contributor Author

@claude review one last time the changes by this PR deeply and confirm there are no regressions in block processing or RPC path.

@claude

claude Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Claude finished @svlachakis's task in 3m 8s —— View job


PR Review: EVM Stream — final regression-focused pass

Confirming the two questions you asked: block-processing regressions and RPC-path regressions. I re-read the shared dispatch core, the gas policy, the stream interpreter, the fused cores, and the CodeInfo build gating against the current HEAD.

  • Read all changed EVM files (fresh pass)
  • Verify block-processing path is untouched (no stream, no specialized dispatch)
  • Verify RPC/cancelable path correctness (gas, OOG, jumps, overflow)
  • Verify shared VirtualMachine.cs / instruction changes don't affect block processing
  • Synthesize verdict

Block processing — no regression ✅

Block processing is structurally excluded from every new code path, enforced by a compile-time flag, not a runtime check:

  1. Stream never engages. ExecuteCall selects RunByteCode<_, OffFlag> for non-cancelable tracers (VirtualMachine.cs:1188-1192). The stream gate at VirtualMachine.cs:1219-1221 requires TCancelable.IsActive || StreamInterpreter.ForceAllContexts. Block-processing tracers are non-cancelable and ForceAllContexts is test-only (default false), so the gate is folded to false by the JIT.
  2. No stream is even built. Because of && short-circuit, VmState.Env.CodeInfo.GetOrBuildStream() is never called on the block path — so no StreamOp[] is built, scheduled, or retained for block-processing contracts. This matches the design claim ("never builds or runs a stream").
  3. Dispatch is master-parity. With TCancelable=Off, RunByteCodeCore takes the plain function-pointer table (VirtualMachine.DispatchSpecialized.cs:195-203), with POP kept inline (functionally identical to InstructionPop). The hot direct-dispatch switch is entirely inside if (TCancelable.IsActive), which folds away.
  4. New shared hooks are no-ops. EthereumGasPolicy.OnBeforeInstructionTrace / OnAfterInstructionTrace are empty (EthereumGasPolicy.cs:462-466) and aggressively-inlined → zero cost.
  5. The ref int pc register-residency refactor (loop-local pc, written back via programCounter = pc) is behavior-preserving: pc is seeded from programCounter and flushed after each dispatch; nothing reads programCounter in between.

The only block-processing-visible change is the inline-POP special case and the register-residency temp — both behavior-identical to master. OOG detection uses TGasPolicy.IsOutOfGas (DispatchSpecialized.cs:206), matching master semantics.

RPC / cancelable path — no regression ✅

All consensus-divergence risks raised across the 11 prior passes + @benaadams' two batches are fixed in the current code:

Risk Site Status
OOG dead-check (ulong < 0) precharge Stream.cs:80, jumps :252/:272, metered :490-493 TryConsume / IsOutOfGas — no signed-negative trick remains
Full-stack StackOverflow skipped on fused jumps Stream.cs:247-251, 267-271 ✅ Guards stack.Head >= MaxStackSize - 1 before charging jump gas, mirroring InstructionPush2
0x5B PUSH-immediate as jump target InstructionStream.cs:271-275 TryBuild returns null → bytecode loop produces InvalidJumpDestination
Mid-block / oversized-precharge landing Stream.cs:83-101, RunMeteredSegment ✅ Exact per-op metered fallback
Boundary landing inside a fused pair Stream.cs:375-381 InvalidEntryInvalidJumpDestination, fails loud

Gas correctness spot-checks hold: block gas (blockGas[BlockIndex]) is the exact sum of in-block costs, charged once via TryConsume; the precharge uses the bounds-checked TryConsume (interface default is safe for any policy — no unchecked ulong wrap). Fused operand order FusedConstBinaryCore(a=const, b=stack_top) matches the unfused op(top, below) for non-commutative ops; the full-stack overflow guard Head == MaxStackSize - 1 matches a real PUSH. The stream is gated to Shanghai+ (spec.IncludePush0Instruction), where the precharged in-block opcode set is fork-invariant.

The shared InstructionStreamCache is keyed by CodeHash and only ever populated/read from the cancelable path (GetOrBuildStream short-circuits on the block path), so it cannot leak streams into block processing.


One residual (already acknowledged, not a found defect)

Plugin RPC paths (OP/Taiko) are unverified, per the author. Block processing on these chains is unaffected (same cancelable gate). On their RPC path, if the tracer is cancelable and the fork is Shanghai+, the stream will engage with the plugin's TGasPolicy. The in-block op set is fork-invariant arithmetic/stack, and all gas operations route through the IGasPolicy interface (TryConsume/SetOutOfGas/IsOutOfGas have safe default impls), so it should be correct — but it hasn't been differential-tested on those networks. Worth a sanity check if OP/Taiko RPC is in scope; it does not block the mainnet merge.


Verdict

No regressions found in block processing or the RPC path. Block processing is provably excluded at compile time; the RPC path is differential-tested and every prior Critical/High/Medium finding is resolved in the current code. Both @benaadams and @LukaszRozmej have approved. @asdacap's era-mainnet replay (in progress) is the right final gate before merge — that directly exercises the block-processing path on real history.
• branch evm-stream

@svlachakis
svlachakis merged commit d2deee0 into master Jun 30, 2026
560 checks passed
@svlachakis
svlachakis deleted the evm-stream branch June 30, 2026 08:09
LukaszRozmej added a commit to LukaszRozmej/membership-1 that referenced this pull request Jul 8, 2026
Name / Identifier
Stavros Vlachakis

Team / Project
Nethermind

Start date of relevant projects
July 2025 (part-time)
February 2026 (full-time)

Proposed weight
Full (1.0)

Summary of work / eligibility

Stavros joined Nethermind in March 2025. Since July 2025, he has contributed part-time to the Nethermind Ethereum Execution Client (Core team) alongside other responsibilities. Since February 2026, he has worked full-time on Nethermind Client, with 100+ merged PRs in total. He owns the JSON-RPC for Nethermind Client. He is also expected to contribute extensively to Frame Transactions on Hegota.

Representative work:

- EIP-4444 history expiry (EraE): implemented the EraE archive format end-to-end — era export/import and remote download with SHA-256 verification. (#10812 (NethermindEth/nethermind#10812))
- JSON-RPC (owner): broad spec-compliance and Geth-parity work plus new endpoints — e.g. Geth-compatible error codes (#11335 (NethermindEth/nethermind#11335)) and eth_signTransaction / raw-transaction methods (#11517 (NethermindEth/nethermind#11517), #11521 (NethermindEth/nethermind#11521)).
- Streaming for large RPC responses: streaming approach for trace_* and debug_* results, avoiding buffering huge responses in memory. (#11755 (NethermindEth/nethermind#11755), #11693 (NethermindEth/nethermind#11693)).
- EVM & execution performance: eth_call interpreter/dispatch optimizations (#11965 (NethermindEth/nethermind#11965)), EVM memory pooling and SLOAD / flat-state read caching (#11991 (NethermindEth/nethermind#11991), #12043 (NethermindEth/nethermind#12043)).
- State & consensus reliability: correct persisted-code tracking in the code DB (#11714 (NethermindEth/nethermind#11714)), forkchoice canonical-chain corruption healing after beacon sync (#10876 (NethermindEth/nethermind#10876)).

All merged PRs: https://github.com/NethermindEth/nethermind/pulls?q=is%3Apr+author%3Asvlachakis+is%3Aclosed (edited)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants