proofs: Rework blob handling

[mbaxter]

Description

This PR fixes a medium severity bug in the fault proof system related to EIP-4844 blob data handling. For more context on this issue, see the disclosure of this bug in the technical proposals governance forum.

User assets are not at risk given Optimism’s multi-layered approach to security. Production fault-proof systems are protected by existing safeguards: monitoring, the air gap, the ability to blacklist games, and the ability to fall back to permissioned dispute games. Superchain chain operators currently relying on permissionless fault proofs, and chain servicers running infrastructure for these chains, have been proactively notified of this bug.

Technical Overview

Fix blob preimage handling in the op-challenger and op-program.

Blob field element preimage keys are constructed by concatenating a blob commitment with a blob polynomial evaluation point z: blobCommitment ++ z¹. The op-challenger uses this z value in the preimage key to reconstruct the target blob element by evaluating the blob polynomial at this point. The op-challenger then collects all of the relevant blob data and pushes it to the PreimageOracle via PreimageOracle.loadBlobPreimagePart.

Previously, the z value used in this process was set to the index of the target field element in the blob. However, because of the way that EIP-4844 blobs works, evaluating the blob polynomial at this index produces some value that is not our target field element. This causes the op-challenger to push incorrect blob data to the PreimageOracle. Further, the op-program uses the same incorrect z value to pull data from the preimage oracle. When op-program is run through the off-chain Cannon CLI, it pulls the blob data from the local op-program host which has correctly stored the blob data. However, when the op-program executes on-chain, it pulls the invalid data from PreimageOracle.sol leading to a divergence in behavior between the on- and off-chain executions of Cannon.

This PR reworks the way that we store blob preimage data. We now set z to one of the predefined evaluation points² from EIP-4844. This allows us recover the correct blob field element when evaluating the blob polynomial at z. We also now perform additional validations to ensure that the result from the blob polynomial evaluation matches our expectation. The updated op-challenger will now push the expected blob data to the PreimageOracle, and the op-program will retrieve this blob data using the correct keys to retrieve the correct data.

Primer on EIP-4844 Blobs

Blobs are used in the OP stack to hold batched transaction data. An EIP-4844 blob is a byte string that represents a polynomial in evaluation form defined over a finite field. The byte string is a concatenations of 4096 32-byte field elements. Each field element represents the value of the polynomial at one of 4096 predefined evaluation points. These predefined points are the 4096th roots of unity in the finite field (in bit-reversed order).

$P(x) = blobPolynomial = a_0 + a_1*x^1 + … + a_{4095}*x^{4095}$

$r = rootsOfUnity = [ω^0, ω^1, ω^2,...,ω^{4095}]$

$s = bitReversalPermutation(r) = [s_0,s_1,...,s_{4095}]$

$blob = [P(s_0), P(s_1), ..., P(s_{4095})]$

A succinct proof that a field element (or substring of the batch data) is inside a blob consists of the following:

• 48-byte blob commitment provided by L1
  • This commits to a specific polynomial $P$ - the blob polynomial
• $z$ - the 32-byte proof challenge point
  • To prove a field element at index i in the blob, this must the permuted root of unity at the same index: $s_i$
• $y$ - the 32-byte field element being proven
• 48-byte KZG proof of the evaluation $P(z) = y$, where $P$ is our blob polynomial.

The challenger generates these four values to verifiably load blob preimages, as field elements, into PreimageOracle.sol.

We previously used $z = i/32$ (the index of a given field element) to generate the blob proof data. However, if we evaluate the polynomial at this $z$ value, we will not retrieve the target blob field element but some other value not present in the blob byte string. We are now using the ith permuted root of unity: $z=s_i$.

Summary of Changes

Production

Rework blob handling:

• Rework preimage.go challenger trace util:
  • Key blob fields using a root of unity rather than the index into the blob for the target blob field
  • Add a few additional validaton checks
• Rename PreimageOracleData.BlobFieldIndex to the more generic PreimageOracleData.ZPoint
• Update op-program l1 oracle client
  • Use the new blob key format
  • Cache the roots of unity
• Update op-program host prefetcher to use new key format

Other changes:

• Modify cannon run command stop-at-preimage flag to support an optional step constraint. If set, the vm will only stop if it has reached or exceeded the specified step value.
  • Also update provider.go with new utilities for constructing this flag
• Add a method to oracle.go to pull preimageParts data from the PreimageOracle contract

Tests

• Temporarily disable op-program-compat in CI - will need a follow-up PR to re-enable this
• Add test for new oracle.go method to pull primageParts data
• Update preimage_test.go to match new logic and run tests over more than 1 field element
• Add new e2e test helpers
  • Add utilities to query for and validate data in the PreimageOracle
  • Add utility to find a target preimage load step that is odd
  • Add utility to challenge to a preimage load at a specified step
• Update e2e test testOutputCannonStepWithPreimage :
  • Challenge to an odd preimage load step to induce the challenger to perform a step-defend action
  • Add test cases to target various offsets into the blob preimage being loaded
  • Add test cases to target blob field elements at various indexes within the blob
  • Verify the MIPS contract performs the preimage load step() as expected
• op-program/client/l1/oracle_test.go
  • Add test for op_program l1 oracle roots of unity generation
  • Add test for getBlob
• op-program/host/prefetcher/prefetcher_test.go
  • Update test to use new key format

Footnotes

1. Note that this value is then hashed and the first byte is set to the blob key type in order to construct the final key. ↩

2. These are the bit-reversed roots of unity defined in the Ethereum consensus specs for EIP-4844. ↩

[codecov]

Codecov Report

Attention: Patch coverage is 67.36842% with 31 lines in your changes missing coverage. Please review.

Project coverage is 46.43%. Comparing base (d0254ff) to head (6eda03f).
Report is 4 commits behind head on develop.

Files with missing lines	Patch %	Lines
cannon/cmd/run.go	0.00%	11 Missing ⚠️
op-challenger/game/fault/trace/utils/provider.go	0.00%	7 Missing ⚠️
op-challenger/game/fault/trace/utils/preimage.go	70.00%	4 Missing and 2 partials ⚠️
op-challenger/game/fault/contracts/oracle.go	62.50%	2 Missing and 1 partial ⚠️
op-challenger/game/fault/types/types.go	0.00%	2 Missing ⚠️
op-program/client/l1/oracle.go	93.75%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop   #15354      +/-   ##
===========================================
+ Coverage    46.29%   46.43%   +0.14%     
===========================================
  Files         1227     1227              
  Lines       102944   103022      +78     
===========================================
+ Hits         47659    47841     +182     
+ Misses       51925    51805     -120     
- Partials      3360     3376      +16     

Flag	Coverage Δ
cannon-go-tests-32	63.99% <0.00%> (+1.90%)	⬆️
cannon-go-tests-64	58.68% <0.00%> (+1.54%)	⬆️
contracts-bedrock-tests	94.30% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
op-program/host/prefetcher/prefetcher.go	53.73% <100.00%> (+0.10%)	⬆️
op-service/testutils/random.go	85.00% <100.00%> (+0.59%)	⬆️
op-challenger/game/fault/types/types.go	47.82% <0.00%> (ø)
op-program/client/l1/oracle.go	77.55% <93.75%> (+37.25%)	⬆️
op-challenger/game/fault/contracts/oracle.go	82.31% <62.50%> (-0.66%)	⬇️
op-challenger/game/fault/trace/utils/preimage.go	77.41% <70.00%> (-4.07%)	⬇️
op-challenger/game/fault/trace/utils/provider.go	0.00% <0.00%> (ø)
cannon/cmd/run.go	0.00% <0.00%> (ø)

... and 16 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Inphi]

LGTM. s/out to @ImTei for flagging the blob issue in the challenger.

[mbaxter]

LGTM. s/out to @ImTei for flagging the blob issue in the challenger.

+1 🔥

Also shoutout to @Inphi for the initial investigation and preliminary fix :)

[Inphi]

LGTM. s/out to @ImTei for flagging the blob issue in the challenger.

+1 🔥

Also shoutout to @Inphi for the initial investigation and preliminary fix :)

and @protolambda for designing the fix.

[BlocksOnAChain]

Awesome work, thank you @mbaxter @pauldowman @Inphi for making this happen.