fix(ergo-sigma): SBox-from-constant keeps real transaction_id/output_index - #23
Merged
Merged
Conversation
…index read_ergo_box parses the 32-byte transaction id and the VLQ output index from the box tail, but the SBox-constant EvalBox in sigma_to_value hardcoded transaction_id: [0u8;32] and output_index: 0, discarding them. This corrupted ExtractCreationInfo (0xC7) — whose R3 reference is transactionId.toBytes ++ Shorts.toByteArray(index) (FIXED 2-byte big-endian) — and ExtractBytesWithNoRef (0xC4), whose strip length is 32 + VLQ(index) (so a zeroed index strips the wrong number of bytes for index >= 128). Now carries the parsed ergo_box.transaction_id / ergo_box.index. Matches Scala ErgoBox.get(ReferenceRegId) and bytesWithNoRef. Closes sbox-const-discards-txid-index (3 entries: Box_properties_equivalence 0xC7 #10/#11 and 0xC4 #8). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
Warning Review limit reached
More reviews will be available in 37 minutes and 33 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
arkadianet
added a commit
that referenced
this pull request
Jul 3, 2026
* docs(fuzz): interface contracts + known-bug catalog for fuzz-differential harness Foundation for the continuous fuzz-differential harness (Rust node vs Scala reference/ergo-core 6.0.2). Fixes the sidecar RPC / replay-driver I/O / generator-output / divergence-record contracts before any parallel build, and commits the 25-entry known-bug rediscovery catalog that gates the generators. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(fuzz): commit streamed-replay reproducibility pins from :9053 Height -> (headerId, stateRoot) pins for the differential replay driver. Makes a retired block range reproducible with zero committed block bytes: the driver asserts the archival node still serves the pinned headerId at each height before applying, and diffs Rust root_digest() against the pinned stateRoot. Covers the hermetic genesis seed (1-5) plus the known-gnarly incident heights (1805523 adproofs, 1808895 register stall, 1818476 indexer degrade, etc.). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(difftest): structure-aware SER generators + coverage gate Add `ergo-difftest/src/gen/`: structure-aware, grammar-position generators for the five SER surfaces (ergo_tree, constant, ergo_box_candidate, transaction, header), replacing the weak byte-mutation baseline for the highest-bug-density wire surface. Each generator runs two modes: on-manifold (build valid structs, serialize via the real ergo-ser writers) and adversarial-assembly (hand-place values the writers never emit). A FeatureSet records which adversarial features each output touched; declared_vocabulary + Coverage make a campaign's bug-surface coverage measurable, and gen_structured_at makes (seed,iter, surface) reproducible. Features map 1:1 to known SER bugs and are all reachable: header version 0x80..0xFF (#8), tree version/size/cseg bits + sizeless non-SigmaProp root (#17/#9/#25), declared size != body len (#19), FunDef nTpeArgs>=0x80 (#14), ill-formed-UTF-8 STypeVar (#1), VLQ > i32::MAX (#20), SUnsignedBigInt in a pre-v3 tree (#21), compact Relation2 bool pair (#12), v6-typed register (#5), off-curve group element (#4), empty outputs / 0-amount token (#23). Wire-in: run_structured_campaign feeds gen output through the same hermetic no-panic + fixed-point invariants and records the coverage union; a `--structured` CLI flag prints the per-surface coverage report and can feed the oracle differential. tests/gen_coverage.rs asserts full declared-vocab coverage per surface, determinism, on-manifold accept rate, no hermetic bugs, and that each adversarial feature lands on its intended verdict. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(difftest): integrate known-bug re-injection gate (Slice 5) Manifest (25 bugs), 9 apply-clean re-injection patches, reinject_gate.sh runner, and a hermetic --check-canonical flag. Kept the foundation's canonical docs over the agent's origin/main-based reconstruction. Merged the --check-canonical CLI flag alongside --structured. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(fuzz): findings + acceptance methodology + triage queue Honest signal assessment after running the generators through the re-injection gate + live JVM oracle. Proven: coverage 16/16; #12 rediscovery via clean-vs-patched Canonical delta (356->1321). Surfaced (not papered): (A) bare-ergo_tree over-reports because node validates in layers vs JVM's monolithic serializer -> acceptance must be per-surface + differential, not absolute count; (B) the oracle campaign reports non-reproducing divergences -> Slice 3 minimize+repro-verify is load-bearing. Escalated TRIAGE-001 (ValUse undefined-id: node Accept / JVM Reject) for a human consensus-truth call — not resolved here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(fuzz): correct findings — no signal-integrity bug; reduce surface is clean Root-caused the divergence output with systematic debugging + campaign instrumentation. CORRECTIONS to the prior findings doc: - WITHDRAWN 'non-reproducing divergences' — was my triage error (repro'd the node's canonical-OUTPUT field 080208d3 instead of the actual INPUT 080108d3). Findings reproduce exactly via input_hex. - The bare ergo_tree ~24% divergence floor is PARSE-SURFACE ARTIFACTS (benign canonical writer diffs [node retains original bytes in consensus], deferred GE curve-check, ValUse-undefined) — ALL agree on the consensus-complete reduce surface. - The reduce surface is CLEAN: 0 divergences/1500 on clean HEAD → trustworthy consensus differential. This is where consensus bugs should be hunted. - TRIAGE-001 downgraded (agrees on reduce; not a consensus bug). Acceptance gate methodology (per-surface + differential) STANDS and is sharper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(difftest): eval-rich SIGMA generators for the reduce surface (Slice 2c) Add sigma_expr generator emitting well-typed ErgoTree bodies that reduce non-trivially against the consensus-complete reduce oracle surface (SELF at 1M nanoErg, activated v3). Vocabulary: sigma props, boolean logic, Int/Long/ BigInt arithmetic, comparisons, collection ops, context accessors, registers/ tuples/options, and the eval/cost bug edges: atLeast (#13), early-vs-late Coll equality (#15), token-collection equality (#16), deserialize nodes (#3). reduce clean baseline = 0 divergences / 3000 (seed 1). Re-injection patches prove the generator rediscovers each mapped bug via a clean-vs-patched delta on reduce: #13 AcceptReject (P:d3|1481 vs Reject), #15 Canonical (P:d2|45 vs 43), #16 Canonical (P:d2|168 vs 60). #3 reported as a coverage gap (the reduce surface bypasses reductionWithDeserialize on both sides). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(difftest): streamed archival replay driver (Slice 1b) New `ergo-difftest --bin replay`: pulls full blocks from a live Scala archival node (:9053) one at a time over plain HTTP/1.1, applies each in-process via validate_full_block_parallel + StateStore::apply_block (forced full validation), and diffs the Rust root_digest() against the Scala-committed stateRoot. JSONL divergence stream + summary; exits non-zero on divergence or pin mismatch. Verifies covered heights by hash against replay-pins.json. This makes the immutable chain the oracle — the mechanism to retire committed block fixtures. Proof: --from 1 --to 200 -> {blocks:200, tx_total:200, divergences:0, pins_verified:5} against a live mainnet node. The driver FOUND a real Rust-vs-Scala bug on block 3: ergo-rest-json decodes the Autolykos v1 pow `d` field as signed two's-complement, but Scala serializes it unsigned (asUnsignedByteArray). Per operator decision the production fix is DEFERRED (see gitignored dev-docs/fuzz-differential/autolykos-v1-d-signed-vs- unsigned-bug.md); production ergo-rest-json is left UNTOUCHED. The driver works around it locally + byte-exact in correct_v1_pow_d (strip the spurious sign byte), clearly documented as removable once the production fix lands. Only non-difftest change is a lint-only cfg_attr(allow(dead_code)) on ergo-state PersistJob (the field is read only under test-helpers; ergo-difftest now builds ergo-state as a dep). No consensus logic touched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(difftest/slice3): minimize + classify + auto-file divergence pipeline Slice 3 of the fuzz-differential harness. Three additions: * `minimize.rs` — deterministic ddmin-style minimizer: chunk removal, truncation, single-byte deletion, fixed-point loop. `minimize_divergence` builds the predicate from `diff` (same surface + kind + verdict class), re-verifies minimized result. * `regressions.rs` — `DivergenceRecord` (§4 schema), `Triage` (Pending / KnownArtifact), `classify` (parse-surface reconciles-on-reduce rule), `auto_file` (sha256-addressed JSON + QUEUE.md Pending-only gate), `classify_and_file` pipeline. * CLI `--minimize [--regressions-dir D]`: after campaign, minimize+classify+file each unique divergence; `--repro --minimize --surface <s>` for single inputs. * `tests/minimize.rs` (9 hermetic tests): shrinks to [0xAB], correctness invariant, determinism, fixed-point, empty reduction, noop-on-unsatisfied, record JSON round-trip, QUEUE gate (Pending queued / KnownArtifact not), idempotency. Gate: fmt clean, clippy -D warnings clean, all 35 tests pass (16 unit + 9 gen_coverage + 9 minimize + 1 selftest). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore(difftest): gitignore generated regressions/ directories Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore(difftest): set default-run=difftest so bare `cargo run` still resolves Adding the replay binary made `cargo run -p ergo-difftest` ambiguous; pin the default to difftest so the documented `cargo run -p ergo-difftest -- ...` invocations keep working. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(difftest/slice1a): validate + verify_avl differential surfaces + avl_frame wire Adds two new JVM-oracle differential surfaces (Slice 1a): * `validate` — stateless structural TX validity (mirrors Scala `ErgoTransaction.statelessValidity()`; no UTXO/chain state). Rust: `validate_structural` + `read_transaction`. JVM: `ErgoTransactionSerializer.parseBytes` + `.statelessValidity()`. * `verify_avl` — AVL+ batch-proof verification twin of `ergo_sigma::avl::AvlVerifier`; targets bug #6 (valid-but-wrong proof → `ergo_avltree_rust` panics at op-time while Scala fails closed). Clean: both sides REJECT (guard catches panic). Patched (re-injection): Rust `Outcome::Bug` vs JVM REJECT. Rust: `AvlVerifier` via `avl_frame::AvlFrame::decode`. JVM: `CAvlTreeVerifier` (via reflection; constructor is `private[eval]` in Scala source despite `public` in bytecode). New shared module `ergo-difftest/src/avl_frame.rs` defines the wire layout (startingDigest‖keyLen‖valueLenOpt‖proofLen‖proof‖ opCount‖[opTag‖key‖(val)?]*) used by both Rust and Scala, with encode/decode + 7 unit tests including the trigger-generation test that emits the bug #6 hex. Re-injection patch: `ergo-difftest/known_bugs/patches/avl-verifier-panic.patch` removes `catch_unwind` from `AvlVerifier::guarded()`. Manifest entry for bug #6 (`avl-verifier-panic`), surface `verify_avl`, class `panic`, `wire_reachable=true`, trigger hex pinned. Gate: fmt clean, clippy clean (-D warnings), all 43 tests pass. JVM oracle: compiles, answers `ergo_tree`/`validate`/`verify_avl`. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(difftest): persistent CI/nightly fuzz harness — fuzz_one + cargo-fuzz scaffold + coverage gate - ergo-difftest/src/fuzz.rs: stable fuzz_one(surface, data) entry point (real logic on stable; panics on Outcome::Bug → libFuzzer crash signal; silent on unknown surface; 5 unit tests: valid/garbage/unknown/all-surfaces/empty) - difftest --min-coverage <ratio>: machine-checkable coverage gate exits non-zero when union ratio < threshold; CI uses 0.80 (50k iters hits 1.000) - ergo-difftest/fuzz/: cargo-fuzz scaffold detached from workspace via [workspace] in fuzz/Cargo.toml (proven by clippy --workspace clean); 6 targets (ergo_tree, constant, ergo_box_candidate, transaction, header, sigma_expr) each a 3-line shim calling fuzz_one - ergo-difftest/fuzz/corpus/: 17 committed seed files from real mainnet vectors (failing_tree_*.hex, fee_proposition, height-1 header, genesis tx, recent box) + hand-crafted constant seeds - .github/workflows/ci.yml: +difftest job (stable, gating) — 50k iters + min-coverage 0.80 + fuzz_one unit tests on every PR/push to master - .github/workflows/fuzz.yml: nightly scheduled job (02:00 UTC) — 2M-iter structured campaign + corpus mutation run; NOT gating Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore(test-vectors): curate mainnet fixtures — retire 3 dead-weight files + manifest Retire 3 committed fixtures with ZERO consumers anywhere (verified across .rs/.toml/.sh/.md): input_boxes_205000_205200.json (2.1MB), ergotrees_700000_700200.json (0.6MB), boxes_recent_range.json (0.9MB). 101MB -> 97MB committed mainnet, zero coverage loss. Add FIXTURES.md — the curated keep-list mapping every remaining fixture to the CI-run key test that needs it + why it can't shrink. KEY FINDING: the committed ranges are NOT redundant bloat; each large file backs a distinct CI test with exact count assertions (e.g. chain_validate_1_10000's first multi-tx block is at height 3355, so a 1-1000 sibling loses unique coverage), and the streamed replay driver ADDS deep coverage but can't run in CI. So retirement beyond dead weight is a real CI-coverage trade — documented as optional, not taken. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(test-vectors): retire bulk contiguous ranges -> replay driver; curate to key uses Per the curation principle (committed CI fixtures cover each tx/script TYPE + each TRANSITION, not long contiguous runs), retire the two bulk contiguous ranges (~77MB) whose only unique coverage is cumulative state-accumulation -- which the streamed replay driver reproduces against the live chain, deeper and reproducibly (pinned by hash): - 1_10000 trio (tx+headers+digests, 27MB): deep test chain_validate_1_10000 DELETED; short-seed sanity stays in chain_validate_1_1000; deep coverage moves to replay --from 1 --to 10000. - 1761000_1762000 trio (tx+input_boxes+headers, 51MB): recent_block_validation DELETED; EIP-27 stays hermetic via reemission.rs (20 tests); EIP-37 boundary preserved via new curated headers_1761792_1761795_eip37_curated.json (4 hdrs). Rewired the 4 non-deep consumers (difficulty/header_validation/header_sync EIP-37 + decode v2 roundtrip) to the curated file; trimmed transactions_roundtrip broad cases; m7_mainnet_corpus skip-if-missing; deleted throwaway m7_fee_discovery. Wire the replay driver as the fuzz.yml replay job (gated on REPLAY_NODE_URL secret, no-ops on hosted runners). FIXTURES.md rewritten as the durable curated manifest. Committed mainnet: 101MB -> 20MB. Whole-workspace test + clippy green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(fuzz): name cargo-fuzz targets by surface so committed seeds auto-load cargo-fuzz auto-loads corpus/<target-name>, but targets were fuzz_<surface> while the committed seed corpora are corpus/<surface> — so seeds never loaded. Rename the 6 targets fuzz_<surface> -> <surface> (cargo fuzz run ergo_tree now auto-loads corpus/ergo_tree). Found by actually running cargo-fuzz on nightly. Nightly verification (1.98.0-nightly + cargo-fuzz 0.13.2): the ergo_tree target builds with ASan (48s) and fuzzes coverage-guided (~24.8k exec/s). It surfaced a parse->serialize fixed-point violation (read_ergo_tree accepts X; re-read of write(X) hits the type-recursion depth guard) that bounded stable campaigns missed. Oracle triage: node and JVM AGREE at every step (both accept X -> same canonical b1; both reject b1 on re-read) -> bug-for-bug PARITY with Scala, NOT a consensus divergence (a shared codec quirk / KnownArtifact). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(fuzz): nightly cargo-fuzz CI job + exclude the SER-003 type-depth guard from Bug class Add the cargo-fuzz-nightly job (fuzz.yml): matrix over the 6 surfaces, installs nightly + cargo-fuzz, bounded coverage-guided smoke per target seeded from the committed corpus, uploads crash artifacts. Non-gating; the deeper libFuzzer complement to the stable structured campaign. Verified live (nightly 1.98.0 + cargo-fuzz 0.13.2): targets build with ASan and fuzz coverage-guided (~24.8k exec/s). Fix the parse->serialize fixed-point check to NOT flag the type-recursion depth guard: MAX_TYPE_DEPTH (100) is a documented stack-overflow safeguard, not a consensus boundary (Scala TypeSerializer has no type-depth limit — only the 4096-byte proposition cap), i.e. QA SER-003 kept doc-only. A re-decode tripping ONLY that cap is that documented divergence on a near-boundary re-encoding, not a codec inconsistency -> WriteRejected, not Bug. Narrow: the value/expression tree-depth guard (Scala MaxTreeDepth=110) IS a real consensus limit and still counts as a Bug. Selftest still green (detector keeps teeth for real panics). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(fuzz): gitignore the fuzz crate Cargo.lock (regenerated on nightly) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(fuzz): CI compile blocker (worktree-relative include path) + CodeRabbit findings CI blocker: replay.rs include_str! used a 6-level '../' path that only resolved inside the nested .claude/worktrees checkout (reaching the sibling main repo); CI's clean checkout couldn't find it -> ergo-difftest failed to compile on all platforms. Correct to '../../../' (repo-root relative, works in both). CodeRabbit review (PR #155): - difftest.rs: route the 'validate' oracle surface to the 'transaction' generator (was falling back to ergo_tree -> both sides reject -> zero differential signal). - ErgoSerdeOracle.scala: bounds-check readBytes before Arrays.copyOfRange, which zero-pads (not throws) on a truncated verify_avl frame -> could silently accept. - avl_frame.rs: cap the untrusted op_count pre-allocation (multi-GB Vec DoS). - difftest.rs: reject --check-canonical without --repro (was a silent no-op). - regressions.rs: make the QUEUE.md append idempotent (no duplicate line on re-file). - ci.yml / fuzz.yml: add least-privilege 'permissions: contents: read'; drop the empty 'needs: []' (actionlint error). - interface-contracts.md: tag a fenced block. Verified locally: fmt --all, build --bins, clippy, test -p ergo-difftest, the --structured --min-coverage difftest job, both workflows' YAML, and the Scala oracle still compiles + answers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(fuzz): address CodeRabbit round 2 (still-valid findings) Verified each against current code; fixed the still-valid ones, skipped the already-fixed with reason. Fixed: - ci.yml/fuzz.yml: persist-credentials: false on the new-job checkouts (top-level permissions were already added); difftest job timeout-minutes: 15. - transactions_roundtrip.rs: drop the two retired ranges from EXPECTED_TX_FILES so the (ignored) manifest-coverage guard reflects the retirement. - minimize.rs: truncation phase now ascends to the SHORTEST satisfying prefix (the descending scan kept the longest — least reduction). Real minimizer bug. - difftest.rs: reuse the to_hex helper for the canonical-gate hex. - tests/minimize.rs: assert QUEUE.md has exactly one [PENDING] after re-file (covers the regressions.rs dedup). - m7_mainnet_corpus.rs: skip-guard now checks all three retired fixtures (txs/ headers/input_boxes), not just txs, so a partial extraction can't panic. - ErgoSerdeOracle.scala: unwrap InvocationTargetException from the reflective CAvlTreeVerifier ctor so REJECT reports the real cause, not the wrapper. Skipped (already fixed in d3b070a): validate->transaction routing, --check-canonical guard, replay include_str path, avl_frame op_count cap, QUEUE dedup, readBytes bounds, needs:[] removal, top-level permissions, md fence (text tag satisfies MD040). verify_avl structured fallback left as-is: no avl_frame generator exists, so it is exercised via the #6 re-injection trigger, not the structured sweep. Verified: fmt --all, clippy (difftest/ser/mempool), ergo-difftest tests, YAML, Scala oracle compiles + answers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: arkadianet <rkadias@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change
read_ergo_boxparses the 32-byte transaction id and the VLQ output index from the box tail, but the SBox-constantEvalBoxinsigma_to_valuehardcodedtransaction_id: [0u8;32]andoutput_index: 0, discarding them. This corrupted:transactionId.toBytes ++ Shorts.toByteArray(index)= 32-byte txId ++ fixed 2-byte big-endian index (ScalaErgoBox.get(ReferenceRegId)). Zeroing produced 34 zero bytes.32 + VLQ(index)bytes; a zeroed index (VLQ len 1) strips the wrong count for any index >= 128, leaving stray trailing bytes.Now carries the parsed
ergo_box.transaction_id/ergo_box.index.Conformance
sbox-const-discards-txid-index(Box_properties_equivalence 0xC7 docs: correct shipped-doc drift from the mining/digest PRs #10/feat(api): surface node mode identity on the overview page #11 and 0xC4 fix(ergo-node): survive digest-mode handshake, sync, and API seams #8).Tests
New
sbox_constant_preserves_txid_and_index_in_creation_infouses a multi-byte index (22588) so a zeroed index is unmistakable. Fullergo-sigmasuite green (312),cargo fmt/clippyclean.codex review: clean (no introduced correctness issues).
🤖 Generated with Claude Code