Skip to content

test(codegen): split test_codegen.py into 21 feature-focused files (#419) - #837

Merged
aallan merged 6 commits into
mainfrom
refactor/419-split-test-codegen
Jul 1, 2026
Merged

test(codegen): split test_codegen.py into 21 feature-focused files (#419)#837
aallan merged 6 commits into
mainfrom
refactor/419-split-test-codegen

Conversation

@aallan

@aallan aallan commented Jul 1, 2026

Copy link
Copy Markdown
Owner

What

Splits the monolithic tests/test_codegen.py (21,225 lines, 161 test classes — the largest file in the tree, and the codegen oracle for the mutation sweep) into twenty-one feature-focused test files, each under the issue's 1,500-line ceiling, with the shared harness extracted to a new tests/codegen_helpers.py. Mechanical and behaviour-preserving — the companion to the #420 test_checker.py split, and the last Tier 1 roadmap item.

How

New file Tests Lines Covers
test_codegen_expressions.py 89 787 literals, slot refs, arithmetic, comparison, boolean logic, if/let, calls, recursion, pipe
test_codegen_calls.py 32 1,403 statement-position unit calls, tail-call optimization (#517/#549), pair-typed closure params/captures (#535)
test_codegen_infrastructure.py 24 497 module assembly, execute error paths, E602 skips, builtin shadowing, typed holes, example round-trips
test_codegen_interpolation.py 33 1,275 string interpolation + the E615 loud fallthrough channel (#630)
test_codegen_effects.py 65 1,264 State/effect/Exn handlers, async futures, Random
test_codegen_data_types.py 67 1,339 ADT metadata/constructors, match, tuples, generic-mono regressions
test_codegen_arrays.py 82 1,427 Byte type, array lit/bounds/length/range/concat, compound arrays, array utilities
test_codegen_refinements.py 57 924 assert/assume, quantifiers, refinement aliases + runtime guards (#746)
test_codegen_strings.py 113 1,275 string literals + IO bindings, WAT escaping, core string ops, char classification, utilities
test_codegen_string_builtins.py 149 1,339 parse/base64/URL builtins, search/transform (#198), to-string (#106)
test_codegen_numeric.py 85 1,064 math builtins (#199), conversions (#208), Float64 predicates (#212)
test_codegen_io.py 42 822 IO operations (#135), Markdown + Regex host bindings
test_codegen_collections.py 59 888 Map + Set (#62), wrapper-handle tagging (#578)
test_codegen_json.py 59 984 Json collection + typed accessors
test_codegen_decimal.py 57 780 Decimal collection + monomorphization
test_codegen_host_effects.py 57 928 Html/Http/Inference, provider dispatch, postcondition host-import propagation (#823)
test_codegen_nat_guards.py 39 1,048 @Nat underflow (#520) + narrowing (#552/#747) runtime guards
test_codegen_translator_fixes.py 27 528 #475 call-translator regression fixes
test_codegen_gc_alloc.py 39 895 layout, bump allocator, GC core, shadow stack, grow/worklist overflow
test_codegen_gc_rooting.py 23 1,167 opaque-handle/host-walker rooting (#347/#490/#692/#695/#743)
test_codegen_gc_reclamation.py 21 708 transient reclamation (#573; stress trio #738), bucket occupancy (#706)
Total 1,219 (unchanged, incl. the 10 stress-marked)

The shared harness — the eleven _compile* / _run* / WAT-and-GC assertion helpers (four of which lived interspersed between classes mid-file) plus the _IO_PRELUDE / _INLINE_BUILTIN_NAMES fixture constants — moves to tests/codegen_helpers.py, imported per file via the repo's from tests.<module> import pattern. The six pre-existing test_codegen_* modules (closures, contracts, coverage, invariant_e699, modules, monomorphize) are untouched.

Faithfulness proof

Every class, helper def, and constant moves byte-for-byte: a per-block differential against git show HEAD:tests/test_codegen.py confirms 174/174 blocks byte-identical (161 classes + 11 defs + 2 consts), each appearing exactly once. Per-file import blocks are computed from actual body usage — not via ruff --fix, which rewrites function-local imports inside test bodies (the #420 lesson) — and ruff check passes without --fix.

An independent 4-auditor adversarial verification pass (faithfulness / hygiene / docs / AC, each finding judged by a separate refuter) ran before commit. It confirmed the split faithful and surfaced stale cross-references the initial sweep missed, all fixed in this PR: 13 # overlaps pointers in test_execute_characterization.py, 6 browser-parity docstrings in test_browser.py, 2 in vera/codegen/assembly.py (one emitted into generated WAT comments for allocation-using modules), 1 in vera/environment.py, 4 more single-ref test files, 5 conformance .vera headers — plus the one column-0 @pytest.mark.stress decorator the split seam had (legally but visibly) separated from its class, now re-attached flush.

CodeRabbit's two review rounds then drove post-verification hygiene commits (distinct from the pure-move commit, which stays provably byte-identical in this PR's history): the compile helpers no longer leak their temp .vera file and gained full type annotations, the check_doc_counts.py empty-table sentinel is pinned by two mutation-validated regression tests, and the four carried-over test-body idioms were modernised in-PR per maintainer preference (as_posix() path normalisation, parse_to_ast instead of the tempfile parse dance, a shared _compile_example() helper, and the precondition-trap assertion narrowed to the empirically-confirmed WasmTrapError kind == "contract_violation") — closing #838.

Deviations from the issue text (both precedented)

Roadmap: Tier 1 closed

#419 was the last Tier 1 item, so the ROADMAP's Tier 1 section collapses to a done-sentence alongside Tier 0's, and the tier count in the intro drops to two. KNOWN_ISSUES' "Refactoring needed" table is now empty → replaced with No files currently need decomposition. (the No known bugs. convention), with scripts/check_doc_counts.py taught to accept that state.

No version bump

Mechanical refactor — recorded as a CHANGELOG [Unreleased] bullet; rides the next (feature) release.

Verification

  • per-block byte-identity differential vs HEAD → 174/174
  • pytest --collect-only -m ""1,219 (== original; 1,209 + 10 stress)
  • the 21 files → 1,209 passed, 10 deselected
  • full pytest tests/ → unchanged totals (5,511 passed / 22 by-design skips)
  • pytest tests/ -m stressall 26 stress tests pass (incl. the moved Mark the TestHostHandleReclamation573 trio as stress tests #738 GC-reclamation trio, 3m07s) — everything normally deselected was actually run
  • all 103 conformance programs pass (5 header-comment updates included)
  • ruff check (no --fix) → clean
  • python scripts/check_doc_counts.py → consistent (5,559 tests / 80 files)
  • python scripts/check_site_assets.py, check_version_sync.py → green (no bump)
  • pyproject.toml [tool.mutmut] oracle selection → the 21 new files (dangling-path check: all exist)

Closes #419
Closes #838

Summary by CodeRabbit

  • Testing
    • Reorganised the monolithic code generation suite into many feature-focused test_codegen_*.py modules.
    • Added shared codegen helpers to standardise compilation, execution, and common WAT/GC assertions.
    • Expanded end-to-end and WAT-pinned coverage across expressions, calls/tail calls, arrays, collections, data types, decimals, effects, GC allocation/rooting/reclamation, host effects, IO, JSON, numerics, interpolation, and refinements.
  • Documentation
    • Updated contributor/testing guidance and references (changelog/roadmap/known issues) to match the new layout.
  • Maintenance
    • Updated mutation-testing selection and enhanced refactoring-count checking (including an “empty table” convention).

)

The monolithic tests/test_codegen.py (21,225 lines, 161 classes -- the
largest file in the tree and the codegen mutation oracle) is split into
twenty-one feature files, each under the issue's 1,500-line ceiling,
alongside the six pre-existing test_codegen_* modules (untouched):

  expressions 89   calls 32          infrastructure 24  interpolation 33
  effects 65       data_types 67     arrays 82          refinements 57
  strings 113      string_builtins 149  numeric 85      io 42
  collections 59   json 59           decimal 57         host_effects 57
  nat_guards 39    translator_fixes 27  gc_alloc 39     gc_rooting 23
  gc_reclamation 21                              total: 1,219 (unchanged)

The shared harness moves to a new tests/codegen_helpers.py: the eleven
_compile*/_run*/WAT-and-GC assertion helpers (four of which lived
interspersed between classes mid-file) plus the _IO_PRELUDE and
_INLINE_BUILTIN_NAMES fixture constants, preserved in original order.

Mechanical and behaviour-preserving: every class, helper def, and
constant moves byte-for-byte -- verified by a per-block differential
against git HEAD (174/174 blocks byte-identical). All 1,219 tests
(including the 10 stress-marked) carry over unchanged; 1,209 pass with
the default addopts, 10 deselect. Per-file import blocks are computed
from actual body usage (not via ruff --fix, which rewrites function-
local imports inside test bodies); ruff is clean without --fix.

The issue's original 9-file plan was drawn at 10,019 lines / 118
classes; at 21,225 / 161 the same theme boundaries yield 21 files.
Helpers use a plain tests/codegen_helpers.py module rather than the
issue's conftest.py suggestion, matching the #420 precedent (they are
assertion helpers, not fixtures).

An adversarial 4-auditor verification pass (faithfulness / hygiene /
docs / AC) ran pre-commit and surfaced stale cross-references the
initial sweep missed; all fixed here: tests/test_execute_
characterization.py (13 "# overlaps" pointers), tests/test_browser.py
(6 browser-parity docstrings), vera/codegen/assembly.py (2 -- one
emitted into generated WAT comments for allocation-using modules),
vera/environment.py (clamp fallthrough note), test_runtime_traps /
test_int_overflow_codegen / test_string_length_soundness /
test_check_changelog_updated (1 each), 5 conformance .vera headers,
and the single column-0 @pytest.mark.stress decorator re-attached
flush to its class (the split seam had legally-but-uglily separated
them; the stress trio was run explicitly to confirm the marker held:
26 stress tests pass).

Docs lockstep: TESTING.md (21-row table, 60 -> 80 file count, helpers
prose, workflow step, feature-mapping refs), pyproject [tool.mutmut]
oracle selection (the functional one -- a dangling path would silently
zero the codegen sweep's oracle), CONTRIBUTING.md, KNOWN_ISSUES.md
("Refactoring needed" now empty -> "No files currently need
decomposition.", with scripts/check_doc_counts.py taught the empty-
section convention), CHANGELOG.md [Unreleased]. ROADMAP.md: #419 was
the last Tier 1 item, so the Tier 1 section collapses to a done-
sentence alongside Tier 0's. No version bump (no functional change).

Closes #419

Co-Authored-By: Claude <noreply@anthropic.invalid>
@coderabbitai

coderabbitai Bot commented Jul 1, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 2fec3cc7-8e41-4eca-9d03-b32f43b593f0

📥 Commits

Reviewing files that changed from the base of the PR and between 2d5e6ae and 10caeb9.

⛔ Files ignored due to path filters (5)
  • tests/conformance/ch07_io_read_char.vera is excluded by !**/*.vera
  • tests/conformance/ch07_io_sleep.vera is excluded by !**/*.vera
  • tests/conformance/ch07_io_time_stderr.vera is excluded by !**/*.vera
  • tests/conformance/ch07_random_effect.vera is excluded by !**/*.vera
  • tests/conformance/ch09_math_builtins.vera is excluded by !**/*.vera
📒 Files selected for processing (39)
  • CHANGELOG.md
  • CONTRIBUTING.md
  • KNOWN_ISSUES.md
  • ROADMAP.md
  • TESTING.md
  • pyproject.toml
  • scripts/check_doc_counts.py
  • tests/codegen_helpers.py
  • tests/test_browser.py
  • tests/test_check_changelog_updated.py
  • tests/test_check_doc_counts.py
  • tests/test_codegen.py
  • tests/test_codegen_arrays.py
  • tests/test_codegen_calls.py
  • tests/test_codegen_collections.py
  • tests/test_codegen_data_types.py
  • tests/test_codegen_decimal.py
  • tests/test_codegen_effects.py
  • tests/test_codegen_expressions.py
  • tests/test_codegen_gc_alloc.py
  • tests/test_codegen_gc_reclamation.py
  • tests/test_codegen_gc_rooting.py
  • tests/test_codegen_host_effects.py
  • tests/test_codegen_infrastructure.py
  • tests/test_codegen_interpolation.py
  • tests/test_codegen_io.py
  • tests/test_codegen_json.py
  • tests/test_codegen_nat_guards.py
  • tests/test_codegen_numeric.py
  • tests/test_codegen_refinements.py
  • tests/test_codegen_string_builtins.py
  • tests/test_codegen_strings.py
  • tests/test_codegen_translator_fixes.py
  • tests/test_execute_characterization.py
  • tests/test_int_overflow_codegen.py
  • tests/test_runtime_traps.py
  • tests/test_string_length_soundness.py
  • vera/codegen/assembly.py
  • vera/environment.py
🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

  • aallan/vera-bench (manual)

📝 Walkthrough

Walkthrough

The PR splits tests/test_codegen.py into feature-specific modules, adds shared helpers, and updates documentation, tooling, and path references to the new layout.

Changes

Codegen test suite split

Layer / File(s) Summary
Docs and tooling
CHANGELOG.md, CONTRIBUTING.md, KNOWN_ISSUES.md, ROADMAP.md, TESTING.md, pyproject.toml, scripts/check_doc_counts.py, tests/test_check_doc_counts.py, tests/test_check_changelog_updated.py
Docs, roadmap, changelog, mutmut selection, doc-count handling, and path checks are updated for the split test layout and the empty refactoring-section convention.
Shared test helpers
tests/codegen_helpers.py
A new helper module provides shared compile, execute, trap, state, generator, host-import, and reclamation utilities.
Expressions, calls, and infrastructure
tests/test_codegen_expressions.py, tests/test_codegen_calls.py, tests/test_codegen_infrastructure.py
New modules cover expression codegen, tail-call and closure behaviour, and module assembly plus execute-path error handling.
Data types, collections, and arrays
tests/test_codegen_data_types.py, tests/test_codegen_collections.py, tests/test_codegen_arrays.py
New modules cover ADT metadata and constructors, Map/Set behaviour and wrapper tagging, and array builtins and utilities.
Effects, host effects, IO, JSON, decimal, and numeric
tests/test_codegen_effects.py, tests/test_codegen_host_effects.py, tests/test_codegen_io.py, tests/test_codegen_json.py, tests/test_codegen_decimal.py, tests/test_codegen_numeric.py
New modules cover State/Exn/Async/Random effects, HTML/HTTP/Inference host builtins, IO/Markdown/Regex, JSON accessors, Decimal operations, and numeric builtins.
GC allocation, reclamation, rooting, and Nat guards
tests/test_codegen_gc_alloc.py, tests/test_codegen_gc_reclamation.py, tests/test_codegen_gc_rooting.py, tests/test_codegen_nat_guards.py
New modules cover GC allocation and collection structure, wrapper reclamation, shadow-stack rooting, host-handle rooting, and Nat underflow and narrowing guards.
Interpolation and refinements
tests/test_codegen_interpolation.py, tests/test_codegen_refinements.py
New modules cover interpolation inference and diagnostics, plus refinement aliases, quantifiers, and runtime refinement guards.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related issues

Possibly related PRs

  • aallan/vera#693: Directly related test-suite split and shared harness/oracle wiring.
  • aallan/vera#761: Related Nat-guard regression coverage now housed in the split codegen suite.

Suggested labels: tests, ci, docs

🚥 Pre-merge checks | ✅ 8
✅ Passed checks (8 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarises the main mechanical change: splitting tests/test_codegen.py into feature-focused files.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Changelog Covers Public-Surface Changes ✅ Passed The PR is a mechanical test/docs split; no CLI, diagnostics, spec, LSP, or codegen API surface changes are shown, and CHANGELOG.md documents the split.
Spec And Implementation Move Together ✅ Passed Only tests/codegen_helpers.py changed; there are no vera/ or spec/ diffs, so no spec/implementation drift to check.
Diagnostics Carry An Error Code ✅ Passed No new/changed compiler diagnostics in this PR; it’s docs/tests plus a doc-count script, and the diagnostic-field checker passes.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch refactor/419-split-test-codegen

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Jul 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.06%. Comparing base (2d5e6ae) to head (10caeb9).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #837   +/-   ##
=======================================
  Coverage   92.06%   92.06%           
=======================================
  Files          90       90           
  Lines       27321    27321           
  Branches      324      324           
=======================================
  Hits        25154    25154           
  Misses       2159     2159           
  Partials        8        8           
Flag Coverage Δ
javascript 65.58% <ø> (ø)
python 95.06% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@aallan

aallan commented Jul 1, 2026

Copy link
Copy Markdown
Owner Author

Review round 1 — pr-review-toolkit (code-reviewer)

Clean verdict — no critical or important issues. The reviewer focused on the surfaces the deterministic gates don't cover and verified each against the working tree:

  • Counts reconcile exactly: all 21 TESTING.md rows match live pytest collection (incl. nat_guards = 39 via parametrize), summing to 1,219; the 10 stress tests deselect as documented; the doc-counts gate reports 80 files.
  • tests/codegen_helpers.py correct: all 11 helpers + both fixture constants migrated; _INLINE_BUILTIN_NAMES precedes its consumer def; no shared helper left duplicated in any split file (the one _run_bool in test_codegen_strings.py is a pre-existing class-local method, identical in the original).
  • vera/ diffs are comment-only (assembly.py, environment.py) — no code change.
  • scripts/check_doc_counts.py empty-table carve-out is sound: it only accepts the exact sentinel sentence, so a malformed (rather than deliberately emptied) table still trips the gate — no new masking.
  • ROADMAP Tier 1 collapse reads coherently ("two remaining tiers", Mutation testing beyond the soundness core: whole-vera/ sweep + full-sweep reliability #795 prose retargeted); CHANGELOG bullet claims all verify; conventions clean (backticked @Type, no #N misuse).

One low-priority suggestion, deliberately held rather than spending a CI round on it: the pyproject.toml also_copy comment cites test_codegen_infrastructure as the exemplar that reads examples/, but _effects and _strings also do — functionally harmless (also_copy copies the whole directory) and no less precise than the pre-split comment. Will fold in if another commit becomes necessary.

Single HISTORY.md reference at the end instead of one per tier, per
review. Also folds in the pr-review-toolkit round-1 suggestion: the
pyproject [tool.mutmut] also_copy comment now names all three split
files whose tests read examples/ from disk (_infrastructure, _effects,
_strings), not just the first.

Skip-changelog: prose-only tweak to an [Unreleased]-covered PR (roadmap wording + a config comment)

Co-Authored-By: Claude <noreply@anthropic.invalid>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/check_doc_counts.py`:
- Around line 44-49: The sentinel branch in check_doc_counts.py that returns an
empty list for "No files currently need decomposition." is untested, so add a
regression test in tests/test_check_doc_counts.py that explicitly exercises the
parse/check path for that exact message and asserts [] is returned. Use the
existing check_doc_counts flow and the empty-table case as a guide, but make the
new test pin the sentinel string so changes to KNOWN_ISSUES.md wording will fail
loudly.

In `@tests/codegen_helpers.py`:
- Line 141: Add proper type hints to the helper signatures in codegen_helpers:
annotate _compile_with_generator with its return type, and replace the inline
comment on _assert_chain_reclaims’s chain parameter with an explicit Callable
type annotation. Update the module imports to include Callable from typing or
collections.abc so both function signatures comply with the repo’s typing
guidelines.
- Around line 32-46: The _compile helper currently creates a temporary .vera
file with delete=False and never removes it, causing a leak on every call.
Update _compile in codegen_helpers to clean up the temp file after
parse_file/transform/compile complete, while keeping the Windows-safe
create-then-read flow intact. Use the existing _compile, parse_file, transform,
and compile symbols to place the cleanup in the right spot so the temp path is
always unlinked even if parsing or compilation fails.

In `@tests/test_codegen_io.py`:
- Around line 187-215: Replace the manual path normalization in the IO tests
with the preferred pathlib-based conversion. In the affected test helper code
that builds `vera_path` from `tmp_path` (and the analogous `tmp_file` usage in
the roundtrip test), use `Path(...).as_posix()` instead of `replace(os.sep,
"/")`, and add the needed `Path` import near the test imports so the Vera source
string still receives a POSIX-style path.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 5fdd740b-1bd9-4ac3-a806-0b245c1abf42

📥 Commits

Reviewing files that changed from the base of the PR and between 2d5e6ae and 3e01879.

⛔ Files ignored due to path filters (5)
  • tests/conformance/ch07_io_read_char.vera is excluded by !**/*.vera
  • tests/conformance/ch07_io_sleep.vera is excluded by !**/*.vera
  • tests/conformance/ch07_io_time_stderr.vera is excluded by !**/*.vera
  • tests/conformance/ch07_random_effect.vera is excluded by !**/*.vera
  • tests/conformance/ch09_math_builtins.vera is excluded by !**/*.vera
📒 Files selected for processing (38)
  • CHANGELOG.md
  • CONTRIBUTING.md
  • KNOWN_ISSUES.md
  • ROADMAP.md
  • TESTING.md
  • pyproject.toml
  • scripts/check_doc_counts.py
  • tests/codegen_helpers.py
  • tests/test_browser.py
  • tests/test_check_changelog_updated.py
  • tests/test_codegen.py
  • tests/test_codegen_arrays.py
  • tests/test_codegen_calls.py
  • tests/test_codegen_collections.py
  • tests/test_codegen_data_types.py
  • tests/test_codegen_decimal.py
  • tests/test_codegen_effects.py
  • tests/test_codegen_expressions.py
  • tests/test_codegen_gc_alloc.py
  • tests/test_codegen_gc_reclamation.py
  • tests/test_codegen_gc_rooting.py
  • tests/test_codegen_host_effects.py
  • tests/test_codegen_infrastructure.py
  • tests/test_codegen_interpolation.py
  • tests/test_codegen_io.py
  • tests/test_codegen_json.py
  • tests/test_codegen_nat_guards.py
  • tests/test_codegen_numeric.py
  • tests/test_codegen_refinements.py
  • tests/test_codegen_string_builtins.py
  • tests/test_codegen_strings.py
  • tests/test_codegen_translator_fixes.py
  • tests/test_execute_characterization.py
  • tests/test_int_overflow_codegen.py
  • tests/test_runtime_traps.py
  • tests/test_string_length_soundness.py
  • vera/codegen/assembly.py
  • vera/environment.py
🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

  • aallan/vera-bench (manual)

Comment thread scripts/check_doc_counts.py
Comment thread tests/codegen_helpers.py Outdated
Comment thread tests/codegen_helpers.py Outdated
Comment thread tests/test_codegen_io.py
…st (#419)

Three of the four findings fixed (the fourth skipped as out of scope,
reasoning in the PR thread):

- tests/codegen_helpers.py: _compile and _compile_with_generator now
  unlink their temp .vera file in a finally (the delete=False +
  manual-unlink Windows-safe pattern from CLAUDE.md; previously every
  call leaked one temp file). _compile_with_generator gains its
  tuple[CompileResult, CodeGenerator] return annotation (CodeGenerator
  promoted to the module import block) and _assert_chain_reclaims's
  chain param is now Callable[[int], str] instead of comment-typed.
  These are deliberate post-verification improvements to the new
  helpers module -- distinct from the byte-identical mechanical move
  proven at 3e01879.
- tests/test_check_doc_counts.py: two regression tests pin the #419
  empty-section sentinel in scripts/check_doc_counts.py -- the exact
  sentence passes, and any other rowless wording still errors (so the
  carve-out cannot mask a malformed table). Mutation-validated: the
  sentinel test goes RED when the carve-out is removed.
- TESTING.md / ROADMAP.md counts follow (+2 tests -> 5,561).

Validated: 1,209 codegen tests + 17 doc-counts tests green, ruff clean,
doc-counts gate green.

Skip-changelog: test-helper hygiene within the #419 PR, covered by its existing [Unreleased] bullet

Co-Authored-By: Claude <noreply@anthropic.invalid>
@aallan

aallan commented Jul 1, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Jul 1, 2026

Copy link
Copy Markdown
✅ Action performed

Full review finished.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/test_codegen_gc_rooting.py`:
- Around line 146-164: Replace the tempfile workflow in the affected gc-rooting
tests with the existing parser helper: use parse_to_ast(src) instead of
NamedTemporaryFile + parse_file(path) + transform(tree), and pass the resulting
AST directly into compile just like the sibling nat-guards tests. Update the
three test methods named test_map_param_shadow_pushed_after_573,
test_set_param_shadow_pushed_after_573, and
test_decimal_param_shadow_pushed_after_573 accordingly, and remove the unused
temp file path handling (including delete=False) since no on-disk .vera file is
needed.

In `@tests/test_codegen_infrastructure.py`:
- Around line 89-190: The TestExampleRoundTrips methods repeat the same
example-file loading and compile pipeline, so extract that boilerplate into a
shared helper in tests/codegen_helpers.py and reuse it at the call sites. Add a
helper like _compile_example that encapsulates the
Path/read_text/parse_file/transform/compile flow, then update the example
round-trip tests to call it instead of duplicating the setup. Keep the existing
compile/execute helpers and align with the established _compile_ok/_run style
used elsewhere in tests/codegen_helpers.py.
- Around line 139-149: The trap assertion in test_safe_divide_trap_on_zero is
too broad and should target the normalized WasmTrapError raised by execute(...).
Update the test to use pytest.raises(WasmTrapError) and then assert the captured
exception’s kind is "contract_violation", so the check verifies the specific
normalized contract-failure path instead of accepting generic wasmtime or
RuntimeError shapes.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: debbfc56-7131-4b60-9d26-f6ee3165dff6

📥 Commits

Reviewing files that changed from the base of the PR and between 2d5e6ae and 7076eb4.

⛔ Files ignored due to path filters (5)
  • tests/conformance/ch07_io_read_char.vera is excluded by !**/*.vera
  • tests/conformance/ch07_io_sleep.vera is excluded by !**/*.vera
  • tests/conformance/ch07_io_time_stderr.vera is excluded by !**/*.vera
  • tests/conformance/ch07_random_effect.vera is excluded by !**/*.vera
  • tests/conformance/ch09_math_builtins.vera is excluded by !**/*.vera
📒 Files selected for processing (39)
  • CHANGELOG.md
  • CONTRIBUTING.md
  • KNOWN_ISSUES.md
  • ROADMAP.md
  • TESTING.md
  • pyproject.toml
  • scripts/check_doc_counts.py
  • tests/codegen_helpers.py
  • tests/test_browser.py
  • tests/test_check_changelog_updated.py
  • tests/test_check_doc_counts.py
  • tests/test_codegen.py
  • tests/test_codegen_arrays.py
  • tests/test_codegen_calls.py
  • tests/test_codegen_collections.py
  • tests/test_codegen_data_types.py
  • tests/test_codegen_decimal.py
  • tests/test_codegen_effects.py
  • tests/test_codegen_expressions.py
  • tests/test_codegen_gc_alloc.py
  • tests/test_codegen_gc_reclamation.py
  • tests/test_codegen_gc_rooting.py
  • tests/test_codegen_host_effects.py
  • tests/test_codegen_infrastructure.py
  • tests/test_codegen_interpolation.py
  • tests/test_codegen_io.py
  • tests/test_codegen_json.py
  • tests/test_codegen_nat_guards.py
  • tests/test_codegen_numeric.py
  • tests/test_codegen_refinements.py
  • tests/test_codegen_string_builtins.py
  • tests/test_codegen_strings.py
  • tests/test_codegen_translator_fixes.py
  • tests/test_execute_characterization.py
  • tests/test_int_overflow_codegen.py
  • tests/test_runtime_traps.py
  • tests/test_string_length_soundness.py
  • vera/codegen/assembly.py
  • vera/environment.py
🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

  • aallan/vera-bench (manual)

Comment thread tests/test_codegen_gc_rooting.py Outdated
Comment thread tests/test_codegen_infrastructure.py
Comment thread tests/test_codegen_infrastructure.py
T and others added 2 commits July 1, 2026 22:34
CodeRabbit's round-2 pass on PR #837 flagged three more pre-existing
idioms inside byte-identically moved test bodies (tempfile parse dance
in gc_rooting, example-loading boilerplate and a broad trap assertion
in infrastructure); together with round 1's os.sep path normalisation
they are collected in #838 rather than modified mid-split, per the
PR's byte-identical-move contract (the #420 -> #835 pattern).

Co-Authored-By: Claude <noreply@anthropic.invalid>
Per maintainer preference these land in this PR rather than deferring
(#838 closes with it; the byte-identical mechanical move remains
provable at 3e01879 in this PR's history):

- test_codegen_io.py: tmp_path/tmp_file.replace(os.sep, "/") ->
  Path(...).as_posix() (the CLAUDE.md fixture convention).
- test_codegen_gc_rooting.py: the three shadow-push tests drop the
  NamedTemporaryFile + parse_file + transform dance for
  parse_to_ast(src) -- one line, no file on disk, and removes the
  three per-run tempfile leaks.
- test_codegen_infrastructure.py: the eight TestExampleRoundTrips
  methods route through a new codegen_helpers._compile_example(name)
  instead of repeating the Path -> read_text -> parse_file ->
  transform -> compile sequence.
- test_codegen_infrastructure.py: the safe_divide precondition test
  narrows pytest.raises((WasmtimeError, Trap, RuntimeError)) to
  WasmTrapError with kind == "contract_violation" -- empirically
  confirmed as the normalised shape execute() produces (CodeRabbit's
  sketch had the import path wrong: vera.codegen.api, not
  vera.wasm.trap).

Validated: 1,209 codegen tests green, ruff clean without --fix,
doc-counts green (TESTING.md line counts follow; the KNOWN_ISSUES #838
row added earlier in this PR is removed again -- fixed, not deferred).

Skip-changelog: test-hygiene within the #419 PR, covered by its existing [Unreleased] bullet

Co-Authored-By: Claude <noreply@anthropic.invalid>
@aallan

aallan commented Jul 1, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Jul 1, 2026

Copy link
Copy Markdown
✅ Action performed

Full review finished.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

♻️ Duplicate comments (1)
pyproject.toml (1)

129-132: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Comment names only one example file per category; harmless but could be more explicit.

The comment says "example round-trips in test_codegen_infrastructure / _effects / _strings" but doesn't name every split file that reads from examples/ on disk (per PR discussion). Functionally harmless since the CLI-args list below already includes all files needed by the in-process oracle.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pyproject.toml` around lines 129 - 132, Clarify the repo-files comment in
pyproject.toml by naming all split example tests that read from examples/ on
disk, not just one representative per category. Update the existing explanatory
comment near the in-process oracle copy list to explicitly mention the relevant
test_codegen_infrastructure, test_effects, test_strings, and test_errors cases
so the intent is unambiguous while keeping the current file list unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/codegen_helpers.py`:
- Around line 44-56: The tempfile cleanup in the helper that creates the source
file can fail because `path` is assigned only after `write()` and `flush()`
succeed; move the filename capture before the first write so the `finally` block
can always unlink it, even if writing fails. Apply the same fix in both helper
functions, including `_compile_with_generator`, and keep using `parse_file`,
`transform`, and `compile` unchanged.
- Around line 107-113: The _run_trap helper is too broad because it catches
multiple exception types from execute(...) instead of the normalized trap type.
Update _run_trap in codegen_helpers to assert only WasmTrapError, and import
that symbol so the test targets the specific trap path exposed by execute(...)
rather than unrelated RuntimeError or wasmtime exceptions.

---

Duplicate comments:
In `@pyproject.toml`:
- Around line 129-132: Clarify the repo-files comment in pyproject.toml by
naming all split example tests that read from examples/ on disk, not just one
representative per category. Update the existing explanatory comment near the
in-process oracle copy list to explicitly mention the relevant
test_codegen_infrastructure, test_effects, test_strings, and test_errors cases
so the intent is unambiguous while keeping the current file list unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 82ece338-3513-451b-94eb-99403b4ab362

📥 Commits

Reviewing files that changed from the base of the PR and between 2d5e6ae and c86a4ed.

⛔ Files ignored due to path filters (5)
  • tests/conformance/ch07_io_read_char.vera is excluded by !**/*.vera
  • tests/conformance/ch07_io_sleep.vera is excluded by !**/*.vera
  • tests/conformance/ch07_io_time_stderr.vera is excluded by !**/*.vera
  • tests/conformance/ch07_random_effect.vera is excluded by !**/*.vera
  • tests/conformance/ch09_math_builtins.vera is excluded by !**/*.vera
📒 Files selected for processing (39)
  • CHANGELOG.md
  • CONTRIBUTING.md
  • KNOWN_ISSUES.md
  • ROADMAP.md
  • TESTING.md
  • pyproject.toml
  • scripts/check_doc_counts.py
  • tests/codegen_helpers.py
  • tests/test_browser.py
  • tests/test_check_changelog_updated.py
  • tests/test_check_doc_counts.py
  • tests/test_codegen.py
  • tests/test_codegen_arrays.py
  • tests/test_codegen_calls.py
  • tests/test_codegen_collections.py
  • tests/test_codegen_data_types.py
  • tests/test_codegen_decimal.py
  • tests/test_codegen_effects.py
  • tests/test_codegen_expressions.py
  • tests/test_codegen_gc_alloc.py
  • tests/test_codegen_gc_reclamation.py
  • tests/test_codegen_gc_rooting.py
  • tests/test_codegen_host_effects.py
  • tests/test_codegen_infrastructure.py
  • tests/test_codegen_interpolation.py
  • tests/test_codegen_io.py
  • tests/test_codegen_json.py
  • tests/test_codegen_nat_guards.py
  • tests/test_codegen_numeric.py
  • tests/test_codegen_refinements.py
  • tests/test_codegen_string_builtins.py
  • tests/test_codegen_strings.py
  • tests/test_codegen_translator_fixes.py
  • tests/test_execute_characterization.py
  • tests/test_int_overflow_codegen.py
  • tests/test_runtime_traps.py
  • tests/test_string_length_soundness.py
  • vera/codegen/assembly.py
  • vera/environment.py
🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

  • aallan/vera-bench (manual)

Comment thread tests/codegen_helpers.py Outdated
Comment thread tests/codegen_helpers.py
…ime (#419)

- _compile / _compile_with_generator restructured so the try/finally
  encloses the tempfile from creation: the temp .vera is unlinked even
  when the write itself raises. (CodeRabbit's minimal reorder -- capture
  f.name before the write -- would not have achieved that: the cleanup
  lived in a later try block that a write failure never entered. The
  close-then-read shape is also strictly Windows-friendlier: the file is
  fully closed before parse_file reopens it.)
- _run_trap narrows pytest.raises((WasmtimeError, Trap, RuntimeError))
  to the normalised WasmTrapError, empirically validated: all 1,209
  codegen-family tests -- every caller of the shared helper -- pass with
  the narrowed type. The three pre-existing test_codegen_* modules keep
  their own local helpers and are untouched.

Skip-changelog: test-helper hygiene within the #419 PR, covered by its existing [Unreleased] bullet

Co-Authored-By: Claude <noreply@anthropic.invalid>
@aallan

aallan commented Jul 1, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Jul 1, 2026

Copy link
Copy Markdown
✅ Action performed

Full review finished.

@aallan

aallan commented Jul 1, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai Re the duplicate comment on the pyproject.toml also_copy explanation (lines 128–132): this was already addressed in 3119dce — the comment now names all four cases that read repo files (the example round-trips in test_codegen_infrastructure / _effects / _strings, and test_errors' error-display-sync against README / docs / spec), with the also_copy file list itself unchanged, exactly as the suggestion asks. No further change needed.

@coderabbitai

coderabbitai Bot commented Jul 1, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
git show 3119dce -- pyproject.toml
sed -n '110,140p' pyproject.toml

Length of output: 2098


@aallan Thanks for the pointer — that matches the ask exactly (naming all four sites that read repo files under also_copy, with the file list itself correctly left untouched). No further action needed on this one from my side.

@aallan
aallan merged commit ce69864 into main Jul 1, 2026
28 checks passed
@aallan
aallan deleted the refactor/419-split-test-codegen branch July 1, 2026 22:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Modernise carried-over test-body idioms in the split test_codegen_* files Split tests/test_codegen.py (10,019 lines) into feature-focused test files

1 participant