Skip to content

Parity: shard the determinism check per fixture - #5

Merged
thejackshelton merged 1 commit into
masterfrom
pr/parity-determinism-sharding
Sep 29, 2026
Merged

thejackshelton merged 1 commit into
masterfrom
pr/parity-determinism-sharding

Conversation

@thejackshelton

@thejackshelton thejackshelton commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Shards the parity determinism check (S5 (c)) into one test per fixture, with no loss of coverage.

Why. The single determinism test compiled every fixture twice, in both environments, with shuffled order-free lists, all inside vitest's fixed 120 s timeout. On an idle machine it already took about 101 s. Under load it timed out in almost every full pnpm test run, and every PR that adds fixtures pushes it closer to the limit. It also couldn't run alone under -t determinism, because it relied on captures made by earlier tests.

What changed (only packages/parity/test/parity.test.ts):

  • One test per fixture. determinism (S5 (c)) is now a block with one test per fixture (195), grouped by fixture group. Each test's timeout is 30 s per environment it runs in. That is about 9× the slowest measured shard.
  • Same checks per fixture. Each shard does what the old loop did for that fixture: compile twice plus once shuffled in every environment, rerun both lanes on the shuffled input, and compare byte-for-byte with the normal pass.
  • Runs alone. Shards use live captures when the earlier fixture tests have run. Otherwise they use the committed capture files and recompute the normal pass themselves.
  • Coverage test. A new final test checks the counts, all derived from the fixture list: every fixture checked, 2 compile comparisons per fixture environment, 1 lane comparison per fixture, and at least 10 fixtures reordered by the shuffle. The shuffle-count assertion moved here from the old test.

What passed:

  • pnpm typecheck
  • pnpm test: 74 files, 1703/1703
  • pnpm exec vitest run packages/parity/test/parity.test.ts -t determinism: 197 passed on its own

The shards sum to 133 s; the slowest is 5.0 s.

A planted fault confirms the check still works. I made compileFixture reverse tree-ordered-sheets' style order on every second call. Only that fixture's shard failed, on a digest mismatch, and the coverage test reported 194/195. The fault was then reverted.

Changed checks:

  • The per-fixture timeout replaces the single fixed 120 s timeout. No assertion was loosened or removed.

🤖 Generated with Claude Code

Note

Shard parity determinism tests per fixture

Replaces the single aggregate determinism test in parity.test.ts with one test per fixture, grouped via the FIXTURE_GROUPS registry. Each test uses a timeout scaled to its environment count and runs the existing repeated, shuffled, and lane comparisons.

  • Authored captures come from the live capture map, with committed files as fallback; missing main outcomes are recomputed.
  • Adds a coverage test with shared counters that verifies every fixture ran and the expected compile and lane comparisons occurred, including at least 10 reordered fixtures.

Macroscope summarized 325ddb3.

@macroscopeapp

macroscopeapp Bot commented Sep 29, 2026

Copy link
Copy Markdown

Approvability

Verdict: Approved at 325ddb3

Macroscope's review found this PR approvable — This PR only restructures the private parity test harness into per-fixture shards, preserving the existing determinism assertions and adding coverage checks. Its runtime effects are limited to CI test scheduling, timeouts, and capture handling; no shipped or customer-facing behavior changes.

You can add or adjust custom eligibility rules. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant