land: batched landing (one proof per batch, prefix bisect, per-PR merges in order)

[thejackshelton]

What changed (head 5d4276d): pnpm land proves PRs in batches. It runs one full pnpm test for up to LAND_BATCH PRs (default 4, 1 to 8). Each PR still gets its own landing commit, its own gh pr merge --match-head-commit, its own record and its own comment. LAND_BATCH=1 behaves as before.

Design

Positions, not one combined commit. A landing commit that merges every PR would put every PR's commits into master with the first merge. GitHub would then mark the other PRs merged with no merge of their own and no --match-head-commit pin. So each PR k gets a position, built the way the merge train built them:

• position k = merge of [position k-1, PR k's tip], then one regen commit;
• position 0 is origin/master.

When PR k merges, master's tree is exactly position k's tree. The tree check still runs after every merge.

Device evidence from each master merge's own tree (AGENTS.md). Each position runs pnpm regen, pnpm typecheck and pnpm evidence:stamp --compare <previous position>:

• If the stamp differs, the device lanes run at that position and are judged against the previous position's evidence, then regen runs again.
• If the stamp is equal, the carried records must judge exactly as the previous position's.

Every position also gets the regen-only check and the floors check against the previous position. Because of this, every master merge commit carries device evidence from its own tree. A batch with no evidence-code change runs no device lanes. A batch with one evidence-changing PR runs the devices once, at that PR's position.

One proof. pnpm test (with the quiet rerun) runs once, on the top position. Positions below the top get no full test of their own, but each has its own regen, typecheck, devices, regen-only and floors checks. Each also passes the checks CI (typecheck plus the platform-free suites) on its own sha before it merges.

Self-check. Before any proof, planPositions checks that the chain is exactly what it claims: each head is one regen commit on a merge of [previous position, member tip], the chain starts on master, and every tip passes tipProblem. A mismatch is fatal.

Publish in order, never ahead. PR k's position is pushed to its branch only after PR k-1 has merged. The driver checks this before the push: the previous position must be in master. So a PR branch never carries another PR's unlanded commits, even when a later step fails. Each publish runs today's steps unchanged:

• push;
• CI on the landing commit;
• pr:review;
• the Claude review gate while UNREVIEWED;
• mergeGate;
• gh pr merge --merge --match-head-commit;
• the tree check;
• moving child PRs to master, then deleting the branch.

Bisect. When the top fails, the driver binary-searches the prefixes. The positions are already built, so each probe is just a test of an existing commit. The first failing position is the culprit, found in at most ceil(log2 n) more proofs:

• The passing prefix lands. Its top passed in the bisect.
• The culprit fails with its own failing run, not the top's. The message adds a note like: "Found by bisecting the batch MIT LICENSE, and ship README and LICENSE in the dragon package #1 Land work through reviewed PRs (Macroscope, PR CI, pr:review) #2 README: Sponsors section #3 README: Sponsors wording and Codex for OSS #4 Parity: shard the determinism check per fixture #5 (4 proofs): master with MIT LICENSE, and ship README and LICENSE in the dragon package #1 Land work through reviewed PRs (Macroscope, PR CI, pr:review) #2 README: Sponsors section #3 passes, adding README: Sponsors wording and Codex for OSS #4 fails."
• The PRs after the culprit go back to the front of the queue and are rebuilt on the new master.

Ejection. Each of these fails one PR with the step name, the label and the comment, as today:

• Before any build, admission checks the PR state, branch, fork and tip, retargets the base, waits for the head's CI, then runs pr:review --wait at the current head. When that head is UNREVIEWED, it also runs the Claude review gate (the precomputed review for the clean head). A review that is not clean ejects the PR (review-before, claude-review-before), and the batch fills from the queue.
• During the build, a merge conflict or any failed position step ejects only that PR. The chain continues on the position before it.
• During publishing, a failure stops the batch at that PR. Every PR after it is requeued, including a bisect culprit, because its verdict assumed that prefix would land.

Every round lands or fails at least one PR, so the loop always ends.

Stacks. A PR whose base is the branch of a PR admitted earlier in the same batch is admitted. The parent's publish moves it to master (retargetChildrenThenDelete). If that did not happen, mergeGate refuses it. If the parent was ejected, the child's build refuses to run unless the parent's head is in the chain (retarget: land its parent first), so the parent's commits never land through the child.

The driver also releases the heavy-lease priority while it waits in admission, as it already does while publishing.

Hooks for CI-OFFLOAD (#135, #144). runBatches (in land-lib.ts) takes its steps as functions: admit, base, build, verify, prove and publish.

• prove(position) is the only full-test call. A CI full-test mode (CI-FULLTEST: the full pnpm test on GitHub runners, sharded (full-test.yml) #144) replaces its body with one dispatched run on the top position's sha, so proving a batch is one CI run, and each bisect probe is one more.
• The device step inside buildPosition is the block Landing driver: LAND_DEVICES=ci (device lanes on GitHub runners, hybrid) #135's LAND_DEVICES=ci replaces. It runs at most once per stamp change rather than once per PR.

Expected time: one build-and-prove (regen per position plus one full test), then about one CI run plus review per PR while publishing, instead of a full build and test per PR. Pipelining (building batch K+1 on batch K's top while K publishes) is the planned second PR.

Outside the spec: AGENTS.md step 7 now describes batched landing.

What passed

• pnpm typecheck passed.
• vitest run packages/parity/test/land.test.ts passed: 66 tests, 16 of them new.
  • Fakes for runBatches:
    • one proof per batch, with positions chained in order;
    • several batches;
    • a bisect that finds the culprit in log2(n) proofs, with its own failure message, lands the PRs before it and requeues the rest;
    • a culprit at position 1, and a batch of one with no bisect;
    • a merge-conflict ejection, with the chain continuing on the previous position;
    • a review ejection before any build, with the batch filled from the queue;
    • a failed publish requeuing everything after it, including the culprit;
    • a fatal error;
    • an unreadable master;
    • a chain that is not what it claims;
    • bisectPrefixes for every n ≤ 8 and every culprit;
    • LAND_BATCH parsing.
  • A scratch git repository: a conflicting member is ejected with no merge left in progress, and the next member chains on the position before it. planPositions accepts the chain, and every regen commit is regen-only and vouchable. An intermediate position holds only its prefix. A reordered or duplicated chain is refused.
• No test was removed or loosened. The existing runQueue tests stay, because runQueue is kept and still exported.
• No full pnpm test: this is tooling only, and the driver proves it.

Review round 1 (precomputed review of a6ad40b: 2 Medium, 3 Low), fixed in 8428aee

Medium: a partial publish could leave master on an untested tree. Fixed.

• When publishing stops after at least one merge, and the landed position is not one a proof passed (the top, or a passing bisect probe), the driver now proves that position's tree, which is master's tree, right away.
• If that proof fails, the driver labels and comments on the landed PR at step master-red, writes the fatal to the status, and stops.
• A resting tree that was already proved is not proved again.

Medium: bisect blamed position 1 when master itself was red. Fixed.

• When the culprit is position 1, the driver proves master first (new op proveMaster).
• If master fails, the driver stops with "master is red". It ejects nobody and writes no bisect note.

Low: non-test errors counted as a failing prefix. Fixed.

• Only a failure at step test is a verdict on a tree (isTestVerdict, proofVerdict).
• Any other error while proving (install, checkout, git) stops the driver as fatal. That covers the top, every bisect probe, master and the resting-tree proof.

Low: ignored outputs of other trees. Fixed.

• proveCommit tests in place only the tree the last build left untouched.
• Any other commit is checked out with git clean -fdX, keeping KEEP_IGNORED (node_modules, vendor/wpt, native build caches and the zig and cargo outputs), and then installed.
• The git clean arguments are pinned on a scratch repository.

Low: the #144 and #135 hooks. Agreed. #144's CI full test must push each proved position to a scratch ref keyed by position (as #135 does with land-devices/pr-<n>), because positions are never on a PR branch before publish. When #135 is merged into buildPosition, its judgeDevices and its scratch commit must use prev and the position tree, not origin/master. #141's solo rerun goes into proveCommit's quiet-rerun block.

AGENTS.md step 7 now says the same.

What passed (8428aee):

• pnpm typecheck passed.
• vitest run packages/parity/test/land.test.ts passed: 70 tests. 4 tests are new this round:
  • "master is red";
  • a red resting tree after a partial publish, a green one, and one that a bisect probe had already proved;
  • a non-test error at a bisect probe and at the top;
  • ignored-output cleanup.
  • Also updated: the culprit-at-position-1 cases now prove master before blaming.

Catch-up with master (#141), at 5d4276d

origin/master is merged in, and #141's solo rerun is now part of proveCommit:

• Each failing file is rerun alone on a quiet machine, and every one must pass.
• The quiet hold stays in place through the rerun (land: hold the quiet request through the quiet test rerun #139).
• The whole suite reruns instead when the run crashed or more than SOLO_RERUN_MAX files failed.

A failed solo rerun is still a test verdict, so bisect and the master proofs treat it as a failing tree.

Low 1 (round 2): the master proof is skipped when master's tree, ignoring docs/goals/**, equals a commit whose full test passed in this run (provedTree, tested).

What passed (5d4276d):

• pnpm typecheck passed.
• vitest run packages/parity/test/land.test.ts passed: 72 tests, including land: rerun each failing test file alone on a quiet machine #141's tests.

🤖 Generated with Claude Code