Repository navigation
Re-review joins a verification worker started on the previous head and fails the new head on the old head's CI result #4527
Copy link
Copy link
Open
Labels
bugSomething isn't workingSomething isn't workingneeds-handoffPipeline-runtime refactor: supervised pan handoff only, never autonomous pickup (TENET-10)Pipeline-runtime refactor: supervised pan handoff only, never autonomous pickup (TENET-10)substrate-improvementNon-blocker Panopticon infrastructure improvement — file, don't patch inlineNon-blocker Panopticon infrastructure improvement — file, don't patch inline
Description
Problem
A re-review requested after a fix push can join a verification worker that started on the previous head. That worker then fails the new head by using the old head's CI result. The work agent is told to fix a failure that no longer exists, and the attempt counts against the verification budget.
Observed on PAN-4498, 2026-10-04 (all times UTC):
- 03:43:09: verification worker 1090636 starts on head
c9994b06. - 03:48:29: the CI test job fails on
c9994b06(.overdeck/verification/2026-10-04T03-48-29-663Z-c9994b06.json). - 03:52:19 to 03:52:59: the agent commits the fix (
119d14af78e) and a follow-up (20690565301), pushes, and runspan review request. - 03:53: the dashboard log shows
[request-review] Joining live verification worker 1090636 for PAN-4498. - 04:00:52: the CI test job passes on
20690565(2026-10-04T04-00-52-390Z-20690565.json,"outcome": "passed"). - 04:09:47: worker 1090636 finishes and writes
2026-10-04T03-43-09-839Z-20690565.json. The record is labelledhead8: "20690565", but it failstestwith"The CI test job already failed on this head (c9994b06)…". It writes002-verification-gate-failed.mdto the agent.
Root cause
src/lib/cloister/verification-worker-supervisor.tsline 178, inrunSupervisedVerificationInternal:Review verification never setsconst sameHead = options.syncTargetBranch !== false || existing.headAnchor === headAnchor;
syncTargetBranch: false, sosameHeadis alwaystrue. A request on a new head joins any live worker, whatever head it started on. The comment above says review verification "may legitimately change HEAD while syncing the target branch". That sync is a merge of the target branch into the worker's own run, but it is used to excuse a different case: the agent pushing new commits.src/lib/cloister/verification-runner.ts~lines 578-590:ciTestRedcallsreadLatestCiTestResult(workspacePath, headShort)with theheadShortcaptured when the worker started. A joined worker therefore applies the old head's CI verdict to the new head.
Fix
- In
runSupervisedVerificationInternal, join a live worker only when the requested head is the worker's launch head, or descends from it only through the target-branch sync merge the worker itself made. When the agent's branch tip has moved to a commit the worker did not create, kill the worker and start fresh: this is the existing "for a different workspace HEAD" path.- Implementation checkpoint: record the worker's launch-time branch tip in the worker state, next to
headAnchor. Compare the requested tip with it. If the existing state cannot tell the two cases apart, the fallback is to never join a review-verification worker whose launch tip differs from the requested tip.
- Implementation checkpoint: record the worker's launch-time branch tip in the worker state, next to
- In
verification-runner.ts, read the CI test result for the head that is actually being verified at verdict time, not the launch-timeheadShort. A record labelledhead8: Xmust never carry a verdict computed for another head.
Acceptance criteria
- Given a live review-verification worker launched on head A, when
pan review requestarrives with the branch tip at a new agent commit B, then the supervisor kills the A worker and starts a fresh one on B. Unit test in the supervisor tests, using fake timers for any wait. - Given a live review-verification worker whose only HEAD change is its own target-branch sync merge, when a duplicate request arrives, then it still joins (no regression).
- Given CI recorded
failedfor head A andpassedfor head B, when verification of B reaches theci-mode test gate, thentestpasses (runner unit test). -
docs/PIPELINE-GATES.md(verification gate section) states when a re-review joins a live worker and when it restarts it.
Routing
This is review-pipeline machinery (TENET-10). Route it through a supervised pan handoff, not an unattended pan start.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workingneeds-handoffPipeline-runtime refactor: supervised pan handoff only, never autonomous pickup (TENET-10)Pipeline-runtime refactor: supervised pan handoff only, never autonomous pickup (TENET-10)substrate-improvementNon-blocker Panopticon infrastructure improvement — file, don't patch inlineNon-blocker Panopticon infrastructure improvement — file, don't patch inline
Activity