Skip to content

Re-review joins a verification worker started on the previous head and fails the new head on the old head's CI result #4527

Open

Description

@eltmon

Problem

A re-review requested after a fix push can join a verification worker that started on the previous head. That worker then fails the new head by using the old head's CI result. The work agent is told to fix a failure that no longer exists, and the attempt counts against the verification budget.

Observed on PAN-4498, 2026-10-04 (all times UTC):

  • 03:43:09: verification worker 1090636 starts on head c9994b06.
  • 03:48:29: the CI test job fails on c9994b06 (.overdeck/verification/2026-10-04T03-48-29-663Z-c9994b06.json).
  • 03:52:19 to 03:52:59: the agent commits the fix (119d14af78e) and a follow-up (20690565301), pushes, and runs pan review request.
  • 03:53: the dashboard log shows [request-review] Joining live verification worker 1090636 for PAN-4498.
  • 04:00:52: the CI test job passes on 20690565 (2026-10-04T04-00-52-390Z-20690565.json, "outcome": "passed").
  • 04:09:47: worker 1090636 finishes and writes 2026-10-04T03-43-09-839Z-20690565.json. The record is labelled head8: "20690565", but it fails test with "The CI test job already failed on this head (c9994b06)…". It writes 002-verification-gate-failed.md to the agent.

Root cause

  1. src/lib/cloister/verification-worker-supervisor.ts line 178, in runSupervisedVerificationInternal:
    const sameHead = options.syncTargetBranch !== false || existing.headAnchor === headAnchor;
    Review verification never sets syncTargetBranch: false, so sameHead is always true. A request on a new head joins any live worker, whatever head it started on. The comment above says review verification "may legitimately change HEAD while syncing the target branch". That sync is a merge of the target branch into the worker's own run, but it is used to excuse a different case: the agent pushing new commits.
  2. src/lib/cloister/verification-runner.ts ~lines 578-590: ciTestRed calls readLatestCiTestResult(workspacePath, headShort) with the headShort captured when the worker started. A joined worker therefore applies the old head's CI verdict to the new head.

Fix

  1. In runSupervisedVerificationInternal, join a live worker only when the requested head is the worker's launch head, or descends from it only through the target-branch sync merge the worker itself made. When the agent's branch tip has moved to a commit the worker did not create, kill the worker and start fresh: this is the existing "for a different workspace HEAD" path.
    • Implementation checkpoint: record the worker's launch-time branch tip in the worker state, next to headAnchor. Compare the requested tip with it. If the existing state cannot tell the two cases apart, the fallback is to never join a review-verification worker whose launch tip differs from the requested tip.
  2. In verification-runner.ts, read the CI test result for the head that is actually being verified at verdict time, not the launch-time headShort. A record labelled head8: X must never carry a verdict computed for another head.

Acceptance criteria

  • Given a live review-verification worker launched on head A, when pan review request arrives with the branch tip at a new agent commit B, then the supervisor kills the A worker and starts a fresh one on B. Unit test in the supervisor tests, using fake timers for any wait.
  • Given a live review-verification worker whose only HEAD change is its own target-branch sync merge, when a duplicate request arrives, then it still joins (no regression).
  • Given CI recorded failed for head A and passed for head B, when verification of B reaches the ci-mode test gate, then test passes (runner unit test).
  • docs/PIPELINE-GATES.md (verification gate section) states when a re-review joins a live worker and when it restarts it.

Routing

This is review-pipeline machinery (TENET-10). Route it through a supervised pan handoff, not an unattended pan start.

Activity

  1. self-assigned this
    on Oct 4, 2026
  2. added
    bugSomething isn't working
    substrate-improvementNon-blocker Panopticon infrastructure improvement — file, don't patch inline
    needs-handoffPipeline-runtime refactor: supervised pan handoff only, never autonomous pickup (TENET-10)
    on Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingneeds-handoffPipeline-runtime refactor: supervised pan handoff only, never autonomous pickup (TENET-10)substrate-improvementNon-blocker Panopticon infrastructure improvement — file, don't patch inline

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions