Skip to content

Dashboard restart during verification loses the review dispatch; re-request discards the finished result and re-runs all gates #4532

Description

@eltmon

Problem

A dashboard restart while a verification worker is running loses that verification's review dispatch. The worker survives the restart and writes a passed result, but the code that waits for the result and then dispatches the reviewer lived in the old dashboard process. When the deacon later re-requests review, the supervisor ignores the finished result and starts a fresh full verification. That costs another ~25-45 minutes of serialized gate time (the single CPU admission slot, PAN-4311) for a head that already passed.

Observed on PAN-4515, 2026-10-04 (times UTC):

  • 03:52:21: verification starts; worker 1870433 (~/.overdeck/verification-workers/pan-4515/worker-1791085939727-1693887.log).
  • 04:21: the dashboard restarts (pan reload + pan restart approve).
  • 04:36:43: the worker writes result-1791085939727-1693887.json = {"outcome":"passed"}, and the journal records verification.passed. No reviewer is dispatched.
  • 04:41:52: deacon-lite re-requests: Agent re-review request (1/25): a dashboard restart during verification left the review undispatched.
  • 04:41:54: a new worker (worker-1791088912886-10483.log) starts a complete re-run.

PAN-4257, PAN-4498 and PAN-4508 had workers running across the same restart and are exposed in the same way.

Root cause

src/lib/cloister/verification-worker-supervisor.ts, runSupervisedVerificationInternal, line 174:

if (existing && existing.workspacePath === workspacePath && isProcessAlive(existing.pid) && !existsSync(existing.resultPath)) {

A worker is joined only while it is alive and has no result yet. A worker that has already finished, and whose result nobody consumed because its waiter died with the old dashboard process, falls through to a fresh worker. Nothing on dashboard boot re-attaches waiters to workers that survived the restart (activeSupervisorRuns is in-memory only).

Fix

  1. Consume an unconsumed result. In runSupervisedVerificationInternal, when the existing worker state has a result file, the result is unconsumed, and the worker's verified head equals the current workspace HEAD (or the HEAD differs only by the worker's own target-branch sync merge, as PAN-4527 defines it), return that result without starting a new worker. Record consumption in the worker state so one result is never consumed twice.
  2. Re-attach on boot. On dashboard boot, scan ~/.overdeck/verification-workers/*/state.json. For each live worker, or each finished worker with an unconsumed result, resume the same post-verification continuation that request-review runs (push the verified branch, dispatch review), as if the original request were still waiting.
  3. Coordinate with PAN-4527 (head matching) and PAN-4531 (short-circuit on red CI). The head-equality rule must be the same in all three.

Acceptance criteria

  • Given a worker state with a finished passed result for the current HEAD and no consumer, when review is re-requested, then no new worker starts and review dispatches (supervisor unit test).
  • Given the same, but the agent has pushed a new commit since, then a fresh worker starts (no regression with PAN-4527).
  • Given a live worker at dashboard boot, when it later writes passed, then the review dispatches without any re-request (boot-adoption test; use fake timers for polling).
  • A result is consumed at most once (unit test).
  • docs/PIPELINE-GATES.md states that verification survives a dashboard restart and that results are adopted, not re-run.

Routing

This is review/verification machinery (TENET-10): supervised pan handoff, not pan start. Bundle it with PAN-4527 and PAN-4531.

Activity

  1. self-assigned this
    on Oct 4, 2026
  2. added
    bugSomething isn't working
    substrate-improvementNon-blocker Panopticon infrastructure improvement — file, don't patch inline
    needs-handoffPipeline-runtime refactor: supervised pan handoff only, never autonomous pickup (TENET-10)
    on Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingneeds-handoffPipeline-runtime refactor: supervised pan handoff only, never autonomous pickup (TENET-10)performancesubstrate-improvementNon-blocker Panopticon infrastructure improvement — file, don't patch inline

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions