Skip to content

pan close refuses landed, deployed issues: deploy probe times out on a stalled event loop; stale 'running' verification artifact blocks row 3 #4543

Description

@eltmon

Problem

pan close refuses landed, deployed issues for two reasons that are not real Definition-of-Done failures. Observed 2026-10-04 09:10-09:16 UTC, closing out 18 issues that were merged and deployed in build cd27acfd:

  1. Deploy row (8) times out on a stalled event loop. 6 of 22 close-out attempts failed with dod:deploy — dashboard not reachable at http://127.0.0.1:3011 … The operation was aborted due to timeout: PAN-4509, PAN-4506, PAN-4383, PAN-4514 and PAN-4541 on the first pass. Each succeeded on retry. /api/health normally answers in under 1 ms, but a 0.5 s sampler during close-outs recorded 4.66 s at 09:15:17 and 4.05 s at 09:15:51, each during a running pan close. Failures followed a successful close-out of the previous issue.
  2. Verification row (3) blocks on a stale running artifact. PAN-4498 and PAN-4508 fail with dod:verification — verification artifact: running at 2026-10-04T05:44:24Z (and 05:45:37Z), although both PRs passed verification (passed at heads 3fe790ef and f91b6577) and merged with green main CI. Their workspaces/feature-pan-<n>/.overdeck/verification-latest.json is {"status":"running", "head8": null}. It was left by a duplicate verification run (started by a deacon-lite re-request at 05:06) that ended {"outcome":"skipped","reason":"The pull request already merged; pre-merge verification no longer applies."} without updating the latest artifact.

Root cause

  1. src/lib/lifecycle/dod-gate.ts line 278 probes once with a 3 s budget:
    const response = await fetch(url, { signal: AbortSignal.timeout(3000) });
    No retry. Separately, something in the dashboard server's close-out handling blocks its event loop for about 4-5 s.
  2. checkVerificationRow (dod-gate.ts ~lines 445-468) treats a missing artifact on landed work with green main CI as a pass (!artifact && settlement?.landedWork && settlement.mainVerifyStatus === 'pass'), but a non-terminal running artifact as a miss. The merged-skip path in src/lib/cloister/verification-runner.ts (skipMergedVerification) returns skipped without rewriting the latest artifact to a terminal state.

Fix

  1. Deploy probe: retry /api/health up to 3 times with a 5 s timeout per attempt and a 2 s pause between attempts, using fake timers in tests. Report the miss only when all attempts fail.
  2. Event-loop stall (implementation checkpoint): capture a CPU profile (node --cpu-prof on the dashboard, or --inspect plus a sampled profile) while running pan close on an issue. Name the blocking call with file:line in the PR. If it is a sync child process or sync file scan on the request path, make it async (repo rule: no execSync in server-reachable code). If no single cause is found within the issue's scope, document the profile findings in the PR and file a follow-up. Do not leave the stall unmentioned.
  3. Verification row: treat a non-terminal artifact (running, or any status that is not passed/failed) the same as a missing one when settlement.landedWork and settlement.mainVerifyStatus === 'pass' hold, and say so in the row's observed text. Also make the merged-skip path in verification-runner.ts write a terminal skipped latest artifact, so no new stale running files appear.

Acceptance criteria

  • Given readJson fails twice with a timeout and then returns a health body, checkDeployRow passes (unit test, fake timers).
  • Given a running verification artifact, landedWork: true and mainVerifyStatus: 'pass', checkVerificationRow returns pass with an observed note naming the stale artifact (unit test).
  • Given a running artifact and mainVerifyStatus not pass, the row is still a miss (unit test).
  • Given the runner's merged-skip path, the latest verification artifact ends with a terminal skipped status (runner unit test).
  • The PR names the event-loop stall's cause with file:line and fixes it, or documents the profile and links a follow-up issue.
  • docs/MERGE-WORKFLOW.md (close-out section) describes the deploy-row retry and the stale-artifact rule.
  • After merge and deploy, pan close PAN-4498 and pan close PAN-4508 complete with no --accept-* override.

Activity

  1. self-assigned this
    on Oct 4, 2026
  2. added
    bugSomething isn't working
    substrate-improvementNon-blocker Panopticon infrastructure improvement — file, don't patch inline
    planningIssue is in planning/discovery phase
    in-progressWork is actively being done
    and removed
    planningIssue is in planning/discovery phase
    in-progressWork is actively being done
    on Oct 4, 2026
  3. eltmon commented on Oct 4, 2026

    @eltmon
    OwnerAuthor

    🤖 Agent completed work:

    Deviations: (1) WI-7 found the stall outside the PRD's suspects. GET /api/costs/summary ran 3 sync full scans of the 386 MB cost log per request (~4.8 s blocked); God View polls it every 30 s. A read-only live sampler reproduced it: 4.67 s and 4.45 s, 34.6 s apart, with no close-out involved. Fixed with an async chunked reader, readEventsSince (max loop block 10 ms measured), in src/lib/costs/events.ts, costs/index.ts and routes/costs.ts. None of those files were in files_scope. Membership refresh and the closed-issue reaper were ruled out by measurement. D7's kill -USR1 and the membership refresh route were not used; profiling ran out of process. /api/costs/stream still blocks ~1.6 s per poll -> follow-up PAN-4544. (2) WI-5 shipped a TS2367 that per-file tsc missed; full npm run typecheck caught it and a trailing commit with the same Item trailer fixed it. (3) The WI-4 panel render test was added after the item closed, in a separate commit. (4) pan task left spec lifecycle fields dirty; committed as chore(workspace). AC sub-items were closed through one evidence trailer commit. Delivered: skipped verification outcome + finalize helper; the post-gate merged skip rewrites its own running artifact; lint node and gates panel render skipped; DoD row 3 passes a stale non-terminal artifact on landed green work; row 8 retries /api/health 3x (5 s timeout, 2 s apart); docs and pan-close skill row numbering fixed. Full typecheck and lint pass; all touched tests pass.

  4. added
    mergedMerged to main
    verifying-on-mainMerged — awaiting verification on main
    and removed on Oct 4, 2026
  5. eltmon commented on Oct 4, 2026

    @eltmon
    OwnerAuthor

    Closed via close-out ceremony

    Definition of Done
    # Row Status Observed
    1 Review passed ✓ pass PR #4545 reviewDecision: APPROVED
    2 Tests passed ✓ pass PR #4545 checks: all 19 check(s) green
    3 Verification green on the branch ✓ pass verification artifact: passed at 2026-10-04T10:13:07.530Z
    4 Merged to main ✓ pass Pull request #4545 merged at 2026-10-04T10:33:11Z; PR #4545 MERGED at 2026-10-04T10:33:11Z
    5 Post-merge lifecycle ran ✓ pass canonical state: verifying_on_main; no running work/planning agents (stored running, not live: agent-pan-4543, planning-pan-4543)
    6 Verified on main ✓ pass required checks concluded successfully on b149038: test, lint, build (22), reject-planning-paths
    7 Version strings propagated – skip project declares no version_sync; ship step not applicable
    8 Deployed ✓ pass build commit b149038 contains merge b149038
    9 Close-out teardown verified ✓ pass workspace removed; feature branch kept (close_out.delete_feature_branch is off); Closed 1 Herdr pane(s); No shadow state found; No legacy planning directory found; Stopped Docker containers; No orphaned processes found; No worker worktrees; Removed git worktree; Pruned 3 agent directories (transcripts and state kept); No strike branch strike/pan-4543 found; Pruned 91 checkpoint ref(s) for agent-pan-4543, planning-pan-4543, strike-pan-4543; No specialist registry entries for PAN-4543

    Gate: passed

  6. added
    closed-outVerified and closed out
    and removed
    mergedMerged to main
    verifying-on-mainMerged — awaiting verification on main
    on Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingclosed-outVerified and closed outsubstrate-improvementNon-blocker Panopticon infrastructure improvement — file, don't patch inline

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions