[BUG] V2 waiting run keeps showing Agent is working and offers Stop for finished work when checkpoint capture stalls #15124

Activity

1. NK-Works commented on Oct 3, 2026

   

   NK-Works

   on Oct 3, 2026

   Author

   More actions

   Evidence addendum (logic repro, no provider CLI needed):

   phaseFor(waiting) returns running per packages/shared/src/agentAwareness.ts:99-105, and derivePendingBackgroundWork treats waiting as settled per packages/shared/src/orchestrationV2PendingBackgroundWork.ts:37-43,109-116. Successful turns are persisted as waiting at apps/server/src/orchestration-v2/RunExecutionService.ts:633-634 until CheckpointCaptureService.ts:90-104 flips them. Repro script output:

   repro waiting: waiting run -> running => UI shows 'Agent is working' even though provider turn succeeded, checkpoint pending
   

   Visual diagram prepared locally at /tmp/opencode/shot.svg and /tmp/opencode/v2-waiting-evidence.html for the reporter to screenshot and drag into this issue (no desktop browser connected in this session, so no PNG captured here).
2. NK-Works

   changed the title [-]V2 waiting run keeps showing Agent is working and offers Stop for finished work when checkpoint capture stalls[/-] [+][BUG] V2 waiting run keeps showing Agent is working and offers Stop for finished work when checkpoint capture stalls[/+] on Oct 3, 2026
3. juliusmarminge commented on Oct 3, 2026

   

   juliusmarminge

   on Oct 3, 2026

   Member

   More actions

   Note

   Grok responding on behalf of Julius.

   Triage

   Thanks for the careful source walk-through, @NK-Works! What the logic repro shows is the designed post-success state, not a misclassification.

   How waiting is supposed to behave

   A successful provider turn is saved as waiting until checkpoint capture commits completed (persistedStatus in RunExecutionService). While a run is in that state:

   • Awareness and the composer treat waiting as still working, and Stop stays available. projectThreadAwarenessV2 maps waiting to the running phase ("Agent is working"), and derivePhase and deriveCanInterruptRunningThread do the same. apps/web/src/session-logic.test.ts checks this.
   • waiting is in SETTLED_FOR_BACKGROUND_WAIT_RUN_STATUSES on purpose, so background tasks that are still running can show up before the checkpoint lands. derivePendingBackgroundWork only includes items that are still active (pending, running, or waiting), not finished work. The test "returns pending work when the latest run is waiting (post-success, pre-checkpoint)" covers this.
   • If that list includes work that will wake the agent, like a subagent or a monitor, the thread is shown as idle rather than stuck on the checkpoint waiting state. The background stop strip appears, and the thread doesn't read as "Agent is working" (presentThreadShell, see the test "parks runtime idle over stale checkpoint waiting when the roster is nonempty").

   So the tests you ran pass because they lock this behavior in. A short "working" window while capture is in flight is expected. active_run_id also excludes waiting, so this isn't an interruptible provider turn.

   What would be a real bug

   This report doesn't show a run that stays stuck after capture gives up. checkpoint.capture retries, and after the worker's maximum attempts (5 by default) the outbox row fails without moving the run off waiting.

   Because waiting still blocks the thread, it would keep looking busy, and queued turns wouldn't start. Stop would target that run, but dispatchRunInterrupt rejects it as "not interruptible" when there's no running provider turn and no active background work. That failure to finish the run is a separate problem from showing the short in-flight state as "working".

   If you've seen a thread stay on waiting after capture stopped retrying, please comment with:

   • The run status
   • The checkpoint.capture effect row (status, attempt count, and error)
   • Roughly how long it stayed that way

   A maintainer will decide on the fix direction once that's clear.
4. juliusmarminge

   added

   needs more infoInitial triage showed no bug. Awaiting more infoInitial triage showed no bug. Awaiting more info

   via-triageFiled through npx t3 triageFiled through npx t3 triage

   on Oct 3, 2026
5. NK-Works commented on Oct 3, 2026

   

   NK-Works

   on Oct 3, 2026

   Author

   More actions

   Thanks for the detailed triage — verified your refs against main@fed41fa88 and you're right that the short in-flight window is by design. Withdrawing the broad "waiting should not show working" claim:

   • apps/web/src/session-logic.ts:1013 maps waiting to running, locked by session-logic.test.ts:103-106.
   • packages/client-runtime/src/state/models.ts:168-171 parks runtime at idle when the background roster is nonempty, locked by entities.test.ts:302 ("parks runtime idle over stale checkpoint waiting"). So with background work present the thread does not read as "Agent is working" — my report overstated that case.
   • EffectWorker.ts:508 defaults maxAttempts to 5 for the checkpoint.capture outbox row.

   Narrowing to the stuck case you describe, which I have not reproduced live in this session: a run remaining in waiting after checkpoint.capture exhausts retries, where the outbox row fails without moving the run off waiting, the thread keeps looking busy, queued turns do not start, and Stop targets the run but dispatchRunInterrupt rejects it as not interruptible (Orchestrator.ts:7779 path).

   I don't have a stuck run here to pull the three items from (run status, checkpoint.capture effect row status/attempts/error, duration). If it helps, I can gather them from a reporter machine with read-only queries against statev2.sqlite (runs + effect outbox rows for checkpoint.capture:<runId>) plus the surrounding server.trace.ndjson — or, if you prefer, I can re-title this issue to that stuck-after-retries shape and drop the in-flight presentation part. Let me know which evidence format you want and I'll supply it.
6. NK-Works commented on Oct 3, 2026

   

   NK-Works

   on Oct 3, 2026

   Author

   More actions

   Evidence gathered while you review (code-construction + local snapshot, no live stuck run here):

   1. Stuck mechanism exists by construction (verified on main@fed41fa88)

   • Failure path emits no run transition: CheckpointCaptureService.ts:90-104 returns CheckpointCaptureExecutionError when the capture target is incomplete (missing run/rootNode/scope, scope mismatch, no provider thread). Only the success path (:136-174) commits run.updated (waiting -> completed) plus checkpoint.captured. The existing test exercises this: CheckpointCaptureService.test.ts:316-321 executes with a missing scope and asserts the error is returned before attempting the real capture.
   • Retry exhaustion also leaves the run untouched: EffectWorker.ts:688-691 calls outbox.fail once attemptCount >= maxAttempts (default 5, :508) without any run-status write. So a persistently failing checkpoint.capture:<runId> effect ends as failed in the outbox while the run row stays waiting — the exact stuck shape you describe (busy-looking thread, queued turns blocked, Stop targeting a non-interruptible run per Orchestrator.ts:7779).

   2. Local snapshot has no stuck rows (queries included for reporters)

   Read-only copy of this machine's statev2.sqlite (copied to /tmp, never opened read-write):

   runs by status: cancelled 3, completed 35, failed 5, running 2, waiting 0
   checkpoint.capture effects: 35 succeeded (max attempt_count 1), 0 failed
   

   Queries a reporter can rerun against their statev2.sqlite (read-only open recommended):

   SELECT status, COUNT(*) FROM orchestration_v2_projection_runs GROUP BY status;
   SELECT run_id, thread_id, ordinal, status
   FROM orchestration_v2_projection_runs WHERE status = 'waiting';
   SELECT status, COUNT(*), MAX(attempt_count)
   FROM orchestration_v2_effect_outbox
   WHERE effect_type = 'checkpoint.capture' GROUP BY status;
   SELECT effect_id, thread_id, status, attempt_count, substr(last_error,1,300)
   FROM orchestration_v2_effect_outbox
   WHERE effect_type = 'checkpoint.capture' AND status = 'failed';

   Plus the surrounding server.trace.ndjson window for the stuck runId (the checkpoint.capture attempts and the final fail).

   3. What I still lack

   A live thread actually stuck in waiting with its failed checkpoint.capture row (effect status, attempt count, last_error) and how long it stayed that way. If you can point me at a repro that forces capture to fail persistently (e.g. a scope-mismatch fixture or a failing CheckpointStore.capture), I'll drive it here and attach the run + outbox rows. Otherwise, happy for this issue to be re-titled to the stuck-after-retries shape.
7. NK-Works commented on Oct 3, 2026

   

   NK-Works

   on Oct 3, 2026

   Author

   More actions

   Deterministic repro now in hand (temporary test, since removed — not committed):

   Temp test: failed capture leaves the run in waiting with zero events committed

   Setup mirrors CheckpointCaptureService.test.ts: thread + run in waiting applied to an in-memory ProjectionStore, then CheckpointCaptureService.execute against a missing scope (incomplete target, same class of failure as a persistently failing capture):

   • execute returns CheckpointCaptureExecutionError (Failure exit).
   • The mocked EventSink.commitCommand is never called (0 commits — no run.updated, no checkpoint.captured).
   • getThreadProjection still shows the run as waiting.

   Result under Node 24 (repo tests need node:sqlite):

   RUN  v5.0.1
   Test Files  1 passed (1)
   Tests  1 passed (1)
   

   Combined with EffectWorker.ts:688-691 (outbox.fail after maxAttempts, default 5 at :508, with no run write), a checkpoint.capture effect that keeps failing ends as failed in orchestration_v2_effect_outbox while the run row stays waiting indefinitely — thread looks busy, queued turns stay blocked, Stop targets a run dispatchRunInterrupt rejects (Orchestrator.ts:7779).

   Live snapshot (this machine, read-only copy): no stuck rows currently — runs: 3 cancelled / 35 completed / 5 failed / 2 running / 0 waiting; checkpoint.capture effects: 35 succeeded, 0 failed. Reporter queries from my previous comment still apply for any machine that does hit it (run status + effect row status/attempts/last_error + duration).