flaky: live Git Bash background job resolves with no handle on windows-latest

The flake

test/background-jobs.test.ts > session-local background job state > launches and disposes a real Git Bash background job fails intermittently on windows-latest:

AssertionError: expected undefined to be defined
  at background-jobs.test.ts:603

Observed on PR #503, Pi 0.80.10 / windows-latest. A re-run of the identical commit passed, so it is intermittent rather than a regression. The PR that surfaced it changed only Markdown, CHANGELOGs and version strings — no executable code — so nothing in the diff can reach a test that spawns a real Git Bash process.

Why this is NOT the same bug as #495

Worth separating, because the obvious reading is wrong and would lead to the wrong fix.

#495 was a deadline problem: a 5s vitest timeout could not cover a cold CPython launch, and the symptom was Test timed out in 5000ms.

This one is not a timeout at all. The assertion that fails is:

const job = await awaitLiveCondition(
  runtime.launch({ ... }),
  "the live Git Bash launch",
  () => "no job handle - the contained launch never returned",
);
expect(job).toBeDefined();          // <- line 603, this is what fails

awaitLiveCondition rejects with a descriptive did not settle within Nms error when it times out. That error was not raised. So the helper resolved — with undefined — comfortably inside its 60s WINDOWS_LIVE_TEST_TIMEOUT_MS budget.

In other words: runtime.launch(...) returned no job handle, promptly and without erroring. Raising the timeout would change nothing.

What that points at

A launch that resolves to undefined rather than throwing is a spawn that did not produce a tracked job. Candidates worth checking, in rough order:

shellPath resolution. The test resolves a real Git Bash. If that lookup returns a path that is momentarily unusable on the runner image, a launch could fail soft.
createBackgroundJobRuntime(...) returning a runtime whose launch can resolve empty — note the ! on its construction; if the failure is upstream of launch, the handle is simply never registered.
A race between process start and job-table registration — the handle is read before the tracker has recorded it, so the read is legitimately empty rather than late.

(3) would be the most concerning of the three, because it means the runtime can report success while owning nothing — and this is the module that tracks background jobs for cleanup, so an untracked job is a leaked process.

Why it matters beyond a re-run

[GATE ] | [REPO] | Merge readiness aggregates this check, and the release preflight requires that gate green for the exact commit being tagged (#385). A flake here blocks the release lane for that SHA, and it trains readers to re-run a red gate without reading it — the habit the gate exists to prevent. It cost one re-run cycle on #503.

Acceptance criteria

AC-1: the root cause is identified — specifically, whether launch can resolve undefined on a successful spawn (a tracking race) or only on a failed one.
AC-2: if a launch cannot produce a handle, it fails loudly rather than resolving empty; expect(job).toBeDefined() should never be the thing that discovers it.
AC-3: if it is a registration race, the job table is proven to contain the job before launch resolves — an untracked background job is a leaked process, which is what this module exists to prevent.
AC-4: the test passes across repeated windows-latest runs without a re-run.

Relevant sources

plugins/ca-pi/tools/test/background-jobs.test.ts:588-606 (runLiveGitBashJob, awaitLiveCondition)
the createBackgroundJobRuntime / launch implementation under plugins/ca-pi/tools/src/
Related but distinct: flaky: pi security-evidence test times out on windows-latest (5s deadline covers a cold Python launch) #495 (a genuine deadline, already fixed)

AC-1 answered: it is a designed refusal, not a tracking race

I filed this with three candidate causes and flagged (3) — a registration race, where the runtime reports success while owning nothing — as the concerning one, since an untracked background job is a leaked process.

That branch is ruled out. Reading plugins/ca-pi/tools/src/background-jobs.ts:

async launch(input: BackgroundJobLaunchInput): Promise<Readonly<BackgroundJobSnapshot> | undefined>

undefined is a declared outcome, not an accident. There are five refusal paths, and every one of them transitions the job to a terminal state before returning:

#	condition	disposition
1	runtime disposed / stopping / unhealthy	no job created
2	unparseable input, or a stale authorization lease	no job created
3	`manager.createJob` refused	no job created
4	authorization went stale after publish	job → `cancelled`, published
5	`#openTree` threw	job → `failed`, published, ownership released

Path 5 is the one that fired on CI: the Git Bash spawn threw, launch refused correctly, released ownership, and marked the job failed. The runtime is fail-closed and behaving as designed — it never owns an untracked process. AC-3 as originally written does not apply.

What is actually wrong

The refusal is correct; the diagnosis is impossible.

} catch {
  releaseOwnership();
  this.#pendingOwnership.delete(ownership);
  const terminal = this.#manager.transitionJob({ id: job.id, state: "failed" });
  if (terminal !== undefined) this.#publish(terminal, "completed");
  return undefined;
}

The spawn error is discarded — not bound, not logged, not carried on the terminal snapshot. So a real environment failure (a Git Bash path that was momentarily unusable, a transient EBUSY, a permissions blip) is indistinguishable from every other refusal, and reaches the operator as:

AssertionError: expected undefined to be defined

The test's own diagnostic compounds it — "no job handle - the contained launch never returned" — which is false. The launch did return; it returned a refusal. So the one message a reader gets actively misdescribes what happened.

There are 13 bare catch { blocks in this module. Several are legitimately "producer behaviour is authoritative", but this one sits on the path where a real spawn failure is the most likely cause and the least visible.

Revised acceptance criteria

~~AC-1~~ answered: designed refusal on spawn failure (path 5); no tracking race.
~~AC-3~~ withdrawn: the job table is never left holding an untracked process — every refusal transitions to a terminal state first.
AC-2 (revised, and now the substance of this issue): a launch that fails to spawn must carry why. The activity event shape is {kind, id, label, state} with no detail field, so this needs a deliberate call about the contract — extend the event, add a diagnostics channel, or reject rather than resolve on path 5 specifically. Path 5 is distinguishable from paths 1-4: those are policy refusals the caller can predict, this one is an environment failure the caller cannot.
AC-4 (unchanged): the test passes across repeated windows-latest runs. Note that fixing the diagnostics does not by itself fix the flake — it makes the next occurrence explain itself instead of asserting undefined.

Not fixed here

Surfacing the reason means changing a published contract (the activity event) or the refusal semantics of one path. Both deserve a decision rather than being slipped in behind a flake fix, so this is diagnosis only.

test(pi): a failed live Git Bash launch now says why (#504) (#505)

AC-2 addressed in #508; AC-4 remains open and this issue stays open on it.

~~AC-1~~ answered previously: designed refusal on spawn failure, no tracking race.
AC-2 done. The spawn path now reports a per-launch diagnostic drawn from a closed vocabulary — a recognized Windows containment reason via Windows ca-pi adapter suites flake on hard 5s timing budgets #428's windowsRefusalReasonFromMessage(), or an errno shape — and the background bash tool surfaces it where it already asked health().diagnostic and got nothing. Per-launch rather than runtime-scoped because MAX_ACTIVE_JOBS is 4 and a shared slot would report one launch's environment failure as another's.
~~AC-3~~ withdrawn previously.
AC-4 open. Diagnostics do not stop the flake. The next windows-latest occurrence will name its cause instead of asserting on undefined; until a real run does that, or the flake stops recurring, AC-4 is unproven.

Rejected along the way: extending the activity event (display-only by contract, control-stripped, 128-code-point cap), and rejecting on path 5 (splits the return contract for the sole caller and ~20 assertions). A bounded raw error.message was the richest option but the channel is open by construction — realpathSync throws already embed the shell path and cwd.

fix(pi): a failed background-job spawn now says why (#508)

fix(ca): the coverage gate could not run, in any tree (#507) (#513)

AC-4 evidence — the flake has not recurred since the diagnostic landed

AC-1 was answered (a designed refusal on spawn failure, not a tracking race), AC-2 shipped in #508, AC-3 was withdrawn. This issue stayed open on AC-4 alone: "the test passes across repeated windows-latest runs", with the note that the next occurrence would name its cause rather than assert on undefined.

Measured across every CI run since #508 merged (2026-07-26) — 59 runs, 30 of which reached the windows-latest ca-pi Adapter contract job:

outcome	count
success	29
failure	1
cancelled	8 (superseded pushes)

The one failure is not this bug. In run 30365732056 the live test passed —

✓ launches and disposes a real Git Bash background job  4567ms

— and the job failed later, in test_pi_process_tree.py, with AssertionError: controller pid protocol timed out. That is a separate, previously unreported flake; filed separately so this investigation does not inherit its lead, on the same reasoning that split #504 from #495 and #515.

The test is genuinely executing, not silently skipping. Spot-checked run 30366639611: ✓ launches and disposes a real Git Bash background job 11001ms — a real Git Bash launch. Worth recording that the observed durations range from ~4.5s to ~11s against a 60s ceiling, which is consistent with the timing sensitivity the original diagnosis suspected, and is the reason the ceiling is not the thing to tune.

Closing

AC-4's own wording admitted two ways to satisfy it: a real occurrence that explains itself, or the flake ceasing to recur. 30 consecutive clean runs of this specific test, spanning two days and every merge in that window, is the second.

Stated honestly: 30 runs is evidence, not proof, for a flake whose base rate was never established — it was seen once, on #503. If it returns, #508's per-launch diagnostic now names the cause from a closed vocabulary instead of surfacing expected undefined to be defined, so the next occurrence is a diagnosis rather than a re-run. That safety net is the durable outcome here.

Closing on AC-4. Reopen on the next occurrence, with the diagnostic attached.

docs(ci): explain the known-red windows coverage cell in place (#521, #…

fix(core): a coverage figure is the union across a tree's supported h…

Reopening: the next occurrence, with the diagnostic attached — as promised

I closed this on AC-4 evidence (29/30 clean windows runs) with the explicit note "Reopen on the next occurrence, with the diagnostic attached." This is that occurrence, on PR #553, run 30540398282, windows-latest · Pi 0.80.10.

#508's per-launch diagnostic did exactly what it was built for. The failure no longer reads expected undefined to be defined with nothing else. It names its own cause:

AssertionError: launch refused because the spawn threw:
  Windows Job Object holder refused containment (stalled at ATTACHED after 15002ms): ready-timeout

What that tells us, and it changes the diagnosis

This is not the tracking race that was ruled out under the old AC-1, and it is not an unusable shellPath. It is #428's Windows Job Object containment holder timing out while arming — stalled in state ATTACHED for 15002 ms against what is evidently a 15 s ready deadline, then reporting ready-timeout.

So launch refused deliberately, on a real environment failure, and the refusal is correct behaviour. The open question is now much narrower than when this issue was filed:

Why does the holder stall at ATTACHED? It reached attachment and then failed to signal ready. That is a specific, addressable state, not a mystery undefined.
Is 15 s a real deadline or a guess? A hosted Windows runner is materially slower at process creation than a developer box — the same lesson the #387 bounded-diff test exceeds its 5s budget on windows-latest, then EBUSY on cleanup #542 took three passes to learn. If the holder legitimately needs longer under load, the deadline is the defect.
Same-OS control: the windows-latest · Pi 0.80.5 cell passed in the same run (7m9s), as did all four macOS/ubuntu cells. So it is load- or timing-sensitive, not version-specific.

Revised acceptance criteria

~~AC-1~~ / ~~AC-2~~ / ~~AC-3~~ unchanged from before (answered, shipped in fix(pi): a failed background-job spawn now says why (#504) #508, withdrawn).
AC-4 (revised): the containment holder's arm-to-ready path is either made deterministic, or its deadline is set from a measurement on the slowest supported host rather than a guess — explicitly not "raise it until it stops failing". the #387 bounded-diff test exceeds its 5s budget on windows-latest, then EBUSY on cleanup #542's precedent applies: measure the population, and if the budget is genuinely too tight say so with the number.
AC-5 (new): the stall state and elapsed time are already reported; whatever the fix is, that diagnostic must survive it. It is the only reason this occurrence is actionable instead of another re-run.

Unrelated to #553's refactor — that PR touches Python hook libraries, not the ca-pi TypeScript background-job runtime.

Root-cause assessment (from the #580 investigation)

Filed while fixing #580 (bridge.test.ts hung-tree cancellation flake). Assessed whether it shares that root cause. Same conceptual category (a fixed window racing real Windows process-launch latency under CI load), different concrete mechanism - the #580 fix (plugins/ca-pi/tools/src/bridge.ts's killTree, plugins/ca-pi/tools/test/bridge.test.ts) touches neither background-jobs.ts nor the Windows Job Object containment stack this issue exercises, so it does not cover this issue and this is a diagnosis, not a fix. Not empirically reproduced - this dev box (32 logical cores) could not be loaded enough to trip the timing window either for #580 or for this.

AC-1: can `launch` resolve `undefined` on a successful spawn?

Yes, by construction - several branches in BackgroundJobManager#launch (plugins/ca-pi/tools/src/background-jobs.ts:769-850) return undefined after #openTree(...) has already returned a live tree, without surfacing any diagnostic to the caller:

if (slot.settling !== undefined) { const clean = await slot.settling; if (!clean || ...) return undefined; return this.#manager.getJob(job.id); } (background-jobs.ts:823-828) - fires if the child's close/error listeners (registered just above, background-jobs.ts:812-816) settle the slot before launch()'s own continuation gets there. This is a genuine race between the child's async lifecycle events and launch()'s sequential awaits, not a deadline.
if (!ready) { await this.#settle(slot, "startup_failure", "failed"); return undefined; } (background-jobs.ts:829-832) - I traced tree.cleanup.ready() (process-tree.ts:946-953) and, for the win32 contained path, it reads an already-resolved promise set during spawnProcessTree (process-tree.ts:757, 838) - so by the time #openTree has returned successfully this should normally already be true. If it is ever false here, spawnProcessTree itself would ordinarily have thrown instead of returning, so this branch firing at all would itself be worth an issue.
try { tree.child.stdin.end(launch.stdin); } catch { await this.#settle(slot, "startup_failure", "failed"); return undefined; } (background-jobs.ts:841-842) - matches the issue's candidate feat(claude): Complete .claude/ system rewrite — routing-table-driven orchestration #1 (a shellPath that becomes momentarily unusable): an EPIPE here (child already exited) silently produces the same "resolved undefined" symptom.
const active = this.#manager.transitionJob(...); if (active === undefined) return this.#manager.getJob(job.id); (background-jobs.ts:843-844) - if the job was already transitioned/removed by a concurrent #settle (triggered by an early close/error), getJob can itself return undefined here.

All four are variations of the same shape: the child's close/error handlers race launch()'s own remaining sequential work, and if the child settles first (e.g., because the Windows Job Object containment handshake underneath it hit real, load-dependent latency and something downstream reacted badly to it), launch() quietly returns undefined through one of these branches. None of them go through reportRefusal/describeSpawnRefusal (background-jobs.ts:800-801) - that diagnostic path only covers #openTree throwing, not the tree opening successfully and then racing its own teardown.

AC-2/AC-3 direction (not implemented here)

Per AC-2, launch failing loud instead of resolving empty likely means: give each of the above branches (not just the #openTree catch) a reportRefusal-shaped diagnostic before returning undefined, so a caller (and this test) can tell "no handle because X" apart from "no handle, no idea why." Per AC-3, since the underlying registration in #slots.set(job.id, slot) (background-jobs.ts:807) happens before the ready-check/stdin/transition steps that can undo it, the job is briefly tracked and then untracked on these paths - worth confirming explicitly that no process outlives that window untracked, which is the leak AC-3 is about.

Leaving open; no code changes made against this issue's surface.

fix(ca-pi): stop bridge.test.ts's hung-tree cancellation flake racing…

Navigation Menu

Repository navigation

flaky: live Git Bash background job resolves with no handle on windows-latest #504

The flake

Why this is NOT the same bug as #495

What that points at

Why it matters beyond a re-run

Acceptance criteria

Relevant sources

AC-1 answered: it is a designed refusal, not a tracking race

What is actually wrong

Revised acceptance criteria

Not fixed here

AC-4 evidence — the flake has not recurred since the diagnostic landed

Closing

Reopening: the next occurrence, with the diagnostic attached — as promised

What that tells us, and it changes the diagnosis

Revised acceptance criteria

Root-cause assessment (from the #580 investigation)

AC-1: can `launch` resolve `undefined` on a successful spawn?

AC-2/AC-3 direction (not implemented here)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

flaky: live Git Bash background job resolves with no handle on windows-latest #504

Description

The flake

Why this is NOT the same bug as #495

What that points at

Why it matters beyond a re-run

Acceptance criteria

Relevant sources

Activity

SUaDtL commented on Jul 26, 2026

AC-1 answered: it is a designed refusal, not a tracking race

What is actually wrong

Revised acceptance criteria

Not fixed here

SUaDtL commented on Jul 26, 2026

SUaDtL commented on Jul 28, 2026

AC-4 evidence — the flake has not recurred since the diagnostic landed

Closing

SUaDtL commented on Jul 30, 2026

Reopening: the next occurrence, with the diagnostic attached — as promised

What that tells us, and it changes the diagnosis

Revised acceptance criteria

SUaDtL commented on Aug 5, 2026

Root-cause assessment (from the #580 investigation)

AC-1: can launch resolve undefined on a successful spawn?

AC-2/AC-3 direction (not implemented here)

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

AC-1: can `launch` resolve `undefined` on a successful spawn?