Repository navigation
flaky: a task escalates under full-suite load, flipping exit 0 to 2 in the farm artifact-publication tests #515
Description
Activity
- changed the title
[-]flaky: a 12-task/6-way farm run escalates one task under full-suite load (farm.test.ts #387)[/-][+]flaky: a task escalates under full-suite load, flipping exit 0 to 2 in the farm artifact-publication tests[/+]on Jul 26, 2026 Second occurrence — a different test, same signature. Retitling warranted.
The title scopes this to the 12-task/6-way case. A second full-suite run hit a different test in the same
describeblock:FAIL farm.test.ts > farm artifact publication (#397 / #387) > #387: a failed LATEST report pointer does not fail a run whose authoritative receipt landed Done. green=1 escalate=1 blocked=0 : expected 2 to be +0 ❯ farm.test.ts:1657:37 expect(result.code, result.out).toBe(0);That test does not set
FARM_CONCURRENCYand runs two tasks, not twelve. So the trigger is not six-way concurrency specifically — AC-4 as filed is too narrow.Revised characterisation
What both occurrences share is the only thing that matters: one task escalates, under full-suite load, in a test that asserts a clean exit 0.
runExitCodemapsescalated || blocked || aborted || cleanupFailuresto 2, so the exit code is a truthful report of the escalation — the flake is upstream of it.Every affected test is in the
farm artifact publication (#397 / #387)block, and each one deliberately sabotages an artifact path (an unwritable latest pointer, a regular file where the diffs directory belongs) while asserting the run still exits 0. Whatever escalates is either (a) genuinely provoked by that sabotage and merely racy, or (b) unrelated contention that happens to land in the block with the strictest exit-code assertions.Distinguishing (a) from (b) is the actual first step, and it is cheap: the escalating task's
noteis already in the report JSON these tests write. Capturing it on a failure would name the cause outright.Frequency
Three full-suite runs on 2026-07-26 during the #511 baseline work:
run result 1 (main @ c9c5cbc, clean tree) #387: the unavailable-diff list is bounded…FAILED —green=11 escalate=12 (main @ c9c5cbc, clean tree) all green 3 (branch, +1 unrelated test file) #387: a failed LATEST report pointer…FAILED —green=1 escalate=1Both failures reproduce only in full-suite runs; each passed in isolation. Roughly one full run in two, which is high enough to redden CI on most PRs and is what makes it worth fixing rather than tolerating.
Amended acceptance criteria
- AC-1 (replaces): the escalation is explained — the escalating task's
note/reason is captured on failure and named, before any fix is chosen. - AC-2: unchanged — if the cause is harness contention, the tests are made deterministic without weakening what they assert.
- AC-3: unchanged.
- AC-4 (replaces): no change dodges the problem by relaxing the exit-code assertion, reducing task counts, or lowering
FARM_CONCURRENCY— the clean-exit-0-under-sabotage contract is the thing under test.
- AC-1 (replaces): the escalation is explained — the escalating task's
Diagnostic attempt — AC-1 not closed, but several hypotheses ruled out
Timeboxed investigation while working #511. I did not capture the escalating task's note, so AC-1 is still open. What I did establish narrows the search enough to be worth recording, mainly so the next attempt does not repeat the stressor that does not work.
Method
The run reports die with the harness
tmpDir(rmSyncin bothafterEachhooks), so a flaked escalation leaves nothing to read. I patched both hooks locally — uncommitted, since reverted — to copy everyfarm-report.jsonout before the wipe, then tried to force the failure.Results
condition outcome reports captured unexpected escalations idle machine, 4 consecutive full runs all pass 44 0 14 synthetic CPU hogs / 32 cores pass — 0 28 synthetic CPU hogs / 32 cores pass 59 0 40 synthetic CPU hogs / 32 cores 23 tests fail — mass The 40-hog run is not the flake. It collapses the suite wholesale — smoke tests,
farm-dist, everything — rather than producing the isolatedgreen=N escalate=1signature seen in the wild. Every non-green result captured at 28 hogs was an intentional escalation from a test that exercises an escalation path on purpose (drift:,gaming:,worker error:, timeouts). None was the flake.What this rules out
- Not a deterministic dispatcher race. 4 idle runs and 59 captured reports produced zero unexpected escalations.
- Not pure CPU contention. 28 of 32 cores saturated is heavy, and the suite passed clean. Synthetic CPU load is the wrong stressor and the next attempt should not spend time on it.
What it points at
All three real-world occurrences happened while other agent work was running concurrently — subprocess-heavy, mixed I/O:
vitestin other trees,git,npx,nodespawns. That is process-spawn and filesystem contention, not arithmetic. Busy-loops do not reproduce it because they never contend for the thing that matters.Two candidates worth trying next, in order:
- Spawn-table / handle pressure — the failing tests spawn the farm, which spawns
bash -c/cmd.exe /cper gate, which spawnsnode. Concurrentnpm/gitelsewhere multiplies that. A stressor that spawns thousands of short-lived processes would test this; a busy-loop would not. - Filesystem contention on
%TEMP%— every one of these tests builds a temp git repo undertmpdir(), and so does every other suite in the tree. Antivirus or indexing on that path under concurrent churn would fit the timing.
Frequency, updated
Six full-suite runs on 2026-07-26 during #511 slice 1, on an otherwise-busy machine: 3 failed. Four further runs on an idle machine: 0 failed. Both failing tests are in
farm artifact publication (#397 / #387)and both assert a clean exit 0 while deliberately sabotaging an artifact path.Recommendation
Keep AC-1 as written — the note is still the thing worth having. The cheap way to get it is not to reproduce on demand but to capture on failure: have the artifact-publication tests dump the run report (or just the non-green results) into the assertion message when the exit code is not 0.
expect(result.code, result.out)already passes the farm's stdout, which does not include per-task notes; adding them would have named this on the first occurrence instead of the sixth.That is a small, self-contained change to the harness and it does not weaken any assertion — it makes the existing one legible. Worth doing before the next investigation rather than after.
Second distinct test in this family, observed 2026-07-27
Same file and same load conditions, but a different test than the one this issue was filed for:
FAIL farm.test.ts > farm artifact publication (#397 / #387) > #397: two concurrent farm processes on one repo both keep complete, independently parseable receiptsObserved once in three consecutive
ca/toolsfull-suite runs on Windows (9 files, 489 tests, ~70s each), on an otherwise-green tree during #525's work. Green on the two other runs and on the immediately preceding and following runs.Not caused by the change it was observed under. #525 alters only the mutation
riskNotestring formatting; this test exercises concurrent receipt publication and never reaches that code.Worth recording because it widens the issue: this was filed as one flaky test (
#387, a task escalating under six-way concurrency). It is now at least two, both infarm artifact publication (#397 / #387), both involving concurrent farm processes contending on one repo. That points at the shared-contention setup those tests build rather than at either assertion — which is a different fix from hardening one test, and probably a cheaper one.The
diagnose()helper added in #518 lifts non-green results andreport.blockedinto the assertion message, so the next occurrence of the#387case should be self-describing. It is worth checking whether this#397case is wired into the same helper; if not, that is the smallest next step toward a root cause.- added 5 commits that reference this issue
on Jul 29, 2026
The flake
plugins/ca/tools/farm.test.ts > farm artifact publication (#397 / #387) > #387: the unavailable-diff list is bounded and still reports the true totalfails intermittently under full-suite load:The captured
result.outon the failing run:Observed 2026-07-26 on
main@c9c5cbc,windows, during anpm --prefix plugins/ca/tools run coveragerun. Re-running the same test in isolation passed, and a second full-suite run passed. So it is intermittent and load-sensitive rather than a regression.Why the exit code moved
runExitCode(farm.ts) mapsescalated || blocked || aborted || cleanupFailuresto 2. The test plans 12 tasks atFARM_CONCURRENCY: "6"against the in-process mock server. On the failing run, 11 tasks went green and one escalated, so exit 2 is a truthful report of what happened — the flake is that a task escalated at all, not that the exit mapping is wrong.That points at contention rather than at the assertion:
farm.test.tsruns alongside four other suites, several of which also spawn subprocesses, and this is the only case in the file that drives six concurrent workers.Why it is not #504
#504 is a
ca-pibackground-jobs.test.tsfailure — a live Git Bash launch resolving with no handle onwindows-latest. Different tree, different module, different symptom. Filed separately so neither investigation inherits the other's wrong lead.Why it matters
It reddens a suite that merge decisions are gated on. A green CI is one of the two standing conditions for merge authority in this repo, and a test that fails once per few full runs trains readers to re-run rather than read — the habit the gate exists to prevent. It cost one full coverage cycle while establishing the #511 baseline.
Acceptance criteria
farm.tsrather than the harness, it is fixed there and the fix carries a test that fails against the current dispatcher.FARM_CONCURRENCYbelow 6 merely to dodge the contention — the six-way path is the thing under test.Not in scope
Found while establishing the coverage baseline for #511. Deliberately not fixed inside that backfill: #511 adds no behaviour and modifies no pre-existing test (its AC-4), and this needs both.