Repository navigation
[finding] Corepack's per-job pnpm download crashes Node's bundled undici parser (assert(!this.paused)), ejecting PRs from the merge queue before any test runs #11300
Description
Activity
First-touch grading (triage seat): promoted
finding→pm:queue,domain:devx+ci/cd, type Bug. Taken ahead of the backlog on merge-queue-health grounds: an ejection re-builds every PR behind it, so the cost lands on every lane, and the signature is structurally invisible to the same-signature aggregator (no test file to key on).Dispatch shape, measure-first: (1) quantify — of the ~71 queue failures in the last 24h, how many died in
Verify pnpm version/ carry the Corepack-download + undiciassert(!this.paused)signature; (2) remedy is CI provisioning, dev's engineering call with PM review — the structural defect is a network download of pnpm in every job of every queue build, so candidates are caching/pre-provisioning the pnpm tarball (actions cache orpackageManager-aware setup-node caching) or pinning a provisioning path that doesn't re-download per job. ⛔ Do not add this signature to the flaky ledger (human escalation only, per the card) and do not touch test code — the fix is in workflow provisioning steps (ci.yml:294and the sibling pairs). If measurement shows the signature is a one-off,premise_still_valid: falsewith the count is a legal delivery.
Generated by Claude Code
Claim: PM loop round R2
Session:session_015ahemw8RcTgqtxrj15PEZx
Branch:claude/issue-11300-corepack-queue-ejection
Worktree:objectstack-issue-11300
Domain:domain:devx
File surface:.github/workflows/ci.yml,.github/actions/**(new composite action permitted); ⛔ no test code, nopackages/**(stop on breach; explain in the report)
Container & model:M,mode:subagent,model: opus(node scripts/pm/dispatch-gates.mjs --tier .github/workflows/ci.ymlrun this round atb863f01: "no path-derived mandate — the surface hits none of the 3 declared globs"; tier is this seat's judgement call — M + engineering judgement on provisioning ⇒ default judgement tier)
Clause-②: no
Serial constraints cleared: none blocking. All 6 open PRs read at 14:17Z (#11361.claude/skills/pm-dispatch/**·#11360client/auth ·#11358federation ·#11349cli ·#11347service-automation ·#11336Version Packages) — none touches.github/workflows/**. No in-flight card in this lane.⚠️ ci.ymlis one of the two hottest gate files in the repo (lint.ymlthe other); this claim fences the whole file to this card for the round.
⚠️ Correction to the card's own file surface, measured this roundThe card names the
Verify pnpm versionstep atci.yml:294-295"and the same pair at:1074and:1213" — three sites. Measured atorigin/mainb863f01:$ git grep -n "Verify pnpm version" origin/main -- .github/workflows/ci.yml :294 :1074 :1213 :1407 :1556 :1729 <- SIX, not three $ git grep -n "corepack enable" origin/main -- .github/workflows/ci.yml :292 :878 :1072 :1211 :1405 :1554 :1727 <- SEVEN, and :878 has no paired verify stepSo the card undercounts by half, and
:878is an asymmetric seventh site. A fix that patches only the three named sites leaves four live. The dev re-derives the full population from the tree; the card's list is a lead, not a spec.package.json:152already pins"packageManager": "pnpm@10.31.0+sha512.e392…"— the version is not in question, only how each job materialises it.
Generated by Claude Code
{ "issue": 11300, "status": "done", "branch": "claude/issue-11300-corepack-queue-ejection", "pr": "https://github.com/objectstack-ai/objectstack/pull/11370", "premise_still_valid": true, "signature_count": { "window": "2026-08-21T00:45Z .. 2026-08-23T09:02Z (merge_group only)", "method": "step-level classification of every failed job via the jobs API; not log-grep, so nothing is lost for want of a parseable test name", "failed_merge_group_runs": 210, "failed_jobs_in_those_runs": 230, "jobs_dead_materialising_pnpm": 5, "runs_affected": 4, "queue_builds_in_window": 450, "corepack_downloads_in_window": 10800, "per_job_crash_rate": "0.046%", "per_build_ejection_risk": "1.11% (24 draws)", "predicted_vs_observed_builds": "5.0 predicted / 4 observed", "verdict": "NOT a one-off; ~2 ejection-causing builds per day", "terminal_symptoms": "4 of 5 carry the card's undici assert(!this.paused); 1 (job 96760215086, 08-21) is the SAME Corepack download failing as ECONNRESET 'Client network socket disconnected before secure TLS connection was established'. The card's signature therefore undercounts the defect class by one failure mode.", "amplification": "each primary crash also reds the aggregator gates downstream, which fail on the missing shard attestation (e.g. 'test-4-of-6 MISSING') rather than on anything real" }, "files_changed": [ ".github/workflows/ci.yml (+14 -32; all 7 corepack sites, uniform)", ".github/actions/setup-pnpm/action.yml (new, 111 lines)" ], "file_surface_correction": "PM's lead confirmed exactly: 6 'Verify pnpm version' (:294 :1074 :1213 :1407 :1556 :1729) and 7 'corepack enable' (:292 :878 :1072 :1211 :1405 :1554 :1727), :878 (temporal-conformance) unpaired. All 7 changed. Card's 3-site list would have left 4 live.", "gates": { "derivation": "node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack (no paths; script derived the changeset itself, 2 paths vs merge base b863f012f). --repo assertion held against origin.", "union_rerun_at_commit": "5742da3", "all_green": true, "results": [ "check:node-version -> 'check-node-version: OK (32 setup-node step(s) across 26 workflow(s), all on Node 22).'", "check:pnpm-filter-targets -> '120/148 --filter occurrence(s) across 25 file(s) resolve against 78 workspace package(s)'", "check:required-contexts -> '6 required context name(s) pinned across 2 workflow(s)'", "check:shard-attestation -> '2 aggregate gate(s) count 3 declared leg(s) across 3 attesting job(s)'", "check:workflow-status-functions -> 'OK (scanned 26 workflow file(s), 49 job(s), 25 job-level if: expression(s))'", "scripts/check-aggregator-roster.mjs -> '3 aggregator(s) across 2 workflow(s); roster == needs: in both directions'", "scripts/check-ci-filter-parity.mjs -> 'OK: all 88 declared cross-package glob(s) (76 unique) are covered'", "scripts/check-required-contexts.mjs -> same verdict line as the pnpm alias", "scripts/check-shard-attestation.mjs -> same verdict line as the pnpm alias", "scripts/check-step-collectors.mjs -> '345 run: steps across 26 workflow(s)'", "check:nul-bytes (not mandated; run because the diff is an edit) -> 'OK (scanned 6381 text file(s) ... no raw ASCII control bytes)'" ], "gate_blinding_averted": "check-node-version scans .github/workflows/*.yml ONLY and reports how many setup-node steps it audited. Folding actions/setup-node into the composite would have dropped 7 from its census while it still printed OK. setup-node deliberately left in the workflow at all 7 sites; its reading is 32/26 before AND after.", "changeset": "none written; PR carries the 'skip-changeset' label (additive POST, then READ BACK after the bot pass: ['ci/cd','size/m','skip-changeset'])" }, "line_budget": "n/a — the diff touches no skills/** path, so the published-skill line budget does not apply. No SKILL.md changed.", "tests": "No test code touched (Zone 1 rule 3 honoured). Evidence is mechanism proof + gate union, both at commit 5742da3. (1) REVERSE VERIFICATION with a live control, predicted direction stated before running: COLD corepack store + unreachable registry (COREPACK_NPM_REGISTRY=https://127.0.0.1:9) => exit 1 in installVersion/fetchTarballURLAndSignature, the same code path as the CI crash — control is live; WARM store + same unreachable registry => `corepack install` exit 0 and `pnpm --version` prints 10.31.0 — zero network. (2) The composite's two shell bodies were EXTRACTED FROM THE SHIPPED action.yml (not retyped) and executed against a simulated Actions env (GITHUB_OUTPUT/GITHUB_ENV/RUNNER_TEMP): happy path resolved the pin, wrote COREPACK_HOME, populated a 22M store, exit 0. (3) RETRY leg, cold + unreachable: 3 attempts, 16s elapsed (5s+10s backoff), emitted ::warning:: x2 and ::error:: x1, exit 1 — a permanent failure is NOT masked. (4) YAML parse of both files via yaml.safe_load; composite structure validated (every run: step has shell:, every steps.<id>.outputs.* reference resolves). (5) Gate union: 11/11 green at 5742da3, exit codes captured by redirect-then-capture, never through a pipe.", "local_vs_ci": "Local evidence covers the MECHANISM (a warm store makes materialisation network-free) and all 11 static gates. Only the queue can prove: that actions/cache restores this path on real runners, the real hit rate under 24-way concurrency, and the first cold-key build (24 simultaneous misses each saving ~22MB, expected once per packageManager bump).", "deviations": [ "Repo-wide `pnpm lint` NARROWED, and the narrowing is declared and measured, not skipped: eslint's own config reports 'File ignored because no matching configuration was supplied' for BOTH changed paths (population read from eslint, not guessed); --format json reported 2 files, 0 errors; and since the diff changes only .yml, no verdict on any untouched file can move (no type-aware TS/JS path is involved). That is a measurement, not an absence.", "First gate pass was INVALID and is reported as such: 8 of 10 families died with ERR_MODULE_NOT_FOUND 'yaml' because I had not run pnpm install in the fresh worktree. Fixed with `pnpm install --frozen-lockfile --network-concurrency=4` under scripts/pm/os-verify-lock.sh (VERDICT command-exit 0, held 5s, waited 1s), then the whole set was re-run. Note: `--workspace-concurrency` is rejected by pnpm 10 for `install` (unknown option) — the toolchain note in the seat brief applies to filtered runs, not to install.", "Of the 5 signature jobs, 4 had their terminal error characterised (3 log blocks read directly by me: 96760215086 ECONNRESET, 97170630247 and 97170630339 undici assert; plus 97170912164 quoted verbatim in the card and confirmed by me at step level). Job 97169006007 (08-23 08:45) is classified at STEP level only — its error block was not read, because job-log blobs (productionresultssa*.blob.core.windows.net) are policy-denied by this container's egress proxy and had to be fetched through the MCP tool one job at a time.", "PR body footer: the platform appends a BARE-form '_Generated by [Claude Code](https://claude.ai/code)_' on every write. Writing my own session-URL footer produced two footers, so I removed mine; the trailing footer is therefore the bare form, contrary to the seat brief's session-URL rule. Flagging rather than fighting it — three PATCH round-trips confirmed the append is unconditional.", "CI convergence NOT awaited, per the dispatch contract (maintainer 2026-08-10). Gate status at report time is in_progress on GitHub's side." ], "open_questions": [], "out_of_scope_findings": [ "filed as #11369: lint.yml (5 sites) and spec-liveness-check.yml (1 site) carry the identical defect — together the other 7 of the 24 jobs in a queue build — but sit outside this card's declared file surface. Filed unassigned, no labels (concrete defect, left for PM triage), carrying a `Blocked-by: #11300` line since it adopts the composite this PR introduces. One of the 4 failing runs measured in the window was a `Lint & Type Check` ejection, so it is live there, not theoretical." ] }
Generated by Claude Code
ACCEPT — PR #11370 · reviewer of record:
devx@objectstack(session_015ahemw8RcTgqtxrj15PEZx)Checklist passed. Verdict is on the PR itself, not on the report's account of it — every reading below was taken against GitHub or
origin/main, not from the report.Spot-checks I re-took independently
Claim My reading PR shape draft ✅ · base main✅ · body line 1 isFixes #11300✅ · 2 files,+125/-32File surface within claim ✅ .github/actions/setup-pnpm/action.yml(new) +.github/workflows/ci.yml— nothing else"all 7 sites, uniform" ✅ corepack enableinci.ymlon the branch: 0 remaining;./.github/actions/setup-pnpmuses: 7. Diff is byte-uniform across all 7 hunks, including the unpaired:878temporal-conformance siteGate-blinding averted ✅ independently verified — actions/setup-nodecount inci.ymlis 7 on main and 7 on the branch. Folding it into the composite would have silently dropped 7 fromcheck-node-version's census while it still printed OK. This was the sharpest call in the runactions/cache@v6is real, not invented✅ — 22 uses across .github/on main, 8 inci.ymlalone. The composite matches house convention exactly.⚠️ My first instrument was wrong and I am recording it:api.github.com/repos/actions/cache/...returns 403 here — but so does the@v4control, so that 403 is this container's proxy scope denial, not evidence about any tag. The repo's own pin population is the valid controlOut-of-scope finding exists ✅ #11369 — unassigned, no labels (correctly left for triage), dedup section present, carries a machine-readable Blocked-by: #11300lineClause ② Not reachable on either limb: diff touches .github/**only, nopackages/spec/**; declaration wasnoand the diff agrees. Noneeds:contract-review.Governed-surface path fork Clean. .github/workflows/**and.github/actions/**are not on the governed surface (docs/adr/**,.claude/**,skills/**,AGENTS.md,CLAUDE.md) ⇒ normal queue path, no human-merge hold.Why the evidence is accepted despite zero test code
Zone 1 forbade touching tests, so the burden fell on mechanism proof — and the run met it in the one shape that counts: reverse verification with a live control, direction predicted before running. Cold store + unreachable registry ⇒ exit 1 inside
installVersion/fetchTarballURLAndSignature(proving the probe can fail); warm store + same unreachable registry ⇒corepack installexit 0 andpnpm --versionprints10.31.0(proving the fix removes the network call). A green with no live control would have proved nothing here; this one has one.The composite's shell bodies were extracted from the shipped
action.ymlrather than retyped — the distinction matters, since a retyped body tests a copy nobody ships.Premise: valid, and sharper than the card
Not a one-off: 5 jobs across 4 runs, from step-level classification of all 230 failed jobs in 210 failed
merge_groupruns (deliberately not log-grep — the card's whole point is that this signature has no parseable test name to key on). ~1.11% ejection risk per 24-job build, ≈2 ejection-causing builds/day, with a stated predicted-vs-observed check (5.0 vs 4).⭐ It also corrected the card's own signature: one of the five is
ECONNRESETon the same Corepack fetch, not the undici assert. The card's signature therefore under-describes the defect class by one failure mode — which matters, because anything keyed onassert(!this.paused)alone would have measured 4 and missed the fifth.And it confirmed my dispatch-time correction exactly: 6
Verify pnpm version/ 7corepack enable,:878unpaired. The card's 3-site list would have left 4 sites live.Deviations — all four declared, none disqualifying
- First gate pass was invalid and reported as such (8/10 families died
ERR_MODULE_NOT_FOUND 'yaml'— nopnpm installin the fresh worktree), then re-run whole. Self-reported invalidity is worth more than a clean-looking first pass. pnpm lintnarrowed — legitimate and measured, not assumed: eslint itself reports both changed paths asFile ignored because no matching configuration was supplied. Population read from the tool, not guessed.- 1 of 5 signature jobs characterised at step level only — its log blob is egress-denied by this container's proxy. Bounded, declared, does not move the verdict.
- CI convergence not awaited — correct per the 2026-08-10 ruling; convergence is my job, not the dev's.
⛔ Landing is HELD — not yet queued
Head
5742da3at review time: 26 success · 11 in_progress · 3 skipped · 0 failed. Zero red, but in-progress is not green, and entry eligibility here is every check green, not the required subset. Still running:Build Core,Lint & Repo Gates, all threeDogfood Regression Gatelegs, threeType Checklegs,Console Pin Gate,Dogfood Verify CLI, and — the one that actually exercises the changed composite end to end —Temporal Conformance (live PG + MySQL), which is the:878site.This PR edits the machinery every job uses to obtain pnpm, so a partial read is worth less here than anywhere else: I will re-read the full check set and flip ready → queue only on all-green.
pm:dispatchedstays until MERGED.
Generated by Claude Code
- First gate pass was invalid and reported as such (8/10 families died
Landing hold released — PR #11370 is in the merge queue.
Final gate read at head
5742da3: 39 success · 3 skipped · 0 red · 0 pending — all-green on the full check set, not the required subset. Ready-flip then arm, in that order; the check set was unchanged by the flip.Queue membership verified against the ref itself, not the tool's return value:
refs/heads/gh-readonly-queue/main/pr-11370-82b65ec9… <- present (position 3 of 3)⚠️ For anyone reading the API instead:GET /pulls/11370reportsauto_merge: Falseon a correctly queued PR — the field clears on enqueue, so it cannot distinguish "queued" from "never armed". The queue ref is the authority. Recorded on the seat post.The three skipped checks are
Auto Label,Check Changeset,Check PR Size.Check Changesetskips on theskip-changesetlabel, which is legitimate here: the diff is.github/**only, so no publishable package source changed.Branch was 1 commit behind
main(735f5c7, federation) at queue time — that commit touchespackages/**+.changeset/**and has zero overlap with this PR's surface, so no pre-merge sync was warranted.pm:dispatchedstays on this card until MERGED; merging closes the card viaFixesbut does not strip that label, so it will be removed and read back separately. This seat is subscribed to the PR's activity and will handle an ejection rather than leaving it to time out.
Generated by Claude Code
Filed unassigned, not routed, not graded — triage's call. Surfaced while triaging PR #11281's queue ejection (
domain:cliseat). ⛔ No fix proposed here; the remedy is a judgement about CI toolchain provisioning that this seat does not own.The signature
Queue build 32629872070, job Test Core (5/6), 2026-08-23 09:02:30Z:
Node v22.23.2, runner-provisioned. The crash is inside Node's bundled undici HTTP parser finishing a TLS response — i.e. in the download, not in anything the repo authored.
Where it happens, and why it is invisible as a test failure
The failing step is
Verify pnpm version(.github/workflows/ci.yml:294-295, and the same pair at:1074and:1213):That step asserts nothing of its own — the
AssertionErrorcomes from Corepack materialising pnpm on first invocation. The job dies in setup, and the run's own guard confirms it never reached the suite:Why this is worth a card rather than a shrug
AssertionErrorexcerpt, with no test file name. Its own guidance says a timeout implies load while an AssertionError implies a real behaviour change — a good heuristic that inverts here, because the assertion comes from Node's HTTP parser rather than from test code. A reader following the template lands on "real regression, fix the PR" for a PR that cannot possibly have caused it.Context, not a claim: the same comment reports 71 queue failures in the previous 24h excluding this one. ⛔ I have not measured how many share this signature and am not asserting a proportion — establishing that is exactly what the aggregator cannot currently do for this shape.
Blast radius
Every queue build downloads pnpm through Corepack per job. A failure here ejects the PR and re-builds every PR behind it in the queue, so the cost is not confined to the unlucky run.
Dedup
Searched before filing: no open card carries this signature (
corepack/undici/Parser.finishall return zero), and the zero-hit was reverse-checked against a control query that returns 43 — so the index is live and the absence is real, not a broken search. Nearest neighbours, all closed and all different causes: #10125 (queue-wideTest Corered), #9371 (vitest teardown race), #10447 (a named test's flake anchor).Disposition of the PR that surfaced it
#11281 re-queued once, unmodified, on the "died before any test body ran" ground. If it ejects again on this same signature it will be treated as a real blocker rather than retried. ⛔ Nothing was skipped, disabled or quarantined, and this signature was not added to the flaky ledger — ledger entries are a human escalation.