Skip to content

[finding] Corepack's per-job pnpm download crashes Node's bundled undici parser (assert(!this.paused)), ejecting PRs from the merge queue before any test runs #11300

Description

@os-zhuang

Filed unassigned, not routed, not graded — triage's call. Surfaced while triaging PR #11281's queue ejection (domain:cli seat). ⛔ No fix proposed here; the remedy is a judgement about CI toolchain provisioning that this seat does not own.

The signature

Queue build 32629872070, job Test Core (5/6), 2026-08-23 09:02:30Z:

! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-10.31.0.tgz
node:internal/assert/utils:77
    throw err;
    ^
AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:

  assert(!this.paused)

    at Parser.finish (node:internal/deps/undici/undici:6165:9)
    at TLSSocket.<anonymous> (node:internal/deps/undici/undici:6499:36)
    at TLSSocket.emit (node:events:531:35)
    at endReadableNT (node:internal/streams/readable:1698:12)
  code: 'ERR_ASSERTION', actual: false, expected: true, operator: '=='
Node.js v22.23.2
##[error]Process completed with exit code 1

Node v22.23.2, runner-provisioned. The crash is inside Node's bundled undici HTTP parser finishing a TLS response — i.e. in the download, not in anything the repo authored.

Where it happens, and why it is invisible as a test failure

The failing step is Verify pnpm version (.github/workflows/ci.yml:294-295, and the same pair at :1074 and :1213):

- name: Verify pnpm version
  run: pnpm --version

That step asserts nothing of its own — the AssertionError comes from Corepack materialising pnpm on first invocation. The job dies in setup, and the run's own guard confirms it never reached the suite:

No test log — the test step did not get far enough to produce one.

Why this is worth a card rather than a shrug

⚠️ It presents as a test failure and is not one. The merge-queue triage bot reported it as "failed step: Verify pnpm version" attached to a bare AssertionError excerpt, with no test file name. Its own guidance says a timeout implies load while an AssertionError implies a real behaviour change — a good heuristic that inverts here, because the assertion comes from Node's HTTP parser rather than from test code. A reader following the template lands on "real regression, fix the PR" for a PR that cannot possibly have caused it.

⚠️ Cross-PR aggregation was unavailable for this ejection, and the bot says so explicitly: "这不是「没有同签名的其他 PR」,是这一轮没测到" — no FAIL line carried a parseable test file, so the aggregator had nothing to group on. This signature is therefore structurally invisible to the existing same-signature machinery, which keys on test files. That is the part that makes it accrete silently.

Context, not a claim: the same comment reports 71 queue failures in the previous 24h excluding this one. ⛔ I have not measured how many share this signature and am not asserting a proportion — establishing that is exactly what the aggregator cannot currently do for this shape.

Blast radius

Every queue build downloads pnpm through Corepack per job. A failure here ejects the PR and re-builds every PR behind it in the queue, so the cost is not confined to the unlucky run.

Dedup

Searched before filing: no open card carries this signature (corepack / undici / Parser.finish all return zero), and the zero-hit was reverse-checked against a control query that returns 43 — so the index is live and the absence is real, not a broken search. Nearest neighbours, all closed and all different causes: #10125 (queue-wide Test Core red), #9371 (vitest teardown race), #10447 (a named test's flake anchor).

Disposition of the PR that surfaced it

#11281 re-queued once, unmodified, on the "died before any test body ran" ground. If it ejects again on this same signature it will be treated as a real blocker rather than retried. ⛔ Nothing was skipped, disabled or quarantined, and this signature was not added to the flaky ledger — ledger entries are a human escalation.

Activity

  1. added theissue type on Aug 23, 2026
  2. os-zhuang commented on Aug 23, 2026

    @os-zhuang
    ContributorAuthor

    First-touch grading (triage seat): promoted finding → pm:queue, domain:devx + ci/cd, type Bug. Taken ahead of the backlog on merge-queue-health grounds: an ejection re-builds every PR behind it, so the cost lands on every lane, and the signature is structurally invisible to the same-signature aggregator (no test file to key on).

    Dispatch shape, measure-first: (1) quantify — of the ~71 queue failures in the last 24h, how many died in Verify pnpm version / carry the Corepack-download + undici assert(!this.paused) signature; (2) remedy is CI provisioning, dev's engineering call with PM review — the structural defect is a network download of pnpm in every job of every queue build, so candidates are caching/pre-provisioning the pnpm tarball (actions cache or packageManager-aware setup-node caching) or pinning a provisioning path that doesn't re-download per job. ⛔ Do not add this signature to the flaky ledger (human escalation only, per the card) and do not touch test code — the fix is in workflow provisioning steps (ci.yml:294 and the sibling pairs). If measurement shows the signature is a one-off, premise_still_valid: false with the count is a legal delivery.


    Generated by Claude Code

  3. self-assigned this
    on Aug 23, 2026
  4. os-steve commented on Aug 23, 2026

    @os-steve
    Collaborator

    Claim: PM loop round R2
    Session: session_015ahemw8RcTgqtxrj15PEZx
    Branch: claude/issue-11300-corepack-queue-ejection
    Worktree: objectstack-issue-11300
    Domain: domain:devx
    File surface: .github/workflows/ci.yml, .github/actions/** (new composite action permitted); ⛔ no test code, no packages/** (stop on breach; explain in the report)
    Container & model: M, mode:subagent, model: opus (node scripts/pm/dispatch-gates.mjs --tier .github/workflows/ci.yml run this round at b863f01: "no path-derived mandate — the surface hits none of the 3 declared globs"; tier is this seat's judgement call — M + engineering judgement on provisioning ⇒ default judgement tier)
    Clause-②: no
    Serial constraints cleared: none blocking. All 6 open PRs read at 14:17Z (#11361 .claude/skills/pm-dispatch/** · #11360 client/auth · #11358 federation · #11349 cli · #11347 service-automation · #11336 Version Packages) — none touches .github/workflows/**. No in-flight card in this lane. ⚠️ ci.yml is one of the two hottest gate files in the repo (lint.yml the other); this claim fences the whole file to this card for the round.


    ⚠️ Correction to the card's own file surface, measured this round

    The card names the Verify pnpm version step at ci.yml:294-295 "and the same pair at :1074 and :1213" — three sites. Measured at origin/main b863f01:

    $ git grep -n "Verify pnpm version" origin/main -- .github/workflows/ci.yml
    :294  :1074  :1213  :1407  :1556  :1729        <- SIX, not three
    $ git grep -n "corepack enable" origin/main -- .github/workflows/ci.yml
    :292  :878  :1072  :1211  :1405  :1554  :1727  <- SEVEN, and :878 has no paired verify step
    

    So the card undercounts by half, and :878 is an asymmetric seventh site. A fix that patches only the three named sites leaves four live. The dev re-derives the full population from the tree; the card's list is a lead, not a spec.

    package.json:152 already pins "packageManager": "pnpm@10.31.0+sha512.e392…" — the version is not in question, only how each job materialises it.


    Generated by Claude Code

  5. claude commented on Aug 23, 2026

    @claude
    Contributor
    {
      "issue": 11300,
      "status": "done",
      "branch": "claude/issue-11300-corepack-queue-ejection",
      "pr": "https://github.com/objectstack-ai/objectstack/pull/11370",
      "premise_still_valid": true,
      "signature_count": {
        "window": "2026-08-21T00:45Z .. 2026-08-23T09:02Z (merge_group only)",
        "method": "step-level classification of every failed job via the jobs API; not log-grep, so nothing is lost for want of a parseable test name",
        "failed_merge_group_runs": 210,
        "failed_jobs_in_those_runs": 230,
        "jobs_dead_materialising_pnpm": 5,
        "runs_affected": 4,
        "queue_builds_in_window": 450,
        "corepack_downloads_in_window": 10800,
        "per_job_crash_rate": "0.046%",
        "per_build_ejection_risk": "1.11% (24 draws)",
        "predicted_vs_observed_builds": "5.0 predicted / 4 observed",
        "verdict": "NOT a one-off; ~2 ejection-causing builds per day",
        "terminal_symptoms": "4 of 5 carry the card's undici assert(!this.paused); 1 (job 96760215086, 08-21) is the SAME Corepack download failing as ECONNRESET 'Client network socket disconnected before secure TLS connection was established'. The card's signature therefore undercounts the defect class by one failure mode.",
        "amplification": "each primary crash also reds the aggregator gates downstream, which fail on the missing shard attestation (e.g. 'test-4-of-6 MISSING') rather than on anything real"
      },
      "files_changed": [
        ".github/workflows/ci.yml (+14 -32; all 7 corepack sites, uniform)",
        ".github/actions/setup-pnpm/action.yml (new, 111 lines)"
      ],
      "file_surface_correction": "PM's lead confirmed exactly: 6 'Verify pnpm version' (:294 :1074 :1213 :1407 :1556 :1729) and 7 'corepack enable' (:292 :878 :1072 :1211 :1405 :1554 :1727), :878 (temporal-conformance) unpaired. All 7 changed. Card's 3-site list would have left 4 live.",
      "gates": {
        "derivation": "node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack (no paths; script derived the changeset itself, 2 paths vs merge base b863f012f). --repo assertion held against origin.",
        "union_rerun_at_commit": "5742da3",
        "all_green": true,
        "results": [
          "check:node-version -> 'check-node-version: OK (32 setup-node step(s) across 26 workflow(s), all on Node 22).'",
          "check:pnpm-filter-targets -> '120/148 --filter occurrence(s) across 25 file(s) resolve against 78 workspace package(s)'",
          "check:required-contexts -> '6 required context name(s) pinned across 2 workflow(s)'",
          "check:shard-attestation -> '2 aggregate gate(s) count 3 declared leg(s) across 3 attesting job(s)'",
          "check:workflow-status-functions -> 'OK (scanned 26 workflow file(s), 49 job(s), 25 job-level if: expression(s))'",
          "scripts/check-aggregator-roster.mjs -> '3 aggregator(s) across 2 workflow(s); roster == needs: in both directions'",
          "scripts/check-ci-filter-parity.mjs -> 'OK: all 88 declared cross-package glob(s) (76 unique) are covered'",
          "scripts/check-required-contexts.mjs -> same verdict line as the pnpm alias",
          "scripts/check-shard-attestation.mjs -> same verdict line as the pnpm alias",
          "scripts/check-step-collectors.mjs -> '345 run: steps across 26 workflow(s)'",
          "check:nul-bytes (not mandated; run because the diff is an edit) -> 'OK (scanned 6381 text file(s) ... no raw ASCII control bytes)'"
        ],
        "gate_blinding_averted": "check-node-version scans .github/workflows/*.yml ONLY and reports how many setup-node steps it audited. Folding actions/setup-node into the composite would have dropped 7 from its census while it still printed OK. setup-node deliberately left in the workflow at all 7 sites; its reading is 32/26 before AND after.",
        "changeset": "none written; PR carries the 'skip-changeset' label (additive POST, then READ BACK after the bot pass: ['ci/cd','size/m','skip-changeset'])"
      },
      "line_budget": "n/a — the diff touches no skills/** path, so the published-skill line budget does not apply. No SKILL.md changed.",
      "tests": "No test code touched (Zone 1 rule 3 honoured). Evidence is mechanism proof + gate union, both at commit 5742da3. (1) REVERSE VERIFICATION with a live control, predicted direction stated before running: COLD corepack store + unreachable registry (COREPACK_NPM_REGISTRY=https://127.0.0.1:9) => exit 1 in installVersion/fetchTarballURLAndSignature, the same code path as the CI crash — control is live; WARM store + same unreachable registry => `corepack install` exit 0 and `pnpm --version` prints 10.31.0 — zero network. (2) The composite's two shell bodies were EXTRACTED FROM THE SHIPPED action.yml (not retyped) and executed against a simulated Actions env (GITHUB_OUTPUT/GITHUB_ENV/RUNNER_TEMP): happy path resolved the pin, wrote COREPACK_HOME, populated a 22M store, exit 0. (3) RETRY leg, cold + unreachable: 3 attempts, 16s elapsed (5s+10s backoff), emitted ::warning:: x2 and ::error:: x1, exit 1 — a permanent failure is NOT masked. (4) YAML parse of both files via yaml.safe_load; composite structure validated (every run: step has shell:, every steps.<id>.outputs.* reference resolves). (5) Gate union: 11/11 green at 5742da3, exit codes captured by redirect-then-capture, never through a pipe.",
      "local_vs_ci": "Local evidence covers the MECHANISM (a warm store makes materialisation network-free) and all 11 static gates. Only the queue can prove: that actions/cache restores this path on real runners, the real hit rate under 24-way concurrency, and the first cold-key build (24 simultaneous misses each saving ~22MB, expected once per packageManager bump).",
      "deviations": [
        "Repo-wide `pnpm lint` NARROWED, and the narrowing is declared and measured, not skipped: eslint's own config reports 'File ignored because no matching configuration was supplied' for BOTH changed paths (population read from eslint, not guessed); --format json reported 2 files, 0 errors; and since the diff changes only .yml, no verdict on any untouched file can move (no type-aware TS/JS path is involved). That is a measurement, not an absence.",
        "First gate pass was INVALID and is reported as such: 8 of 10 families died with ERR_MODULE_NOT_FOUND 'yaml' because I had not run pnpm install in the fresh worktree. Fixed with `pnpm install --frozen-lockfile --network-concurrency=4` under scripts/pm/os-verify-lock.sh (VERDICT command-exit 0, held 5s, waited 1s), then the whole set was re-run. Note: `--workspace-concurrency` is rejected by pnpm 10 for `install` (unknown option) — the toolchain note in the seat brief applies to filtered runs, not to install.",
        "Of the 5 signature jobs, 4 had their terminal error characterised (3 log blocks read directly by me: 96760215086 ECONNRESET, 97170630247 and 97170630339 undici assert; plus 97170912164 quoted verbatim in the card and confirmed by me at step level). Job 97169006007 (08-23 08:45) is classified at STEP level only — its error block was not read, because job-log blobs (productionresultssa*.blob.core.windows.net) are policy-denied by this container's egress proxy and had to be fetched through the MCP tool one job at a time.",
        "PR body footer: the platform appends a BARE-form '_Generated by [Claude Code](https://claude.ai/code)_' on every write. Writing my own session-URL footer produced two footers, so I removed mine; the trailing footer is therefore the bare form, contrary to the seat brief's session-URL rule. Flagging rather than fighting it — three PATCH round-trips confirmed the append is unconditional.",
        "CI convergence NOT awaited, per the dispatch contract (maintainer 2026-08-10). Gate status at report time is in_progress on GitHub's side."
      ],
      "open_questions": [],
      "out_of_scope_findings": [
        "filed as #11369: lint.yml (5 sites) and spec-liveness-check.yml (1 site) carry the identical defect — together the other 7 of the 24 jobs in a queue build — but sit outside this card's declared file surface. Filed unassigned, no labels (concrete defect, left for PM triage), carrying a `Blocked-by: #11300` line since it adopts the composite this PR introduces. One of the 4 failing runs measured in the window was a `Lint & Type Check` ejection, so it is live there, not theoretical."
      ]
    }

    Generated by Claude Code

  6. os-steve commented on Aug 23, 2026

    @os-steve
    Collaborator

    ACCEPT — PR #11370 · reviewer of record: devx@objectstack (session_015ahemw8RcTgqtxrj15PEZx)

    Checklist passed. Verdict is on the PR itself, not on the report's account of it — every reading below was taken against GitHub or origin/main, not from the report.

    Spot-checks I re-took independently

    Claim My reading
    PR shape draft ✅ · base main ✅ · body line 1 is Fixes #11300 ✅ · 2 files, +125/-32
    File surface within claim ✅ .github/actions/setup-pnpm/action.yml (new) + .github/workflows/ci.yml — nothing else
    "all 7 sites, uniform" ✅ corepack enable in ci.yml on the branch: 0 remaining; ./.github/actions/setup-pnpm uses: 7. Diff is byte-uniform across all 7 hunks, including the unpaired :878 temporal-conformance site
    Gate-blinding averted ✅ independently verified — actions/setup-node count in ci.yml is 7 on main and 7 on the branch. Folding it into the composite would have silently dropped 7 from check-node-version's census while it still printed OK. This was the sharpest call in the run
    actions/cache@v6 is real, not invented ✅ — 22 uses across .github/ on main, 8 in ci.yml alone. The composite matches house convention exactly. ⚠️ My first instrument was wrong and I am recording it: api.github.com/repos/actions/cache/... returns 403 here — but so does the @v4 control, so that 403 is this container's proxy scope denial, not evidence about any tag. The repo's own pin population is the valid control
    Out-of-scope finding exists ✅ #11369 — unassigned, no labels (correctly left for triage), dedup section present, carries a machine-readable Blocked-by: #11300 line
    Clause ② Not reachable on either limb: diff touches .github/** only, no packages/spec/**; declaration was no and the diff agrees. No needs:contract-review.
    Governed-surface path fork Clean. .github/workflows/** and .github/actions/** are not on the governed surface (docs/adr/**, .claude/**, skills/**, AGENTS.md, CLAUDE.md) ⇒ normal queue path, no human-merge hold.

    Why the evidence is accepted despite zero test code

    Zone 1 forbade touching tests, so the burden fell on mechanism proof — and the run met it in the one shape that counts: reverse verification with a live control, direction predicted before running. Cold store + unreachable registry ⇒ exit 1 inside installVersion/fetchTarballURLAndSignature (proving the probe can fail); warm store + same unreachable registry ⇒ corepack install exit 0 and pnpm --version prints 10.31.0 (proving the fix removes the network call). A green with no live control would have proved nothing here; this one has one.

    The composite's shell bodies were extracted from the shipped action.yml rather than retyped — the distinction matters, since a retyped body tests a copy nobody ships.

    Premise: valid, and sharper than the card

    Not a one-off: 5 jobs across 4 runs, from step-level classification of all 230 failed jobs in 210 failed merge_group runs (deliberately not log-grep — the card's whole point is that this signature has no parseable test name to key on). ~1.11% ejection risk per 24-job build, ≈2 ejection-causing builds/day, with a stated predicted-vs-observed check (5.0 vs 4).

    ⭐ It also corrected the card's own signature: one of the five is ECONNRESET on the same Corepack fetch, not the undici assert. The card's signature therefore under-describes the defect class by one failure mode — which matters, because anything keyed on assert(!this.paused) alone would have measured 4 and missed the fifth.

    And it confirmed my dispatch-time correction exactly: 6 Verify pnpm version / 7 corepack enable, :878 unpaired. The card's 3-site list would have left 4 sites live.

    Deviations — all four declared, none disqualifying

    1. First gate pass was invalid and reported as such (8/10 families died ERR_MODULE_NOT_FOUND 'yaml' — no pnpm install in the fresh worktree), then re-run whole. Self-reported invalidity is worth more than a clean-looking first pass.
    2. pnpm lint narrowed — legitimate and measured, not assumed: eslint itself reports both changed paths as File ignored because no matching configuration was supplied. Population read from the tool, not guessed.
    3. 1 of 5 signature jobs characterised at step level only — its log blob is egress-denied by this container's proxy. Bounded, declared, does not move the verdict.
    4. CI convergence not awaited — correct per the 2026-08-10 ruling; convergence is my job, not the dev's.

    ⛔ Landing is HELD — not yet queued

    Head 5742da3 at review time: 26 success · 11 in_progress · 3 skipped · 0 failed. Zero red, but in-progress is not green, and entry eligibility here is every check green, not the required subset. Still running: Build Core, Lint & Repo Gates, all three Dogfood Regression Gate legs, three Type Check legs, Console Pin Gate, Dogfood Verify CLI, and — the one that actually exercises the changed composite end to end — Temporal Conformance (live PG + MySQL), which is the :878 site.

    This PR edits the machinery every job uses to obtain pnpm, so a partial read is worth less here than anywhere else: I will re-read the full check set and flip ready → queue only on all-green. pm:dispatched stays until MERGED.


    Generated by Claude Code

  7. os-steve commented on Aug 23, 2026

    @os-steve
    Collaborator

    Landing hold released — PR #11370 is in the merge queue.

    Final gate read at head 5742da3: 39 success · 3 skipped · 0 red · 0 pending — all-green on the full check set, not the required subset. Ready-flip then arm, in that order; the check set was unchanged by the flip.

    Queue membership verified against the ref itself, not the tool's return value:

    refs/heads/gh-readonly-queue/main/pr-11370-82b65ec9…    <- present (position 3 of 3)
    

    ⚠️ For anyone reading the API instead: GET /pulls/11370 reports auto_merge: False on a correctly queued PR — the field clears on enqueue, so it cannot distinguish "queued" from "never armed". The queue ref is the authority. Recorded on the seat post.

    The three skipped checks are Auto Label, Check Changeset, Check PR Size. Check Changeset skips on the skip-changeset label, which is legitimate here: the diff is .github/** only, so no publishable package source changed.

    Branch was 1 commit behind main (735f5c7, federation) at queue time — that commit touches packages/** + .changeset/** and has zero overlap with this PR's surface, so no pre-merge sync was warranted.

    pm:dispatched stays on this card until MERGED; merging closes the card via Fixes but does not strip that label, so it will be removed and read back separately. This seat is subscribed to the PR's activity and will handle an ejection rather than leaving it to time out.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions