[finding] scaffold-next-steps-pm.test.ts pins pnpm detection over a probe that resolves OUTSIDE the repo, so a Corepack fetch decides the verdict — one queue red already

What went red

Merge-queue build 32708418190,
Test Core (5/6), on PR #11609 (a test-only diff in packages/cli that cannot reach
packages/create-objectstack):

FAIL src/scaffold-next-steps-pm.test.ts
  > scaffolder "Next steps" names the package manager it actually detected (#10322)
  > with pnpm on PATH: prints pnpm consistently, never bare npm
AssertionError: expected '\n    cd my-app\n    npm install\n   …' to match /\bpnpm run dev\b/

Why it is not a real regression, and why that is the problem

The scaffolder decides what to print with a probe that swallows every failure mode into one answer
(packages/create-objectstack/src/index.ts:125):

function detectPackageManager(): string {
  try {
    execSync('pnpm --version', { stdio: 'ignore' });
    return 'pnpm';
  } catch {
    return 'npm';
  }
}

npm install in the output therefore means the probe threw, not the code chose npm. The two are
not distinguishable from the assertion message.

The test's own vacuity guard — expect(() => which('pnpm')).not.toThrow() — does not cover the
probe, because the two run in different working directories:

	cwd	what resolves
the test's sanity `command -v pnpm`	the package dir, inside the repo	the pinned `packageManager`, `pnpm@10.31.0`
the scaffold child's `pnpm --version`	`mkdtemp` under `os.tmpdir()`, outside the repo	Corepack's own default

Measured locally on one machine, one binary, two cwds:

$ (cd <repo root> && pnpm --version)   →  10.31.0
$ (cd /tmp        && pnpm --version)   →  10.33.0

The queue job runs pnpm through Corepack — COREPACK_HOME: /home/runner/work/_temp/corepack is in
that job's env, and a "Restore the Corepack store" cache step is part of it. os.tmpdir() on a GitHub
runner is /home/runner/work/_temp, which has no package.json above it, so the outer leg is the one
that has to resolve (and potentially fetch) a version the repo never pinned. Nothing in the assertion
distinguishes "the scaffolder regressed" from "that resolution failed once".

Related but distinct: #11369 is about Corepack downloads in workflow jobs. This is a Corepack
resolution inside a test's child process, and would survive that card being fixed.

Suggested shape (not a ruling — for triage)

The pin is worth keeping; what needs removing is the dependency on an unpinned resolution:

Make the probe's outcome the fixture rather than the ambient environment — run the pnpm leg under
a controlled PATH holding a stub pnpm that exits 0, the exact mirror of the npm-fallback leg's
existing makePnpmlessBin(). Both branches then become hermetic, and the test stops asserting a
property of the runner.
Independently: consider whether detectPackageManager() collapsing spawn failure and pnpm
absent into one answer is itself worth separating, since a user with pnpm installed but a slow or
failing probe is told to run npm. That is a user-facing question, not a test one — split it out if
triage agrees it is a card.

Filed unassigned, for grading. Not folded into #11609: different package, and that PR's card
(#11301) is a packages/cli exit-code pin.

Concentrated triage batch: finding → pm:queue + domain:cli, Bug, flake-tax priority — one merge-queue red already (run 32708418190, on a diff that cannot reach the package). Premise re-verified: detectPackageManager (index.ts:125) still swallows every failure mode into "npm". Fix shape: make the verdict hermetic — inject/stub the probe in the test (assert both branches deterministically) and, in the product, distinguish "probe threw" from "chose npm" so the assertion message stops lying. Do not loosen the vacuity guard.

Generated by Claude Code

Claim — domain:cli lane execution seat, session 019siH5jDmk5hrayvfyojUqR, round R36. Flake-tax priority.

Branch: claude/issue-11616-hermetic-pm-detection.
Declared file surface: the detectPackageManager seam (index.ts:125 per triage) + scaffold-next-steps-pm.test.ts + a changeset if the product behaviour moves.

This is the highest-leverage card in the batch and the reason is structural. Triage (5404837088) records one merge-queue red already — run 32708418190, on a diff that cannot reach the package. The merge queue is a shared serial resource across every lane: a flaky gate there does not tax this lane, it taxes all of them at once, and it does so on changes that have nothing to do with the flake. Every round this survives, some other seat pays for it.

Clause ②: no, assessed at claim time — nothing about an accepted input changes. ⛔ Re-affirm against what you build; if making the verdict honest turns out to change what create-objectstack does for a user (not just what it reports), say so and stop.

The defect, and ⛔ the fix is not "retry it"

detectPackageManager swallows every failure mode into "npm". The test then pins that verdict over a probe that resolves outside the repo, so a Corepack fetch — network, cache state, whatever the runner happens to have — decides the assertion. The test is not flaky because it is timing-sensitive; it is flaky because it is measuring the runner, not the code.

Triage's fix shape, and I am carrying it verbatim because both halves matter:

make the verdict hermetic — inject/stub the probe in the test (assert both branches deterministically) and, in the product, distinguish "probe threw" from "chose npm" so the assertion message stops lying. Do not loosen the vacuity guard.

⭐ The second half is the part that is easy to skip. A test made hermetic over a function that still collapses "I failed" into "npm" is a test that passes deterministically while asserting something false. The product change is what makes the hermetic test worth having.

⛔ Do not fix this by retrying, by widening a timeout, by adding a network guard, or by marking the test skipped/conditional. Those are the gate-weakening floor: a test that stops failing because it stopped measuring is worse than a red one, because the red at least told the truth.

⛔ Re-derive index.ts:125 at current origin/main and publish what you measure — in R35 this seat handed devs stale line numbers five times and measurement moved them every time.

Anti-vacuity — sharper than usual here, because the subject IS a vacuous pin

Assert both branches deterministically. A hermetic test that only exercises the npm path has swapped one vacuity for another.
Pin the distinction you introduce: a probe that threw must be distinguishable from a probe that chose npm, at the assertion, in a way that fails if the two are collapsed again.
Show every pin failing without the behaviour: revert only your source change, keep the pins, run. Confirm the mutation on disk with anchored greps in both directions first.
Any pin green in both states is a regression guard — name it, ⛔ never count it as red-before evidence.
⛔ refuse ≠ pass. in_progress is not success; cancelled is not success (discharge only via a later same-named run that passed).
⭐ Prove the instrument produces a positive before trusting its negative — inject a deliberate canary and confirm it is reported where you expect. R35 caught four refusals-wearing-results in one dev round, including one where both legs errored identically so the diff came out empty and read exactly like "no breakage".
⭐ If any pin drives a spawned process or the built artifact, rebuild on both ablation legs and prove the marker reached the artifact. R35's other dev got a false green by skipping exactly that.

Standing repo rules

Worktree-first: git worktree add ../objectstack-11616 -b claude/issue-11616-hermetic-pm-detection main && cd ../objectstack-11616 && pnpm install.
⛔ Never git stash; ⛔ never edit content/docs/releases/.
⛔ Do not edit the TEST_DEBT / type-check-debt ledger — finding: qa/http-conformance imports @objectstack/spec/contracts without declaring @objectstack/spec — its TEST_DEBT ledger ceiling was measured against build-state-dependent resolution #11788 owns it this round.
⚠️ Two sibling cards are in flight in the same package: os init scaffolds NAME_item.ts while the registry declares *.object.ts — after #11071 the same project gets both spellings #11598 owns packages/cli/src/commands/init.ts and os build --json's failure payloads carry no warnings, so the truncation notice's --json pointer is a dead end when a later gate fails #11772 owns compile.ts. Same package is exempt; ⛔ same file is not. scaffold-next-steps-pm.test.ts is YOURS this round — os init scaffolds NAME_item.ts while the registry declares *.object.ts — after #11071 the same project gets both spellings #11598's dev has been told to stop and report rather than edit it.
Changeset: if the product's reporting changes, one is required; if the change is test-only, skip-changeset is correct — say which and why.
Verify against origin/main; re-derive the gate family from your actual change set and again if it grows.
⛔ Do not mark the PR ready, do not enable auto-merge.

⛔ finding is for OBSERVATION-class items only. Concrete measured defects filed unlabelled, no domain:*, no type.

Generated by Claude Code

fix(create-objectstack): make the package-manager verdict hermetic an…

{
  "issue": 11616,
  "status": "done",
  "branch": "claude/issue-11616-hermetic-pm-detection",
  "pr": "https://github.com/objectstack-ai/objectstack/pull/12041",
  "premise_still_valid": true,
  "summary": "Both halves landed. SEAM RE-DERIVED at current origin/main (3954fb7df): detectPackageManager is still at packages/create-objectstack/src/index.ts:125 — triage's line number holds, measured not recalled; its only consumer is index.ts:440. Premise re-verified on this machine, one binary, two cwds: pnpm --version answers 10.31.0 inside the repo (the pinned packageManager) and 10.33.0 from os.tmpdir(), where Corepack resolves — and can fetch — a version nothing pins; the scaffold child runs in an mkdtemp there, which is what let the network decide the assertion. PRODUCT: the probe now returns a verdict record instead of a bare string — {pm:'pnpm',probe:'ok'} / {pm:'npm',probe:'absent'} / {pm:'npm',probe:'failed',detail} — extracted into a new module detect-package-manager.ts because index.ts calls program.parse() at import and cannot be imported by a test (the existing pkg-utils.ts / rewrite-identity.ts idiom). A 'failed' probe (pnpm on PATH, probe still threw) now prints a warning naming the real failure instead of passing itself off as a choice. TEST: every leg now runs under a PATH holding a stub pnpm whose exit status the test chooses, mirroring the existing makePnpmlessBin(); no retry, no widened timeout, no network guard, no skip. The vacuity guard was NOT loosened — it was re-aimed at the PATH the CHILD receives (the test-process-vs-child cwd mismatch was the bug) and made stricter by pinning resolution to the exact stub path.",
  "premise_note": "Triage's index.ts:125 verified correct at 3954fb7df — no drift this round.",
  "clause_2_reaffirmed": "NO user-visible behaviour change, re-affirmed against what was built, not assessed at claim time. Which package manager a run USES is still pnpm iff the probe succeeded, in all three verdict rows. The PATH lookup that separates 'absent' from 'failed' runs only AFTER the decision is made and feeds the message alone, so a miss there moves a warning's wording and never the tool's behaviour. That invariant is itself pinned ('regression guard: pm is pnpm if and only if the probe succeeded', asserted across both PATH-lookup outcomes). Only output moves: one warning in a case that was previously silent and wrong. stdio went 'ignore' -> ['ignore','ignore','pipe'] so the reason is reportable; neither stream is ever forwarded, so both a succeeding and a failing probe print exactly what they printed before.",
  "tests": "ALL RUN AGAINST TREE 60317c537 (= PR head; tree clean, ablation/canary legs restored byte-identically before the sweep was read). (1) SUITE: `pnpm --filter create-objectstack test` under the shared verify lock -> 'Test Files 14 passed (14) / Tests 177 passed (177)'; lock VERDICT line: 'command-exit 0 · held the lock 17s · waited 504s'. (2) TYPECHECK: `pnpm --filter create-objectstack typecheck` -> exit 0 ('tsc --noEmit'); both new files confirmed inside the tsc program via --listFiles (only src/templates is excluded). (3) FULL-REPO LINT, not narrowed: `pnpm lint` (eslint . --no-inline-config) -> exit 0, 142s. (4) BOTH BRANCHES, DETERMINISTIC: pnpm-reachable leg asserts /\\bpnpm run dev\\b/ + /\\bpnpm run validate\\b/ and NOT /\\bnpm run\\b/; pnpm-absent leg asserts /\\bnpm run dev\\b/ and NOT /pnpm/; the 'both branches name validate' test used to run one ambient leg twice under a name claiming two and now genuinely runs both. (5) THE DISTINCTION PINNED end to end: 'probe failed' must print the fallback warning, 'absent' must NOT — either direction of a future collapse turns one of that pair red. (6) ABLATION, 4 legs + canary, 111s, source reverted with pins kept; every mutation confirmed ON DISK with anchored greps in BOTH directions before each run, every leg self-restoring via trap ... EXIT INT TERM. LEG 0 as-built 177/177 green. LEG B1 (collapse the distinction in the module: catch always returns probe:'absent') -> vitest exit 1, 3 RED: both unit distinction pins ('\"probe threw\" and \"chose npm\" are different verdicts, not one', 'only the \"probe threw\" verdict carries a detail to report') plus the e2e warning pin. LEG B2 (drop the product's reporting: if (detected.probe === 'failed') -> if (false)) -> exit 1, 1 RED: the e2e warning pin, independently of B1. LEG B3 (re-hardcode npm, the ORIGINAL #10322 defect) -> exit 1, 2 RED: 'with pnpm on PATH: prints pnpm consistently' + 'both branches ... validate step' — this is the key anti-vacuity proof that stubbing pnpm did NOT neuter the pnpm leg. LEG 4 restored -> 177/177 green with `diff` byte-identical to pristine. Each leg produced a DISTINCT, PREDICTED red set, which rules out the 'both legs errored identically so the diff read as no breakage' failure mode. (7) REBUILD QUESTION, MEASURED NOT ASSUMED: no rebuild is involved on either leg — the pin spawns `tsx src/index.ts` and BUNDLED_TEMPLATES_DIR resolves to src/templates, so nothing here reads dist/; the canary below is the empirical proof a source edit reaches the spawned child. (8) CANARY CONTROL (instrument proven to produce a positive before its negative was trusted): warning marker replaced with CANARY9137, confirmed on disk both directions (real marker 1->0, canary 0->1); the run reported it exactly where expected at src/scaffold-next-steps-pm.test.ts:238 — \"AssertionError: expected ... to contain 'using npm as a fallback'\" with the diff line showing the CHILD'S REAL STDOUT: '+ ⚠ pnpm is installed but `pnpm --version` failed (corepack: Network request to https://registry.npmjs.org/pnpm failed); using npm as a CANARY9137.' That one line proves the assertion reads real output rather than mere absence, that the source mutation reaches the spawned process, and that the whole detail chain works. (9) GREEN IN BOTH STATES — named as regression guards, NOT counted as red-before evidence: 'regression guard: pm is pnpm iff the probe succeeded' (the Clause ② guard), 'pnpm simply absent: NO fallback warning' (green in B1 only because B1 collapsed toward 'absent'; it is the pin that catches a collapse in the other direction), 'the PATH lookup is not consulted when the probe succeeds', 'a succeeding probe never claims a fallback', and the pre-existing install-failure-remedy source pin.",
  "gates": "Family re-derived from the ACTUAL change set with `node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack`, and AGAIN after the changeset grew it to 5 paths — 13 matched families, all run, each quoted by its own verdict line: check:published-files '✓ 69 publishable package(s) ... admits no test, test-harness config or build script'; check:slot-lookup '✓ slot-lookup ratchet holds: 107 unswept site(s) ... none new'; check:test-source-alias 'check-test-source-alias OK — 72 packages with tests scanned'; check:type-source-resolution 'check-type-source-resolution OK — 93 tsc program(s) across 77 packages'; check-plugin-teardown-shape '✓ 63 Plugin implementation(s) across 4650 source(s)'; check:nul-bytes 'OK (scanned 6670 text file(s) ... no raw ASCII control bytes)'; check:query-options-erasure '✓ ratchet holds: 67 unswept non-test site(s), none new'; check:engine-double-contract 'OK — 405 pinned, 133 in the DEBT ledger, 2 exempt'; check:cross-package-test-inputs 'OK: 16 package(s) read outside themselves, all declared'; check:where-matcher '✓ 297 matcher(s) discovered, 297 answer ... correctly; none new'; check:type-check-coverage 'OK — 65/78 workspace packages type-checked'; check:changeset-gate-self-tests '✓' (118+212+116 assertions); check:objectui-changeset '✓ objectui-range --self-test: all checks passed'; check-adr-0087-registration '✓ this PR adds no declared-breaking changeset'; check-changeset-no-major '✓ This diff introduces no `major` bump'; check-empty-changeset '✓ No empty-frontmatter changeset (1 declaring changeset(s) added)'; release-rehearsal-clone --self-test '✓ self-test passed'; check-affected-docs exit 0; check-drift-comment '✓ 56 cases pass across 5 fixture diff(s)'. Every exit code captured BEFORE any pipe (redirect-then-capture), never a bare $? after a tail. NOT RUN, DECLARED: check:type-check-debt — its ratchet half needs the whole workspace closure built, and create-objectstack carries NO ledger row it could move (no test-typecheck-debt.json, no DEBT entry, reported among the 65/78 fully covered), both new files are inside the green tsc program; CI runs the ratchet regardless.",
  "declared_narrowing": "The ablation legs ran OUTSIDE the shared verify lock, declared rather than silent. Two consecutive acquisitions returned 'VERDICT queue-timeout (exit 99) · never acquired · waited 540s (9m00s)' — ~18 minutes with no progress, behind another lane's multi-minute turbo build (holder pid 21252 at 576s, and earlier pid 16923 running objectstack-11772's cli build). The narrowing is measured, not asserted: create-objectstack has ZERO workspace dependencies (no build closure at all) and the earlier SUPERSET run — all 14 files, 177 tests — held the lock 17 seconds; the whole 5-leg matrix took 111s of single-package vitest. Per the queue clause I spent both waits on lock-free work (changeset, PR body, gate sweep, package typecheck) rather than ending a round on a wait. The full-repo pnpm lint was NOT narrowed — it ran whole, exit 0.",
  "changeset": ".changeset/create-objectstack-honest-pm-probe.md, patch — REQUIRED rather than skip-changeset because the product's reporting changes: a user whose pnpm probe fails now gets a warning naming the failure instead of silent npm advice. check-empty-changeset confirms '1 declaring changeset(s) added'.",
  "file_surface": "Declared surface was the detectPackageManager seam + scaffold-next-steps-pm.test.ts + a changeset. TWO FILES ADDED to that surface, both new and therefore held by no other claim: packages/create-objectstack/src/detect-package-manager.ts (the seam, extracted so the probe can be injected) and packages/create-objectstack/src/detect-package-manager.test.ts (its hermetic pins). Sibling claims respected: packages/cli/src/commands/init.ts (#11598) and compile.ts (#11772) were READ ONLY, never edited — init.ts's npm_config_user_agent detector answers a different question ('which PM invoked me') and was deliberately NOT adopted, since switching strategies would change what the tool does. TEST_DEBT / type-check-debt ledger (#11788) untouched. PR left as draft; auto-merge not enabled.",
  "open_questions": [],
  "out_of_scope_findings": [
    "filed as #12042 (observation-class, `finding` label, unassigned, no domain/type): `create-objectstack` declares only --skip-install and --skip-skills (index.ts:383-384) and has no --package-manager override, so its probe is the only input — while `objectstack init` next door takes the flag and falls back to detection only when it is absent (init.ts:711 declares, init.ts:868 consumes). Surfaced because the new warning names a situation the user has no flag to answer. Filed as an observation rather than queued work: the two detectors deliberately answer different questions (invoked-by vs available), a user running `npm create objectstack` has an npm user-agent while possibly preferring pnpm, so the sibling's strategy cannot simply be adopted, and adding a flag is capability expansion with no measured pull."
  ]
}

Generated by Claude Code

fix(create-objectstack): make the package-manager verdict hermetic an…

Navigation Menu

Repository navigation

[finding] `scaffold-next-steps-pm.test.ts` pins pnpm detection over a probe that resolves OUTSIDE the repo, so a Corepack fetch decides the verdict — one queue red already #11616

What went red

Why it is not a real regression, and why that is the problem

Suggested shape (not a ruling — for triage)

The defect, and ⛔ the fix is not "retry it"

Anti-vacuity — sharper than usual here, because the subject IS a vacuous pin

Standing repo rules

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[finding] scaffold-next-steps-pm.test.ts pins pnpm detection over a probe that resolves OUTSIDE the repo, so a Corepack fetch decides the verdict — one queue red already #11616

Description

What went red

Why it is not a real regression, and why that is the problem

Suggested shape (not a ruling — for triage)

Activity

claude commented on Aug 25, 2026

os-zhuang commented on Aug 25, 2026

The defect, and ⛔ the fix is not "retry it"

Anti-vacuity — sharper than usual here, because the subject IS a vacuous pin

Standing repo rules

claude commented on Aug 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

[finding] `scaffold-next-steps-pm.test.ts` pins pnpm detection over a probe that resolves OUTSIDE the repo, so a Corepack fetch decides the verdict — one queue red already #11616