Skip to content

Commit de3eb88

Browse files
Elon Muskclaude
andauthored
fix(scripts): refuse a logless check-test-completeness run instead of reporting exit 1 (#13152)
A gate that cannot measure must not answer in the vocabulary of a verdict. Run with no turbo-test-log, check-test-completeness exited 1 and printed its usage banner -- indistinguishable at a glance from a finding, and the bare form is exactly what the gate family `scripts/pm/dispatch-gates.mjs` derives prints. Six devs on six unrelated cards each re-derived that it was an invocation error. A missing or unreadable log now answers with the repo's own PREREQUISITE NOT MET / exit 3 refusal convention (check-dual-build-cjs-loads, check-half-states), naming what it needs and saying that nothing was measured. A malformed command line keeps exit 1: a typo in the caller's own line is not an absent prerequisite, and nothing emits one. CI behaviour is unchanged: both call sites in ci.yml pass the argument and guard the file's existence first, and neither branches on the exit code. Verified by running both CI shapes before and after -- byte-identical output, same codes. The argv classification is now a pure exported function, so the exit-code contract is pinned by the self-test that already runs on every invocation rather than asserted in a comment. Claude-Session: https://claude.ai/code/session_01CPrUz21stTFhJRUirdc4yw Co-authored-by: Claude <noreply@anthropic.com>
1 parent 39d5d8a commit de3eb88

1 file changed

Lines changed: 192 additions & 29 deletions

File tree

‎scripts/check-test-completeness.mjs‎

Lines changed: 192 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,44 @@
8484
// [--scheduled <shard-packages.txt> --package-list <turbo-ls.json>]
8585
// node scripts/check-test-completeness.mjs --self-test
8686
//
87+
// ## Exit codes -- and why a REFUSAL has its own one
88+
//
89+
// 0 the log was graded and both questions above came back clean.
90+
// 1 a FINDING: a shortfall (Q1) or a silent scheduled package (Q2). A claim
91+
// about the tree. Also a malformed command line -- see the boundary below.
92+
// 3 PREREQUISITE NOT MET. There was no log to grade, so nothing was measured
93+
// and this says NOTHING about the tree. ⛔ NOT a finding.
94+
//
95+
// This gate used to answer a missing log with exit 1 plus a usage banner, and
96+
// exit 1 plus text on a gate script is, at a glance, a finding. It is not a
97+
// hypothetical cost: six devs on six unrelated cards, in ONE session, each ran
98+
// the bare form, each worked out that it was an invocation error, and each
99+
// reported it as NOT MEASURED. All six got it right -- that is the point. The
100+
// price was not a wrong verdict, it was six correct readers paying the same
101+
// re-derivation, with a seventh, less careful, one red PR away. And the bare
102+
// form is not a mistake any of them chose: the gate family
103+
// `scripts/pm/dispatch-gates.mjs` derives NAMES this script with no argument,
104+
// so the tooling hands out the invocation that cannot succeed.
105+
//
106+
// ⭐ Keep the direction straight, because this file now guards BOTH of them.
107+
// A gate that cannot measure and reports CLEAN is a vacuous pass -- and this
108+
// file already refuses that one: an empty log exits 0 saying "no vitest
109+
// summaries ... nothing to verify" instead of the bare word OK, and a shard
110+
// with ungraded packages says PARTIAL instead of OK. A gate that cannot
111+
// measure and reports RED is the mirror image, and it is what exit 3 ends
112+
// here. Both are the instrument speaking in the vocabulary of a verdict; only
113+
// the sign differs. Neither is fixed by making the message longer -- the EXIT
114+
// CODE is what a reader and a script both act on, so the refusal needs a code
115+
// of its own.
116+
//
117+
// ⛔ The boundary, deliberately: exit 3 is for a MISSING or UNREADABLE LOG --
118+
// the input this gate needs did not exist. A MALFORMED command line (an
119+
// unknown flag, half of the `--scheduled`/`--package-list` pair) stays exit 1.
120+
// That is not a prerequisite the world failed to supply, it is a typo in the
121+
// caller's own line, it names itself as one, and ⛔ nothing emits it: the
122+
// derived family prints the bare form, never a misspelled flag. So it charges
123+
// no reader the toll above.
124+
//
87125
// Reads a saved `turbo run test` log rather than wrapping vitest, so it needs no
88126
// change to the 60+ per-package vitest configs. In CI the test step tees its
89127
// output here. NOTE the tee: `cmd | tee f` reports TEE's exit status, so the
@@ -109,6 +147,12 @@ import { isEntrypoint } from './invoked-as.mjs';
109147

110148
const REPO_ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..');
111149

150+
// The exit-code contract, named rather than spelled inline, so the self-test
151+
// below pins the ACTUAL value each path returns instead of a comment about it.
152+
export const EXIT_OK = 0;
153+
export const EXIT_FINDINGS = 1;
154+
export const EXIT_PREREQUISITE_NOT_MET = 3;
155+
112156
// `@objectstack/cli:test: Tests 381 passed | 3 skipped (384)`
113157
// ^ turbo prefix (absent when vitest runs directly) ^ tallies ^ declared
114158
const SUMMARY = /^(?:(\S+?):test:)?\s*(Test Files|Tests)\s+(.+?)\s+\((\d+)\)\s*$/;
@@ -413,6 +457,118 @@ function describeFromPackageList(listPath) {
413457
};
414458
}
415459

460+
/**
461+
* The shared refusal frame, in the wording `check-dual-build-cjs-loads.mjs` and
462+
* `check-half-states.mjs` already use: what is unmet, why, the way out, and --
463+
* load-bearing -- that nothing was measured, so the exit code says nothing
464+
* about this gate's actual question.
465+
*/
466+
function prerequisiteNotMet({ headline, detail, fix }) {
467+
return {
468+
exit: EXIT_PREREQUISITE_NOT_MET,
469+
lines: [
470+
'',
471+
`check-test-completeness: PREREQUISITE NOT MET — ${headline}`,
472+
'',
473+
...detail.map((l) => (l ? ` ${l}` : '')),
474+
'',
475+
` Fix: ${fix}`,
476+
'',
477+
' Nothing was measured: this gate exited before parsing a single summary line, so',
478+
' this result says NOTHING about whether every test vitest counted actually ran,',
479+
' nor about whether every scheduled package reported.',
480+
' ⛔ It is NOT a finding, and it is not evidence that anything in the tree is wrong.',
481+
` (Exit code ${EXIT_PREREQUISITE_NOT_MET}, distinct from a finding's ${EXIT_FINDINGS} — but a pipe reports the PIPE's`,
482+
' status, so `node scripts/check-test-completeness.mjs | tail -4` reads green either',
483+
' way. Capture `echo "EXIT=$?"` BEFORE any pipe.)',
484+
],
485+
};
486+
}
487+
488+
/**
489+
* Read the command line and answer with EXACTLY ONE of: the paths to work from,
490+
* or the verdict that stops the run. Pure over argv, and exported, so the exit
491+
* codes above are PINNED by the self-test rather than asserted in prose -- the
492+
* code an invocation returns is the whole subject of this contract, and a
493+
* comment claiming "exit 3" is worth nothing if the branch says otherwise.
494+
*/
495+
export function invocationVerdict(argv) {
496+
const stop = (verdict) => ({ verdict, paths: null });
497+
let logPath = null;
498+
let scheduledPath = null;
499+
let packageListPath = null;
500+
for (let i = 0; i < argv.length; i++) {
501+
const arg = argv[i];
502+
if (arg === '--scheduled') scheduledPath = argv[++i];
503+
else if (arg === '--package-list') packageListPath = argv[++i];
504+
else if (!arg.startsWith('--') && logPath === null) logPath = arg;
505+
else {
506+
// ⛔ Stays a finding-shaped exit: a typo in the caller's own line, not a
507+
// prerequisite the world failed to supply. See the header boundary.
508+
return stop({ exit: EXIT_FINDINGS, lines: [`check-test-completeness: unrecognized argument: ${arg}`] });
509+
}
510+
}
511+
if (!logPath) {
512+
return stop(
513+
prerequisiteNotMet({
514+
headline: 'this gate grades a saved `turbo run test` log, and no log was named.',
515+
detail: [
516+
'Usage: node scripts/check-test-completeness.mjs <turbo-test-log>',
517+
' [--scheduled <shard-packages.txt> --package-list <turbo-ls.json>]',
518+
'',
519+
'The log comes from a test RUN. This gate does not run tests and cannot produce',
520+
'one. CI tees it and passes the path on every invocation (.github/workflows/',
521+
'ci.yml -- "Test completeness guard" in Test Core and in Dogfood), so this branch',
522+
'is unreachable there and CI behaviour is unchanged.',
523+
'',
524+
'⚠ Arrived here from the gate family `scripts/pm/dispatch-gates.mjs` derives?',
525+
'That list names this script with NO argument, which is this branch. There is no',
526+
'local log to hand it, so the local reading for this gate is NOT MEASURED.',
527+
'⛔ It is not a red, and there is nothing here to fix.',
528+
],
529+
fix: 'pass a saved `turbo run test` log — or, running the family locally, record this gate as NOT MEASURED.',
530+
}),
531+
);
532+
}
533+
if (Boolean(scheduledPath) !== Boolean(packageListPath)) {
534+
// ⛔ Also stays exit 1, and for the same reason as the unknown flag above:
535+
// half a flag pair is a malformed line, not an absent prerequisite.
536+
return stop({
537+
exit: EXIT_FINDINGS,
538+
lines: [
539+
'check-test-completeness: --scheduled and --package-list go together -- the scheduled ' +
540+
'list carries names only, and the turbo ls document is what resolves each to a directory.',
541+
],
542+
});
543+
}
544+
return { verdict: null, paths: { logPath, scheduledPath, packageListPath } };
545+
}
546+
547+
/**
548+
* A log path was named and could not be read -- absent, a directory, permission
549+
* denied. Same class as naming none: this gate obtained no log, so it measured
550+
* nothing. ⛔ Not a finding.
551+
*/
552+
export function unreadableLogVerdict(logPath, message) {
553+
return prerequisiteNotMet({
554+
headline: `the log named on the command line could not be read: ${logPath}`,
555+
detail: [
556+
`Reason: ${message}`,
557+
'',
558+
'A named-but-unreadable log and no log at all are the same fact for this gate:',
559+
'it has nothing to grade. CI checks the file exists before invoking this script',
560+
'(`[ ! -f "$RUNNER_TEMP/..." ]`), so this branch is not reached there either.',
561+
],
562+
fix: 'point the argument at a saved `turbo run test` log that exists and is readable.',
563+
});
564+
}
565+
566+
/** Print a verdict from either classifier above and leave with its own code. */
567+
function reportVerdict(verdict) {
568+
for (const line of verdict.lines) console.error(line);
569+
process.exit(verdict.exit);
570+
}
571+
416572
function selfTest({ quiet = false } = {}) {
417573
const eq = (actual, expected, what) => {
418574
const a = JSON.stringify(actual);
@@ -708,6 +864,38 @@ function selfTest({ quiet = false } = {}) {
708864
'backstop: a reported failure was charged as silent',
709865
);
710866

867+
// -- The invocation contract: a refusal must not wear a verdict's code. ----
868+
// ⛔ This block exists to make the regression LOUD, not to describe it. The
869+
// defect it pins was measured six times in one session: the bare invocation
870+
// -- the one the derived gate family prints -- answered exit 1 plus a usage
871+
// banner, which reads as a finding. If a later edit sends it back to exit 1,
872+
// these cases red on the very next invocation of this script.
873+
eq(invocationVerdict([]).verdict.exit, EXIT_PREREQUISITE_NOT_MET, 'bare invocation: not the refusal code');
874+
eq(invocationVerdict([]).verdict.exit === EXIT_FINDINGS, false, 'bare invocation: regressed to a finding code');
875+
eq(invocationVerdict([]).paths, null, 'bare invocation: handed back paths to work from anyway');
876+
eq(invocationVerdict([]).verdict.lines.some((l) => l.includes('PREREQUISITE NOT MET')), true, 'bare invocation: no refusal headline');
877+
eq(invocationVerdict([]).verdict.lines.some((l) => l.includes('Nothing was measured')), true, 'bare invocation: does not say nothing was measured');
878+
eq(invocationVerdict([]).verdict.lines.some((l) => l.includes('NOT a finding')), true, 'bare invocation: does not say it is not a finding');
879+
// ⭐ The DUAL, and it is what makes the four assertions above readings rather
880+
// than a classifier that can only ever refuse: the same function returns
881+
// "proceed" for a well-formed line, and returns the path it was given.
882+
eq(invocationVerdict(['t.log']).verdict, null, 'well-formed invocation: refused a usable command line');
883+
eq(invocationVerdict(['t.log']).paths.logPath, 't.log', 'well-formed invocation: lost the log path');
884+
eq(invocationVerdict(['t.log', '--scheduled', 's.txt', '--package-list', 'p.json']).paths.packageListPath, 'p.json', 'well-formed invocation: lost a flag value');
885+
// A named log that cannot be read is the same fact as no log at all.
886+
eq(unreadableLogVerdict('/nope.log', 'ENOENT').exit, EXIT_PREREQUISITE_NOT_MET, 'unreadable log: not the refusal code');
887+
eq(unreadableLogVerdict('/nope.log', 'ENOENT').lines.some((l) => l.includes('NOT a finding')), true, 'unreadable log: does not say it is not a finding');
888+
// ⛔ The deliberate boundary, pinned so it is a decision and not a leftover:
889+
// a malformed command line is NOT a prerequisite and keeps exit 1.
890+
eq(invocationVerdict(['--bogus']).verdict.exit, EXIT_FINDINGS, 'unrecognized argument: exit code moved');
891+
eq(invocationVerdict(['t.log', '--scheduled', 's.txt']).verdict.exit, EXIT_FINDINGS, 'half a flag pair: exit code moved');
892+
// If the codes ever collide the whole distinction is decorative.
893+
eq(
894+
EXIT_PREREQUISITE_NOT_MET !== EXIT_FINDINGS && EXIT_PREREQUISITE_NOT_MET !== EXIT_OK,
895+
true,
896+
'exit codes: the refusal code collides with a verdict code',
897+
);
898+
711899
if (!quiet) console.log('check-test-completeness: self-test OK');
712900
}
713901

@@ -720,40 +908,15 @@ function main() {
720908
// Every invocation, not a lint step -- see the header note on why.
721909
selfTest({ quiet: true });
722910

723-
let logPath = null;
724-
let scheduledPath = null;
725-
let packageListPath = null;
726-
for (let i = 0; i < argv.length; i++) {
727-
const arg = argv[i];
728-
if (arg === '--scheduled') scheduledPath = argv[++i];
729-
else if (arg === '--package-list') packageListPath = argv[++i];
730-
else if (!arg.startsWith('--') && logPath === null) logPath = arg;
731-
else {
732-
console.error(`check-test-completeness: unrecognized argument: ${arg}`);
733-
process.exit(1);
734-
}
735-
}
736-
if (!logPath) {
737-
console.error(
738-
'check-test-completeness: usage: check-test-completeness.mjs <turbo-test-log> ' +
739-
'[--scheduled <shard-packages.txt> --package-list <turbo-ls.json>]',
740-
);
741-
process.exit(1);
742-
}
743-
if (Boolean(scheduledPath) !== Boolean(packageListPath)) {
744-
console.error(
745-
'check-test-completeness: --scheduled and --package-list go together -- the scheduled ' +
746-
'list carries names only, and the turbo ls document is what resolves each to a directory.',
747-
);
748-
process.exit(1);
749-
}
911+
const { verdict, paths } = invocationVerdict(argv);
912+
if (verdict) reportVerdict(verdict);
913+
const { logPath, scheduledPath, packageListPath } = paths;
750914

751915
let raw;
752916
try {
753917
raw = readFileSync(logPath, 'utf8');
754918
} catch (err) {
755-
console.error(`check-test-completeness: cannot read ${logPath} -- ${err.message}`);
756-
process.exit(1);
919+
reportVerdict(unreadableLogVerdict(logPath, err.message));
757920
}
758921

759922
const text = stripAnsi(raw);

0 commit comments

Comments
 (0)