Repository navigation
Commit 2a4a796
ci(timings): give the test-timing parser self-test teeth in lint.yml, not continue-on-error in ci.yml (#18096)
Closes #17097
Clause-②: no
## The pinned first step: the self-test was GREEN before removal
Triage required this before anything else, and it decides the path:
```
$ node scripts/report-test-timings.mjs --self-test
report-test-timings: self-test OK (61 cases across 4 batteries — file-line parsing 22/22,
attribution and the naive-parser controls 9/9, package seconds, slices and cache replays 11/11,
refusals, rendering and exit codes 19/19)
$ echo $?
0
```
Exit **0**, on `origin/main` at `66e34d14d9`, before any edit. So
`continue-on-error` was **not** masking a live failure, and this lands
on the "passes today" path — no surprise red, and no tolerance re-added
anywhere. (Card said ~57 cases; the battery is now 61. Same 4
batteries.)
## What changed, and why not the literal removal
The brief asked for `continue-on-error` to come off the `Capture this
shard's test timings` step. **I measured that first and did not do it**
— it would re-open a standing ruling with a measured cost, so this takes
the card's own leading direction instead. Reporting the divergence
plainly rather than choosing silently:
- That step also runs `--capture`, which parses **real and variable CI
logs**. Removing the key puts it on the shard's PASS/FAIL path.
- #16454 rules the timing feature **report-only**; #14469 is the
measured cost of an unguarded diagnostics step in that exact job — a
FinalizeArtifact 403 on a 313-byte upload evicted a **fully green
shard** from the merge queue.
- `ci.yml` already carried the remedy in its own comment: *"Giving that
self-test teeth means a `check:` step in lint.yml."* That is direction 1
in the card body, named there as the leading candidate because it "keeps
ci.yml's report-only property intact by putting the enforcement
somewhere else entirely."
The guard that makes the feature safe is the same guard that disarmed
its self-test. So the fix **moves the instrument** rather than weakening
the guard:
| file | change |
|---|---|
| `lint.yml` | new **`Test-timing parser self-test`** step in `Lint &
Repo Gates` — no `continue-on-error`, no `if:` — beside the
shard-partitioner self-test it is the sibling of |
| `ci.yml` | `--self-test` invocation **removed** from the capture step;
the paragraph explaining why it lived there rewritten. `--capture` /
`--merge` keep `continue-on-error` — untouched |
| `report-test-timings.mjs` | docblock claimed report-only of *"every
workflow step that runs this"* — now false, and it was the sentence that
made the capture guard read as a licence covering the self-test |
One enforcing run, not one enforcing and one decorative.
`check:self-test-wired` stays green with the invocation moved:
`lint.yml` is a workflow, and the gate credits any workflow that names
the script.
## Acceptance 1 — the verdict can turn the job red
Read back out of the parsed YAML, not out of the diff:
```
step 'Test-timing parser self-test' continue-on-error : ABSENT
step 'Test-timing parser self-test' if : ABSENT
job 'lint' continue-on-error : ABSENT
workflow lint.yml continue-on-error : ABSENT
lint.yml on.push / on.pull_request paths filter : none (runs on every PR)
```
The step is a single command, so the step's exit **is** the self-test's
exit.
## Acceptance 2 — positive control: it DOES go red
Ablation on the committed tree, degrading the real parser back to the
naive shape the controls exist to catch — `TEST_FILE` widened to `/./`,
so the project label is taken as the path (the wrongness CONTROL 3
pins).
On-disk proof taken **before** reading any run — anchor and marker each
counted, never a bare `git diff --stat`:
```
HEAD blob : f861657
before : anchor=1 marker=0
after : anchor=0 marker=1 ← mutation reached disk
mutated blob : 8d87ae409811b5341a3b9f2c742d9802eac951ec
MUTATED_SELF_TEST_EXIT = 1
Error: split: a line with no test file: got {"file":"here","project":"no path"}, want null
```
**Exit 1 ⇒ the step fails ⇒ `Lint & Repo Gates` goes red.** The
tolerance is not merely removed; it has been seen to fire.
Restore settled by **blob identity, never by an exit code** and never by
`trap` alone (#17875 — a `trap` in this container can silently not
fire):
```
restored blob : f861657 == HEAD blob
git diff HEAD : empty
```
No `dist/` leg: the workflow step runs this script from source, so there
is no build for a stale artifact to hide in.
## Acceptance 3 — negative control: still green, no new noise
```
EXIT=0
stdout+stderr lines : 1
bytes : 230
annotation-shaped tokens (line-initial :: or ##[) : 0
```
One line, the same line the step printed under `ci.yml`. Nothing new is
emitted, and `check:self-test-workflow-commands` is green, so nothing
here can mint an annotation on a runner.
## Acceptance 4 — reverse-read, including the zeros
Probe shape stated, because the key/prose distinction is the whole
point. Two independent probes:
1. **Key form, textual:** `grep -rnE
'^[[:space:]]*continue-on-error[[:space:]]*:' .github/workflows/` —
matches only a line-initial YAML key.
2. **Key form, structural:** parse every workflow with `yaml.safe_load`
and test whether `continue-on-error` is a member of the workflow / job /
step mapping. This cannot be fooled by prose at all.
Both agree: **13 keys** across 34 workflow files.
| workflow | keys | text mentions |
|---|---|---|
| `ci.yml` | 8 | 21 |
| `governed-surface-guard.yml` | 4 | 6 |
| `check-links.yml` | 1 | 3 |
| **`lint.yml`** | **0** | **4** |
| **`docs-drift-check.yml`** | **0** | **1** |
| **`release.yml`** | **0** | **1** |
| **`rerun-safety-nightly.yml`** | **0** | **1** |
| every other workflow (27 files) | **0** | **0** |
The four bolded zero rows are exactly the difference the brief flagged:
prose that *mentions* the key, in three cases specifically to say the
step deliberately does **not** carry it. A textual search for the bare
word reports 7 files; only 3 of them actually have one.
**Is any other verdict being masked? Zero — after this PR.** Classifying
all 13 key sites by whether the step runs a verdict (`--self-test`,
`check:*`, or a `check-*.mjs`):
- **0 of 13** now run a verdict. Before this PR the count was **1**:
`ci.yml` → `Capture this shard's test timings`, this card.
- 8 of the 13 are `actions/upload-artifact` /
`actions/download-artifact` — artifact IO, no verdict.
- The 2 remaining `run:` steps in `ci.yml` (`Capture…`, `Publish the
test timing table`) are the report-only timing path #16454 ruled.
- The 4 in `governed-surface-guard.yml` are pnpm setup/cache/install
only. Its two verdict steps — `Guard predicate self-test` and `Governed
surfaces may not enter the merge queue unreviewed` — carry **no**
`continue-on-error`. Correct by construction.
- `check-links.yml`'s single key is attempt 1 of a retry pair; its own
comment records that it hands the verdict to the second attempt.
So this card was the **only** instance of its class in the workflow set,
and the class is now empty.
## Reverse-read of the diff — which existing sentence goes false?
Not a zero. One present-tense bare claim rotted, and it is fixed in this
PR:
> `scripts/report-test-timings.mjs`: *"Every workflow step that runs
this carries `if: always()` and `continue-on-error: true`"*
After this change the `lint.yml` step runs it with neither. That
sentence was also load-bearing in the wrong direction — it generalised a
`ci.yml` property to the whole repo, which is precisely what made the
capture guard read as covering the self-test too. Rewritten to scope the
claim to `ci.yml` and to name the unguarded `lint.yml` leg.
The `ci.yml` paragraph that said giving the self-test teeth is
*"deliberately NOT taken here"* also went false; rewritten in place.
**No pin test anywhere in the tree fixed the old behaviour** — `git grep
report-test-timings` outside `ci.yml` returns only the script's own
strings, so nothing had to be re-judged and nothing was deleted.
## Verification
`node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` derived **63** families from the real change
set. **48 run, all green**, exit codes captured before any pipe.
Highlights:
```
node scripts/check-self-test-wired.mjs :: exit 0
node scripts/check-self-test-wired.mjs --self-test :: exit 0
node scripts/check-self-test-workflow-commands.mjs :: exit 0
node scripts/check-step-collectors.mjs :: exit 0
node scripts/check-ci-filter-parity.mjs :: exit 0
node scripts/check-aggregator-roster.mjs :: exit 0
pnpm check:workflow-step-name-quoting :: exit 0
pnpm check:workflow-status-functions :: exit 0
pnpm check:required-contexts :: exit 0
pnpm check:shard-attestation :: exit 0
pnpm check:select-gate-families :: exit 0
pnpm check:pm-dispatch-gates :: exit 0
pnpm check:nul-bytes :: exit 0
pnpm check:pm-clause2-carriers :: exit 0
node scripts/report-test-timings.mjs --self-test :: exit 0
```
**Declared narrowing** — 14 families not run locally, handed to CI:
- `pnpm check:type-check-debt` :: **exit 3**, PREREQUISITE NOT MET
(wants a build). **NOT MEASURED** — recorded as neither pass nor
finding.
- 13 build-dependent families (`dts-closure`, `dual-build-cjs-loads`,
`lean-entry-closure`, `sourcemap-no-sources-content`, `console-sha`,
`cross-package-test-inputs`, `driver-memory-census`,
`stall-guard-budget`, `stall-guard-headroom`, `type-check-coverage`, and
the rest) read package **build outputs or package source**. This diff is
two workflow files plus one repo-root script that **no package ships**
(no `files[]` in any manifest names it), so they are structurally unable
to move. CI runs the full farm.
## Changeset — measured, then judged
`skip-changeset`, applied as a **label**. Nothing published moves: the
diff is `.github/workflows/**` plus `scripts/report-test-timings.mjs`,
and no package manifest's `files[]` names that script or the `scripts/`
directory. Root `package.json` declares no `files[]`.
---
_Generated by [Claude
Code](https://claude.ai/code/session_012GKcPZbMoGq7WPzKLfRBTU)_
Co-authored-by: Claude <noreply@anthropic.com>1 parent b57671e commit 2a4a796
3 files changed
Lines changed: 51 additions & 12 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
928 | 928 | | |
929 | 929 | | |
930 | 930 | | |
931 | | - | |
932 | | - | |
933 | | - | |
934 | | - | |
935 | | - | |
936 | | - | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
937 | 942 | | |
938 | 943 | | |
939 | 944 | | |
| |||
942 | 947 | | |
943 | 948 | | |
944 | 949 | | |
945 | | - | |
946 | 950 | | |
947 | 951 | | |
948 | 952 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3997 | 3997 | | |
3998 | 3998 | | |
3999 | 3999 | | |
| 4000 | + | |
| 4001 | + | |
| 4002 | + | |
| 4003 | + | |
| 4004 | + | |
| 4005 | + | |
| 4006 | + | |
| 4007 | + | |
| 4008 | + | |
| 4009 | + | |
| 4010 | + | |
| 4011 | + | |
| 4012 | + | |
| 4013 | + | |
| 4014 | + | |
| 4015 | + | |
| 4016 | + | |
| 4017 | + | |
| 4018 | + | |
| 4019 | + | |
| 4020 | + | |
| 4021 | + | |
| 4022 | + | |
| 4023 | + | |
| 4024 | + | |
4000 | 4025 | | |
4001 | 4026 | | |
4002 | 4027 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | | - | |
22 | | - | |
23 | | - | |
24 | | - | |
25 | | - | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
26 | 36 | | |
27 | 37 | | |
28 | 38 | | |
| |||
0 commit comments