ci(test-core): publish the slowest test files and packages beside their pinned weights on every run

[baozhoutao]

Fixes #16454

Clause-②: no — the diff adds a scripts/ tool and ci.yml steps. No schema key, no closed-set member, no registry entry, and no published export: the root manifest is private: true and scripts/ is in no package's files[], so the one new export (attributedLines) crosses no published surface.

Every Test Core run now publishes the slowest 20 test files and slowest 10 packages to the job summary, each package's measured seconds beside its pinned weight from scripts/test-shard-timings.json, plus a merged artifact at 14-day retention. No test file is changed, added or removed.

The measurement that chose the route

The card deliberately did not mandate a capture route, because nobody had checked whether vitest file lines carry a duration at all — this repo sets no slowTestThreshold anywhere (measured: 0 occurrences outside node_modules; control: the same grep hits 3 files inside it).

Route (a), parse the stream. The number: 7 of 7. On a real repo suite (@objectstack/sdui-parser) under a CI-shaped environment, 7 of 7 file lines carried a duration — including a 3 ms one, far under the 300 ms default threshold. The mechanism agrees: BaseReporter.getModuleLog calls getDurationPrefix, which returns empty only when the module has no result.duration; slowTestThreshold picks the colour, never the presence.

Confirmed on real CI rather than only locally — run 34317273493, Test Core (1/6), job 102356069873:

@objectstack/verify:test:  ✓ src/harness.posture-only.test.ts (5 tests) 3651ms

So no second test run and no extra reporter is added to the shard's 30-minute wall.

Hazards found here that the brief did not name

This is what I swept, not a complete list. I swept the vitest 4.1.11 reporter selection and module-line rendering, the real CI log shapes for two packages, the .gitignore interaction, and the gate families scripts/pm/dispatch-gates.mjs derives from this diff. I did not sweep the merge-queue or push-event shapes of these steps, and I did not re-derive the brief's hazards 3 and 6-9 line by line against origin/main.

A. .gitignore swallowed the new script, and every local gate stayed green. .gitignore:91 ignores test-*.mjs anywhere in the tree. The first commit therefore landed without the script while git add -A reported success and all 56 derived gates passed against the on-disk copy. In CI the file would not exist — and because the step that runs it is deliberately continue-on-error, that would have surfaced as a permanently empty table, never as a red. Fixed by naming the script scripts/report-test-timings.mjs; the file is now tracked.

B. Measuring vitest locally inside an AI agent's container measures a reporter CI never uses. vitest 4 picks its default reporter with resolved.reporters.push([isAgent ? 'agent' : 'default', {}]), and std-env's isAgent reads AI_AGENT, CLAUDECODE, CLAUDE_CODE and friends. This container sets both CLAUDECODE=1 and AI_AGENT=claude-code_..._agent, so a local run silently selects the agent reporter, which prints no passing file lines at all. Measured on one 3-file suite: 1 file line under agent, 3 under default — same command, the only difference being env -u AI_AGENT -u CLAUDECODE. Anyone answering the card's route question from a local run would have concluded route (a) was impossible. The refusal text names this, because it is also the most likely way this table silently empties later.

C. ANSI escapes land between the digits and ms. CI runs with colour on, and the duration renders as ESC[32m 88ESC[2mmsESC[22mESC[39m. A /(\d+)ms/ over a raw log matches nothing. Stripping is shared with the sibling guard, not re-spelled.

D. packages/cli file lines carry a vitest project label, and cli is the package the drift column exists for. getEntityPrefix inserts the project name between the state symbol and the path; with colour on it renders as a background-coloured unit — after stripping, a bare word with no delimiter. Real line from the run above:

@objectstack/cli:test:  ✓  unit  test/osplugin.test.ts (6 tests) 88ms

A parser that reads the token after the symbol as the filename loses every cli row — 114 of them in that run. The path is therefore taken as the last token before the (N tests) group that looks like a test file, never as a fixed position.

Both new hazards from the PM's line-number correction are handled

• The slice leg's --only changes which tasks land in that leg's summary, so per-package seconds are accumulated across all summaries found rather than assuming one shape.
• The dependency-closure build leg streams under the same flag. Lines are dropped on the task name (test / test:repo), not on the package prefix, so build output cannot be ingested as test results — pinned in the self-test. (That leg is not in test-core.log today, which concatenates only the test legs; the property holds anyway.)

Report-only, and why every added step is guarded

Test Core is the required branch-protection context and the card says report only, no red.

• The two shard steps sit above the attestation pair, exactly like the run-summary upload, and carry if: always() + continue-on-error: true. Unguarded, a diagnostics fault would abort the job before the attestation pair ran and evict a green shard from the merge queue — the measured [finding] Merge-queue build for PR #14433 evicted with every test green — actions/upload-artifact FinalizeArtifact returned a non-retryable 403 on two shards at the same second, and the fail-closed attestation gate read them as missing #14469 failure. check:shard-attestation still reports the pair as the job's last two steps, unguarded.
• The three aggregator steps sit after Verify test shard results with the same two flags, so they can neither delay, displace nor redden the verdict.
• Steps, not a new needs: — check:aggregator-roster asserts OS_AGGREGATOR_MEMBERS + OS_AGGREGATOR_NON_MEMBERS equals needs: in both directions, and the shard captures are same-run artifacts needing nothing beyond the contents: read the job already has.

⚠️ One consequence worth a maintainer's eye: inside ci.yml no step this card may add is permitted to fail anything, so the script's --self-test runs in CI (which is what check:self-test-wired requires) but its verdict is visible, not enforcing. Giving it teeth means a check: step in lint.yml. I did not take that widening — it is outside the card's stated scope of "ci.yml steps plus one script".

Empty capture is NOT MEASURED, never an empty table

An empty table and a run where nothing could be parsed look identical on a summary page, and only the second needs a person. A merge that parsed nothing prints a named refusal and exits EXIT_PREREQUISITE_NOT_MET, imported from scripts/import-prerequisite.mjs rather than re-picked as the literal 3. Exit 1 stays reserved for findings; this tool never judges, so exit 1 is unreachable and pinned that way.

Note: the brief asked for prerequisiteNotMetText(...). No such export exists on origin/main — import-prerequisite.mjs exports reportPrerequisiteNotMet(importerUrl, verdict, measures), which is about a missing npm dependency, not a missing capture. The constant is imported as instructed; the wording is this tool's own.

Ablation — the controls fire

The self-test carries a deliberately naive prefix-only reference parser, and three controls assert it gets an answer wrong where the real parser gets it right. Degrading the real attribution to prefix-only, proven on disk (let pkg = group; occurrences 1 → 0, marker 1; blob 15476e45 → 7ae420db):

MUTATED_EXIT=1
Error: CONTROL 1: grouped order lost its attribution: got [], want [["@objectstack/spec","src/grouped.test.ts"]]

Restored from HEAD, blob hash back to 15476e45, git diff HEAD clean, RESTORED_EXIT=0.

Verification

• node scripts/report-test-timings.mjs --self-test → 57 cases across 4 batteries, every battery at or above its pinned floor.
• node scripts/check-test-completeness.mjs --self-test → self-test OK — its own 79-case battery is what proves the extracted attributedLines walk is behaviour-identical.
• End-to-end on a real turbo run test --summarize --log-order=stream log: 7 file timings and 1 package window captured, merged and rendered.
• All 56 gate commands derived by scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack were run. 52 exit 0; the 4 non-zero are all exit 3 = NOT MEASURED (check:dts-closure, check:dual-build-cjs-loads, check:lean-entry-closure, check:sourcemap-no-sources-content), each of which reads dist/ and says in its own words "Run pnpm build first. ⛔ This is NOT a pass: nothing was measured." This diff touches no package, so those four are unrelated to it.
• eslint . --no-inline-config --format json ran over the whole repo: 6409 files, 0 errors, 0 warnings. No narrowing was needed or claimed.

Bootstrap window

A pull_request run builds its shards from this ref, so a PR sees a complete table on its first run; there is no back-fill and no window in which a partial merge reads as a whole one. The "parsed X of Y declared" counter under the table states the coverage on every run, including the first.

Not done here, deliberately

scripts/test-shard-timings.json is read-only in this PR. The large, correct drift it will show for @objectstack/cli (458.15 s pinned against a much larger measurement) is the evidence this table exists to surface, not a defect to fix — refreshing that dataset is #16173/#16220's, explicitly sequenced after this.

---

Generated by Claude Code