Skip to content

ci: Test Core's two timing gates exit-grade on schedule alone (C2), with a 20s ceiling floor - #22973

Merged
objectstack-fleet[bot] merged 2 commits into
mainfrom
claude/issue-22949-timing-gates-schedule-only
Oct 12, 2026
Merged

objectstack-fleet[bot] merged 2 commits into
mainfrom
claude/issue-22949-timing-gates-schedule-only

Conversation

@objectstack-fleet

Copy link
Copy Markdown
Contributor

Fixes #22949
Clause-②: no (internal CI tooling; no published surface)
Refs #22966
Refs #22968

Implements ruling 6119186511 (letter C2 for both timing steps, plus an absolute floor; the maintainer's word 「当然短期先尽快修复」): a timing gate exit-grades only on the run profile its baseline was measured on; everywhere else it reports. #22966 (the PR-time drift card) is the drift half of the same ruling and remains open for triage. #22968 (the hourly card) remains open until an uneventful green hourly run after this lands.

What changes

Step (ci.yml) schedule merge_group, pull_request, push, workflow_dispatch
Test Core → "Check the per-package suite-duration ceilings" unchanged: a package over its ceiling exits 1 with an ::error:: annotation the same table prints, each package over its ceiling is a ::warning:: annotation, exit 0
Test Core (N/6) → "Check this shard's timing drift" unchanged: DRIFT past 1.5x exits 1 (and withholds the shard attestation) the same verdict text prints, the DRIFT is one ::warning:: annotation, exit 0
  • Absolute floor (ceiling step only): a package whose effective ceiling is under 20 s (CEILING_FLOOR_SECONDS) is never red on any event and prints below floor. A ceiling at the floor is graded; a ruled raise that lifts a ceiling to the floor or above grades the package again.
  • Unchanged: the formula (window max x 1.25), RULED_CEILING_RAISES, the Q2 B rule for new packages, the dataset and the refresh lane. No edit to scripts/test-shard-timings.json, scripts/measure-test-shard-timings.mjs or .github/workflows/shard-timings-refresh.yml.
  • Refusals stay red on every event. A malformed raise record or ceiling table, or a missing --event, is not a timing reading and does not vary with runner load, so it exits 1 everywhere. The raise record is already checked at PR time by lint.yml's --self-test ("the live record is well-formed"). This is my reading of the ruling's scope (it names "every package over its ceiling or shard over its drift bound"); if the seat reads it as "the step never exits 1 off schedule", the change is small and local to the two render functions.
  • workflow_dispatch reports. The ruling names schedule as the one exit-graded event. The refresh lane reads event=schedule runs only (DEFAULT_RUN_EVENT in scripts/ci/select-shard-timings-run.mjs, no --event override in shard-timings-refresh.yml), and a dispatched run writes the remote turbo cache (TURBO_CACHE rw) where the hourly run only reads it, so it is not the baseline's profile.

H2: how the off-schedule path is spelled

Both scripts take a required --event, and ci.yml passes --event "$GITHUB_EVENT_NAME". The rule (TIMING_GRADED_EVENT = 'schedule', timingGateMode, timingModeNote) is defined once, in scripts/partition-test-shards.mjs, and imported by scripts/check-test-suite-ceilings.mjs (which already imported escapeWorkflowCommandMessage from there).

  • Why not continue-on-error (the ci.yml:1204 spelling): it leaves the ::error:: annotations in place and does not produce the ::warning:: the ruling asks for.
  • Why not an opt-in --report-only set by a YAML expression: the policy would then live in an expression no self-test reads, and a typo in it fails silently in one direction or the other. With --event, the decision table is self-tested in the script, and a missing or empty event is refused (exit 1), never defaulted to either mode.
  • Neither step gains an if: or a continue-on-error. The drift step's comment block carries "NO if: and NO continue-on-error (the red is the point)"; that line still holds as written and is extended, not removed. The other ⛔ lines H1 named (ci.yml:1036-1037 and :1218) bind the attestation pair, which this PR does not touch.
  • The drift verdict was measured first: in graded mode DRIFT prints to stderr with no annotation (pinned by the existing self-test), so partition-test-shards.mjs --check-drift needed the same flag for a warning to exist at all.

The floor: 20 s, and its reading

Read by the domain:devx seat off the table's distribution, as the ruling assigns: on 1f312fdfa (unchanged at this PR's base 246edf5b3) the 72 ceilings run 3.40, 3.69, 4.24, 5.18, 6.39, 9.10 and 16.23 s, then 22.71 s (downstream-contract) and up. 20 s sits in the widest gap of the low band (16.23 to 22.71). Seven packages fall under it: knowledge-ragflow, service-cache, observability, sdui-parser, knowledge-memory, embedder-openai, client-react. The value, its reading and the ruling URL are recorded once, in the gate header ("The absolute floor").

H5: packages the hourly run never grades

The hourly run grades only the packages it executes; a turbo cache replay carries no window. I read the "How much of the run this table saw" block of six hourly schedule runs (MCP get_job_logs on each Test Core job; the 04:09Z set is the complement of that run's graded rows against the 72-package dataset, since the tail read missed its block):

Run Time (2026-10-12) Replayed packages
38192273698 01:32Z 12
38195222269 02:17Z 5
38198819852 03:12Z 27
38202645321 04:09Z 14
38206660804 05:08Z 15
38210849670 06:05Z 11
  • Replayed on every one of the six runs: none. The 02:17Z run replayed only formula, metadata-fs, observability, sdui-parser and spec.
  • Replayed on all three runs since the table went live (04:09Z, 05:08Z, 06:05Z): core, metadata, plugin-hono-server. Each was also replayed at 01:32Z and 03:12Z (5 of 6); the one run that executed them (02:17Z) predates the ceiling table, so none of the three has been graded yet. formula, metadata-fs and spec were replayed on 4-5 of the 6.
  • Handling, as the ruling leaves it: graded on any hourly run that executes them, report-only everywhere else. No mechanism forces them to execute (not in the ruling); I do not think one is needed yet, because the replay set rotates run to run, and the gate header now states the blind spot.

Parity checkers

None was touched; none needed an edit. Run green on the head unchanged: check-self-test-workflow-commands, check-step-collectors, check-workflow-status-functions, check-required-contexts, check-aggregator-roster, check-ratchet-remedy-authority, check-stall-guard-budget, check-stall-guard-headroom, check-workflow-step-name-quoting, check-shard-attestation. H3 ("expect at least one to need a matching edit") did not hold.

Evidence (head ef3d0764b)

Ceiling CLI, one synthetic capture (cloud-connection 200.33 s vs 163.89 s, downstream-contract 25.00 s vs 22.71 s, knowledge-ragflow 4.80 s vs 3.40 s, client 150.00 s vs 179.74 s, against the live dataset):

--event schedule      EXIT=1
::error title=Suite-duration ceiling::@objectstack/cloud-connection: 200.33s, over its 163.89s ceiling. ...
::error title=Suite-duration ceiling::@objectstack/downstream-contract: 25.00s, over its 22.71s ceiling. ...
  --    @objectstack/knowledge-ragflow: 4.80s against 3.40s (1.41x) -- below floor: a ceiling under 20s is never red
  exit-graded: a `schedule` run is the profile the baseline was measured on.

--event merge_group   EXIT=0
::warning title=Suite-duration ceiling::@objectstack/cloud-connection: 200.33s, over its 163.89s ceiling. ... Report only on a `merge_group` run: ...
::warning title=Suite-duration ceiling::@objectstack/downstream-contract: 25.00s, over its 22.71s ceiling. ... Report only on a `merge_group` run: ...
  (the same stderr table, line for line, ending)
  report only: a `merge_group` run is not the profile the baseline was measured on (`schedule`), so a red reading is a warning here and the step exits 0.

no --event            EXIT=1   Error: the run's event is null. ... A missing event is refused rather than guessed.

Drift CLI, one synthetic turbo summary (runtime 916.1 s, service-analytics 463.7 s, client 183.5 s against the live weights = 1.82x):

--event schedule      EXIT=1   stdout empty; stderr "shard-timing-drift: DRIFT -- Test Core (6/6), 1563.3s measured vs 860.2s predicted ... = 1.82x" + "exit-graded: ..."
--event pull_request  EXIT=0   one "::warning title=Test Core shard timing drift::Test Core (6/6): ... past the red bound. ... Report only on a `pull_request` run: ..." + the same stderr text ending "report only: ..."
no --event            EXIT=1   Error: the run's event is null. ...

Self-tests: check-test-suite-ceilings --self-test OK, 58 cases across 9 batteries (two new: the floor, 8 cases; off-schedule reporting, 8 cases; roster floor 7 to 9). partition-test-shards --self-test OK (one new battery, 8 cases; roster floor 12 to 13).

Ablations (each through scripts/ablation-replace.mjs, anchor hit 1 to 0, restored to the HEAD blob, git diff HEAD empty after):

  • floor removed (the comparison against CEILING_FLOOR_SECONDS in gradeCeilings rewritten to compare against 0): ceiling self-test red, floor: a 3.40s ceiling run at 4.20s read over / OVER, exit 1.
  • timingGateMode always 'grade': partition self-test red (C2: merge_group, pull_request, push, workflow_dispatch exit-grade, but only schedule may) and ceiling self-test red (floor: a 22.71s ceiling run at 25s off schedule was not a warning alone: exit 1).

Gates: dispatch-gates --commands on this diff derives 59 commands (the same 59 as at dispatch); --ran reconciliation: 59 accounted, 55 exit 0, 4 NOT MEASURED (exit 3, PREREQUISITE NOT MET: check:dts-closure, check:dual-build-cjs-loads, check:lean-entry-closure, check:sourcemap-no-sources-content read built dist/; this diff touches no package, so no build is owed and CI runs them). pnpm check:pm-dispatch-gates: fast tier 1828 cases, exit 0. The 9 workflow-valued families (the ceiling step's own --captures run among them) are CI's.

Lint, narrowed: eslint over the two edited scripts, 2 files, 0 errors, 0 warnings (--format json). Population: eslint.config.mjs lints **/*.{ts,tsx,mts,cts,js,jsx,mjs,cjs}, so both are in it; ci.yml is not eslint's. Invariance: the config enables no type-aware linting (--print-config shows parserOptions without project), so this diff cannot move a verdict on any untouched file.

Acceptance notes

  • Size: 502 added / 84 removed lines over 3 files; about half the additions are the three new self-test batteries, most of the rest is header and step-comment prose. The shape is the one H6 listed (two step edits, one flag in two scripts, a header, self-test cases).
  • ci.yml's comment above "Download this run's shard test timings" still says the ceiling check "is the one step here that DOES red the job". It remains true on schedule; I left it as is because the file surface is the two timing steps. Carrier: none.

Generated by Claude Code

`--check-drift` takes the run's event (`--event`), required. On `schedule`,
the profile scripts/test-shard-timings.json is measured on, a DRIFT exits 1
as before; on every other event it prints the same text, raises one
`::warning::` annotation and exits 0. A missing or empty event is refused.

Claude-Session: https://claude.ai/code/session_012yK1ddAdAqfzwigRHZq3r2
Co-authored-by: Claude <noreply@anthropic.com>
…one, with a 20s floor

The ceiling check takes the run's event (`--event`, required). On
`schedule`, the profile the ceilings were measured on, a package over its
ceiling reds the step as before; on every other event the same table
prints, each package over its ceiling is a `::warning::` annotation, and
the step exits 0. Refusals stay red on every event.

A package whose effective ceiling is under CEILING_FLOOR_SECONDS (20s) is
never red and prints "below floor"; the header records the floor's reading.

ci.yml passes `--event "$GITHUB_EVENT_NAME"` to both timing steps, which
keep no `if:` and no `continue-on-error`.

Claude-Session: https://claude.ai/code/session_012yK1ddAdAqfzwigRHZq3r2
Co-authored-by: Claude <noreply@anthropic.com>
@objectstack-fleet objectstack-fleet Bot added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Oct 12, 2026
@objectstack-fleet
objectstack-fleet Bot marked this pull request as ready for review October 12, 2026 07:56
@objectstack-fleet
objectstack-fleet Bot added this pull request to the merge queue Oct 12, 2026
Merged via the queue into main with commit eede467 Oct 12, 2026
38 checks passed
@objectstack-fleet
objectstack-fleet Bot deleted the claude/issue-22949-timing-gates-schedule-only branch October 12, 2026 08:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/cd size/l skip-changeset PR has no user-facing published change; bypasses the changeset gate

Projects

None yet

2 participants