Skip to content

Commit eede467

Browse files
ci: Test Core's two timing gates exit-grade on schedule alone (C2), with a 20s ceiling floor (#22973)
Fixes #22949 Clause-②: no (internal CI tooling; no published surface) Refs #22966 Refs #22968 Implements ruling [6119186511](#22949 (comment)) (letter C2 for both timing steps, plus an absolute floor; the maintainer's word 「当然短期先尽快修复」): **a timing gate exit-grades only on the run profile its baseline was measured on; everywhere else it reports.** #22966 (the PR-time drift card) is the drift half of the same ruling and remains open for triage. #22968 (the hourly card) remains open until an uneventful green hourly run after this lands. ## What changes | Step (`ci.yml`) | `schedule` | `merge_group`, `pull_request`, `push`, `workflow_dispatch` | |:--|:--|:--| | `Test Core` → "Check the per-package suite-duration ceilings" | unchanged: a package over its ceiling exits 1 with an `::error::` annotation | the same table prints, each package over its ceiling is a `::warning::` annotation, exit 0 | | `Test Core (N/6)` → "Check this shard's timing drift" | unchanged: DRIFT past 1.5x exits 1 (and withholds the shard attestation) | the same verdict text prints, the DRIFT is one `::warning::` annotation, exit 0 | - **Absolute floor (ceiling step only):** a package whose effective ceiling is under 20 s (`CEILING_FLOOR_SECONDS`) is never red on any event and prints `below floor`. A ceiling at the floor is graded; a ruled raise that lifts a ceiling to the floor or above grades the package again. - **Unchanged:** the formula (window max x 1.25), `RULED_CEILING_RAISES`, the Q2 B rule for new packages, the dataset and the refresh lane. No edit to `scripts/test-shard-timings.json`, `scripts/measure-test-shard-timings.mjs` or `.github/workflows/shard-timings-refresh.yml`. - **Refusals stay red on every event.** A malformed raise record or ceiling table, or a missing `--event`, is not a timing reading and does not vary with runner load, so it exits 1 everywhere. The raise record is already checked at PR time by lint.yml's `--self-test` ("the live record is well-formed"). This is my reading of the ruling's scope (it names "every package over its ceiling or shard over its drift bound"); if the seat reads it as "the step never exits 1 off schedule", the change is small and local to the two render functions. - **`workflow_dispatch` reports.** The ruling names `schedule` as the one exit-graded event. The refresh lane reads `event=schedule` runs only (`DEFAULT_RUN_EVENT` in `scripts/ci/select-shard-timings-run.mjs`, no `--event` override in `shard-timings-refresh.yml`), and a dispatched run writes the remote turbo cache (`TURBO_CACHE` `rw`) where the hourly run only reads it, so it is not the baseline's profile. ## H2: how the off-schedule path is spelled Both scripts take a **required** `--event`, and ci.yml passes `--event "$GITHUB_EVENT_NAME"`. The rule (`TIMING_GRADED_EVENT = 'schedule'`, `timingGateMode`, `timingModeNote`) is defined once, in `scripts/partition-test-shards.mjs`, and imported by `scripts/check-test-suite-ceilings.mjs` (which already imported `escapeWorkflowCommandMessage` from there). - Why not `continue-on-error` (the `ci.yml:1204` spelling): it leaves the `::error::` annotations in place and does not produce the `::warning::` the ruling asks for. - Why not an opt-in `--report-only` set by a YAML expression: the policy would then live in an expression no self-test reads, and a typo in it fails silently in one direction or the other. With `--event`, the decision table is self-tested in the script, and a missing or empty event is **refused** (exit 1), never defaulted to either mode. - Neither step gains an `if:` or a `continue-on-error`. The drift step's comment block carries "NO `if:` and NO `continue-on-error` (the red is the point)"; that line still holds as written and is extended, not removed. The other ⛔ lines H1 named (`ci.yml:1036`-`1037` and `:1218`) bind the attestation pair, which this PR does not touch. - The drift verdict was measured first: in graded mode DRIFT prints to stderr with **no** annotation (pinned by the existing self-test), so `partition-test-shards.mjs --check-drift` needed the same flag for a warning to exist at all. ## The floor: 20 s, and its reading Read by the `domain:devx` seat off the table's distribution, as the ruling assigns: on `1f312fdfa` (unchanged at this PR's base `246edf5b3`) the 72 ceilings run 3.40, 3.69, 4.24, 5.18, 6.39, 9.10 and 16.23 s, then 22.71 s (`downstream-contract`) and up. 20 s sits in the widest gap of the low band (16.23 to 22.71). Seven packages fall under it: `knowledge-ragflow`, `service-cache`, `observability`, `sdui-parser`, `knowledge-memory`, `embedder-openai`, `client-react`. The value, its reading and the ruling URL are recorded once, in the gate header ("The absolute floor"). ## H5: packages the hourly run never grades The hourly run grades only the packages it executes; a turbo cache replay carries no window. I read the "How much of the run this table saw" block of six hourly `schedule` runs (MCP `get_job_logs` on each `Test Core` job; the 04:09Z set is the complement of that run's graded rows against the 72-package dataset, since the tail read missed its block): | Run | Time (2026-10-12) | Replayed packages | |:--|:--|--:| | `38192273698` | 01:32Z | 12 | | `38195222269` | 02:17Z | 5 | | `38198819852` | 03:12Z | 27 | | `38202645321` | 04:09Z | 14 | | `38206660804` | 05:08Z | 15 | | `38210849670` | 06:05Z | 11 | - **Replayed on every one of the six runs: none.** The 02:17Z run replayed only `formula`, `metadata-fs`, `observability`, `sdui-parser` and `spec`. - **Replayed on all three runs since the table went live (04:09Z, 05:08Z, 06:05Z): `core`, `metadata`, `plugin-hono-server`.** Each was also replayed at 01:32Z and 03:12Z (5 of 6); the one run that executed them (02:17Z) predates the ceiling table, so none of the three has been graded yet. `formula`, `metadata-fs` and `spec` were replayed on 4-5 of the 6. - **Handling, as the ruling leaves it:** graded on any hourly run that executes them, report-only everywhere else. No mechanism forces them to execute (not in the ruling); I do not think one is needed yet, because the replay set rotates run to run, and the gate header now states the blind spot. ## Parity checkers None was touched; none needed an edit. Run green on the head unchanged: `check-self-test-workflow-commands`, `check-step-collectors`, `check-workflow-status-functions`, `check-required-contexts`, `check-aggregator-roster`, `check-ratchet-remedy-authority`, `check-stall-guard-budget`, `check-stall-guard-headroom`, `check-workflow-step-name-quoting`, `check-shard-attestation`. H3 ("expect at least one to need a matching edit") did not hold. ## Evidence (head `ef3d0764b`) **Ceiling CLI, one synthetic capture** (`cloud-connection` 200.33 s vs 163.89 s, `downstream-contract` 25.00 s vs 22.71 s, `knowledge-ragflow` 4.80 s vs 3.40 s, `client` 150.00 s vs 179.74 s, against the live dataset): ```text --event schedule EXIT=1 ::error title=Suite-duration ceiling::@objectstack/cloud-connection: 200.33s, over its 163.89s ceiling. ... ::error title=Suite-duration ceiling::@objectstack/downstream-contract: 25.00s, over its 22.71s ceiling. ... -- @objectstack/knowledge-ragflow: 4.80s against 3.40s (1.41x) -- below floor: a ceiling under 20s is never red exit-graded: a `schedule` run is the profile the baseline was measured on. --event merge_group EXIT=0 ::warning title=Suite-duration ceiling::@objectstack/cloud-connection: 200.33s, over its 163.89s ceiling. ... Report only on a `merge_group` run: ... ::warning title=Suite-duration ceiling::@objectstack/downstream-contract: 25.00s, over its 22.71s ceiling. ... Report only on a `merge_group` run: ... (the same stderr table, line for line, ending) report only: a `merge_group` run is not the profile the baseline was measured on (`schedule`), so a red reading is a warning here and the step exits 0. no --event EXIT=1 Error: the run's event is null. ... A missing event is refused rather than guessed. ``` **Drift CLI, one synthetic turbo summary** (`runtime` 916.1 s, `service-analytics` 463.7 s, `client` 183.5 s against the live weights = 1.82x): ```text --event schedule EXIT=1 stdout empty; stderr "shard-timing-drift: DRIFT -- Test Core (6/6), 1563.3s measured vs 860.2s predicted ... = 1.82x" + "exit-graded: ..." --event pull_request EXIT=0 one "::warning title=Test Core shard timing drift::Test Core (6/6): ... past the red bound. ... Report only on a `pull_request` run: ..." + the same stderr text ending "report only: ..." no --event EXIT=1 Error: the run's event is null. ... ``` **Self-tests:** `check-test-suite-ceilings --self-test` OK, 58 cases across 9 batteries (two new: the floor, 8 cases; off-`schedule` reporting, 8 cases; roster floor 7 to 9). `partition-test-shards --self-test` OK (one new battery, 8 cases; roster floor 12 to 13). **Ablations** (each through `scripts/ablation-replace.mjs`, anchor hit 1 to 0, restored to the HEAD blob, `git diff HEAD` empty after): - floor removed (the comparison against `CEILING_FLOOR_SECONDS` in `gradeCeilings` rewritten to compare against 0): ceiling self-test red, `floor: a 3.40s ceiling run at 4.20s read over / OVER, exit 1`. - `timingGateMode` always `'grade'`: partition self-test red (`C2: merge_group, pull_request, push, workflow_dispatch exit-grade, but only schedule may`) and ceiling self-test red (`floor: a 22.71s ceiling run at 25s off schedule was not a warning alone: exit 1`). **Gates:** `dispatch-gates --commands` on this diff derives 59 commands (the same 59 as at dispatch); `--ran` reconciliation: 59 accounted, 55 exit 0, 4 NOT MEASURED (exit 3, PREREQUISITE NOT MET: `check:dts-closure`, `check:dual-build-cjs-loads`, `check:lean-entry-closure`, `check:sourcemap-no-sources-content` read built `dist/`; this diff touches no package, so no build is owed and CI runs them). `pnpm check:pm-dispatch-gates`: fast tier 1828 cases, exit 0. The 9 workflow-valued families (the ceiling step's own `--captures` run among them) are CI's. **Lint, narrowed:** eslint over the two edited scripts, 2 files, 0 errors, 0 warnings (`--format json`). Population: `eslint.config.mjs` lints `**/*.{ts,tsx,mts,cts,js,jsx,mjs,cjs}`, so both are in it; ci.yml is not eslint's. Invariance: the config enables no type-aware linting (`--print-config` shows `parserOptions` without `project`), so this diff cannot move a verdict on any untouched file. ## Acceptance notes - Size: 502 added / 84 removed lines over 3 files; about half the additions are the three new self-test batteries, most of the rest is header and step-comment prose. The shape is the one H6 listed (two step edits, one flag in two scripts, a header, self-test cases). - `ci.yml`'s comment above "Download this run's shard test timings" still says the ceiling check "is the one step here that DOES red the job". It remains true on `schedule`; I left it as is because the file surface is the two timing steps. Carrier: none. --- _Generated by [Claude Code](https://claude.ai/code/session_012yK1ddAdAqfzwigRHZq3r2)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 712361b commit eede467

3 files changed

Lines changed: 502 additions & 84 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 36 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1066,17 +1066,27 @@ jobs:
10661066
# second red on wall time against `timeout-minutes`. Both constants, their
10671067
# measured basis and a self-test battery each live in the script.
10681068
#
1069+
# THE RED IS A `schedule` RED ALONE (C2, both timing steps, ruling
1070+
# https://github.com/objectstack-ai/objectstack/issues/22949#issuecomment-6119186511):
1071+
# the dataset's weights are measured on the hourly `schedule` runs, so
1072+
# only those exit 1 on DRIFT. On `merge_group`, `pull_request`, `push`
1073+
# and `workflow_dispatch` the step prints the same verdict, raises the
1074+
# DRIFT as a `::warning::` annotation and exits 0. The event reaches the
1075+
# script as `--event "$GITHUB_EVENT_NAME"`, which it requires; the rule
1076+
# and its self-test live there (TIMING_GRADED_EVENT, timingGateMode).
1077+
#
10691078
# EXECUTED WINDOWS ONLY. Measurements come from samplesFromSummary() in
10701079
# measure-test-shard-timings.mjs, which drops cache HITs and failed
10711080
# suites, and the prediction is summed over the same executed packages.
10721081
# A replay's near-zero window would read as a shard far FASTER than
10731082
# predicted and vouch for a rotted dataset, so a shard whose packages all
10741083
# replayed is NOT MEASURED, never "fast".
10751084
#
1076-
# NO `if:` and NO `continue-on-error` (the red is the point), and ABOVE
1077-
# the attestation pair for the #6082 reason documented on the upload
1078-
# above: a failed unguarded step skips that pair, so a drift red also
1079-
# withholds this shard's attestation, the fail-closed direction. A shard
1085+
# NO `if:` and NO `continue-on-error` (the red is the point, and the
1086+
# `schedule`-only red is the script's `--event`, so neither is needed),
1087+
# and ABOVE the attestation pair for the #6082 reason documented on the
1088+
# upload above: a failed unguarded step skips that pair, so a drift red
1089+
# also withholds this shard's attestation, the fail-closed direction. A shard
10801090
# with no packages runs no turbo and writes no summary; that is NOT
10811091
# MEASURED here rather than a usage error from the script.
10821092
#
@@ -1109,7 +1119,7 @@ jobs:
11091119
exit 0
11101120
fi
11111121
node scripts/partition-test-shards.mjs --check-drift "${SUMMARIES[@]}" \
1112-
--label "Test Core (${{ matrix.shard }}/6)"
1122+
--label "Test Core (${{ matrix.shard }}/6)" --event "$GITHUB_EVENT_NAME"
11131123
11141124
# Runs even when the suite failed — that is when it earns its keep. It
11151125
# answers TWO questions about a red suite, and needs both to be able to
@@ -1486,12 +1496,12 @@ jobs:
14861496
# ── The per-package suite-duration ratchet (#16468) ──────────────────
14871497
#
14881498
# A GATE, unlike the two reporting steps above: NO `continue-on-error`,
1489-
# so a package whose suite ran past its ceiling reds `Test Core`, naming
1490-
# the package and its slowest test files. The ceiling is the package's
1491-
# slowest executed run in the refresh window x 1.25, generated into
1492-
# scripts/test-shard-timings.json (`ceilings`) by the refresh lane and
1493-
# held by every later refresh; it rises only by a ruled entry in the
1494-
# script's RULED_CEILING_RAISES. A `provisional` package prints "no
1499+
# so on a `schedule` run a package whose suite ran past its ceiling reds
1500+
# `Test Core`, naming the package and its slowest test files. The
1501+
# ceiling is the package's slowest executed run in the refresh window
1502+
# x 1.25, generated into scripts/test-shard-timings.json (`ceilings`) by
1503+
# the refresh lane and held by every later refresh; it rises only by a
1504+
# ruled entry in the script's RULED_CEILING_RAISES. A `provisional` package prints "no
14951505
# ceiling: provisional" and is not red; a package absent from the dataset
14961506
# prints "no ceiling: new package" and is not red until the first refresh
14971507
# after it lands gives it a ceiling, while a package the dataset knows
@@ -1505,6 +1515,20 @@ jobs:
15051515
# samplesFromSummary, and every vitest file line); a capture that did not
15061516
# arrive leaves its packages ungraded and says so, never red.
15071517
#
1518+
# EXIT-GRADED ON `schedule` ALONE (C2, both timing steps, ruling
1519+
# https://github.com/objectstack-ai/objectstack/issues/22949#issuecomment-6119186511):
1520+
# the ceilings come from the hourly `schedule` runs, so only those red
1521+
# this job on a package over its ceiling. On `merge_group`,
1522+
# `pull_request`, `push` and `workflow_dispatch` the step prints the same
1523+
# table, raises each package over its ceiling as a `::warning::`
1524+
# annotation and exits 0; a real slowdown merged to `main` reds the next
1525+
# hourly run. A package whose ceiling is under the absolute floor (20s,
1526+
# CEILING_FLOOR_SECONDS) prints "below floor" and is never red. The
1527+
# event reaches the script as `--event "$GITHUB_EVENT_NAME"`, which it
1528+
# requires; a refusal (a malformed table or raise record) is red on every
1529+
# event. The rule, the floor's reading and their self-tests live in the
1530+
# script.
1531+
#
15081532
# NOT MEASURED, exit 0, while the dataset carries no `ceilings`: they are
15091533
# generated only where the run-summary artifacts are downloadable (the
15101534
# refresh lane), so the first refresh after this lands writes them, and
@@ -1515,7 +1539,7 @@ jobs:
15151539
# captured, so a red shard's packages are graded too.
15161540
- name: Check the per-package suite-duration ceilings
15171541
if: always()
1518-
run: node scripts/check-test-suite-ceilings.mjs --captures "$RUNNER_TEMP/test-timings" --label 'Test Core'
1542+
run: node scripts/check-test-suite-ceilings.mjs --captures "$RUNNER_TEMP/test-timings" --label 'Test Core' --event "$GITHUB_EVENT_NAME"
15191543

15201544

15211545
# ── Temporal conformance against live, non-UTC servers (ADR-0053 D-A3) ─────

0 commit comments

Comments
 (0)