Repository navigation
ci: train CI keeps finding the same mechanical defects — check them before a train, not in it #18499
Description
Problem. Every merge train on 2026-10-12 went red in CI on the same classes of defect, each repaired by hand in the train afterwards. A full CI run was spent to discover something checkable for a fraction of the cost. Approved PRs reach a train without these checks having passed on their own head, and the per-PR runs that do start (autofix-types, SAST, code-quality, verify-generated-types-slm) are almost always cancelled, so their result never exists.
Observed in this session (trains 6, 6b, 7 and 7b):
| Class | Instances |
|---|---|
| Reach floors drift | re-pinned every train: 3 / 6 / 5 / 10 floors. The local population differs from CI's (audio-extension-allowlist: about 6400 local vs 6070 CI), so a locally measured pin can fail CI. |
Stale generated api.ts |
6b (since description), 7 (#18373), 7b (#18372). A route docstring, parameter or header changed without regeneration. |
| Line-keyed records go stale | the Redis-path exemption (backup.py 524→497), the global-channel baseline (1089→1096), the docstring baseline (file moved), the slm_rest_url expected call text, the no-print count. |
| Stale baseline rows | hardcoded-values baseline: 6b (3 rows + 1 count), 7 (the audit_logger.py row). |
Acceptance criteria
- Types on demand:
auto-fix-generated-types.ymlcan be run byworkflow_dispatchon a branch, so a train fetches itsapi.tspatch with one job before opening its PR. Done in the PR that closes the sub-issue below. - Floors pinned from CI: a re-pin uses CI's measured population (CI prints the suggested mid-window value, and the merger applies it), never a local count that CI does not share.
- Records not keyed by line: line-keyed guard records (exemptions and baselines) are keyed by symbol or content instead, so moving code elsewhere in a file can't stale them.
- Per-PR fast lane: the cheap checks (generated types,
--audit-baseline, Semgrep ERROR, the reach floors) produce a non-cancelled result on each approved head before it can board a train. The merger reads that result at queue time.
Merger-side mitigation already in place: a local pre-push gate runs drift guards and floors, Semgrep, and the baseline audit on every train before push.
Activity
mrveiss commentedon Oct 12, 2026
Fifth class (added after train 7b)
autobot-backend/llc/tests/conftest.py:608buildsapi.codebase_analytics.source_storageas a baretypes.ModuleTypewith two hand-set names. fix(codebase-analytics): honest scan outcomes, duplication detectors wired, and forks shown from the one CI detector (#18013, #18067, #17982) #18362 addedget_source_or_none_if_unreadableto the real module and called it fromllc/api/sprints.py, so every llc test running through the stub failed to import, but only once both PRs met in train 7b. Build such stubs from the real module (create_autospecof the real module, or a stub that raises on a missing attribute and is checked againstdir(real)), and add a guard that fails when asys.modulesstub lacks a public name its real module defines.