test(perf): report a changed baseline as a state, not a regression

[mrveiss]

Thinking Path

Multimodal processor startup fails at ~1500 work units against a budget of 600. The benchmark is working correctly — but what it detected is a state change, and it is being rendered as a regression, which sends the reader hunting a slowdown that does not exist.

The evidence:

Budget 600	set at ~1.9x a worst observation of 315.602 units
Those 315.602 units	measured while VisionProcessor's CLIP load raised TypeError (#15054) — the constructor exited early and never loaded the model. The file's own docstring says so
6f9a698e51 (#15297), 2026-08-29 17:34	dropped resume_download, touching multimodal_processor/processors/vision.py — the CLIP load now succeeds
Observed since	1661.438, then 1479.762 on a re-run of the same commit

Reproducible across two runs, and causally tied to a commit that changes the exact path being timed. Nothing got slower: the constructor now does work it had been skipping by crashing.

Two changes, and why both

1. assert_within_work_budget takes calibrated_under. A budget is only meaningful together with the state it was measured in. Without that recorded, a breach caused by a fixed bug is indistinguishable from a genuine slowdown, and the failure text actively misleads:

before:  This is a load-invariant ratio, not a wall-clock ceiling —
         investigate the code, do not raise it.

after:   budget calibrated under: CLIP model actually loading, post-#15297 (6f9a698e51)
         If that state no longer holds this is a RE-BASELINE, not a regression:
         confirm which, then reset the budget and update `calibrated_under`.

The parameter is optional and the old message is unchanged when it is absent, so the other 19 assertions in the file behave exactly as before.

2. The budget goes 600 -> 2200, with the reasoning inline rather than in a commit message.

This is the direction the file explicitly forbids — "Ratchet DOWN as runs report lower — the only direction allowed" — so it deserves the argument, not a quiet edit. That rule is right whenever the baseline is valid. It cannot bind a baseline whose premise was a bug: holding the fixed code to a number measured on the crash path would mean the benchmark permanently fails for having been repaired.

2200 is ~1.3x the higher of the two post-fix readings. Ratchet down from there. The metric is disk-bound (model files, HF cache), which a CPU-shaped work unit cannot normalise — the limitation the file already records at this site — hence a wider band than its CPU-shaped neighbours.

What Changed

• autobot_shared/perf_work_budget.py — optional calibrated_under, reported on breach; docstring records why.
• autobot-backend/system_benchmarks_performance_test.py — budget re-baselined with the provenance above.

Verification

Both files parse. Old and new failure text confirmed to differ only when calibrated_under is supplied.

The real check is the next marker-tests run: the assertion should pass at ~1500 units, and if it ever breaches again the message will say which of the two situations it is.

Note this suite runs rarely — marker-tests.yml triggers on dispatch, on changes to itself, or path-matched PRs — which is why a change that landed on 2026-08-29 was not caught until a release PR exercised it.

Model Used

Claude Opus 5 (1M context)

[github-actions]

✅ SSOT Configuration Compliance: Passing

🎉 No new hardcoded values of either class — ssot and other both block.

Known backlog in pipeline-scripts/hardcoded_values_baseline.txt is suppressed and tracked in #14371.

[mrveiss]

Scaled back — the file is at its size ceiling, and I hit it

code-quality failed on my own change:

autobot-backend/system_benchmarks_performance_test.py: 623 lines (max 600)

The base file is exactly 600 lines. It is at the ceiling, so it cannot absorb a single added line — my explanatory comment added 23.

Trimming did not rescue it: the comment came down to 8 lines and the file was still 613. Nor does collapsing the call, because the one-line form with calibrated_under is 129 characters against a 120-character limit. The change is boxed in on both axes.

What ships instead

• autobot_shared/perf_work_budget.py — the calibrated_under mechanism, unchanged. That file is not near its ceiling.
• system_benchmarks_performance_test.py — a net-zero edit: the budget alone, 600.0 to 2200.0, with a # #15342 marker pointing here. 600 lines in, 600 lines out, longest line 120.

So the re-baseline lands and the mechanism exists, but this call site does not yet pass calibrated_under — there is no room for the argument.

Why not make room

The file has 125 comment lines and 5 consecutive-blank pairs, so space could be reclaimed. I did not, for two reasons: they are other authors' comments, and whitespace churn in a perf PR makes the diff harder to review than the change it carries. Neither is a good trade for four lines.

The finding underneath

A 600-line file at a 600-line ceiling is a maintenance signal. The next person to add a benchmark, or a comment explaining one, hits exactly this wall — and the failure arrives as an unrelated-looking code-quality red rather than "this file is full". That is worth addressing on its own terms, most likely by splitting the class, rather than by whoever next needs a line negotiating for it.

The full re-baseline argument stays in the PR description above; the code now points here rather than restating it.

Verification

lines: 600   (base 600, ceiling 600)
longest line: 120 chars (limit 120)
syntax OK