test(perf): replace 20 absolute wall-clock budgets with runner-calibrated work units (#15055)

[mrveiss]

Thinking Path

autobot-backend/system_benchmarks_performance_test.py contained 20 assertions comparing
wall-clock milliseconds against a hardcoded constant. :311 (processor_startup_time < 500.0)
is the one that fired: passed on run 32837489748, failed on 32839088246 (547.2ms) and
33154304714 (538.9ms).

I confirmed the load-not-code claim rather than inheriting it. git diff --stat 591c0a080 90d88bb60 is one file, repo_tests/tests_that_return_instead_of_asserting_test.py, which that
selection deselects; the same diff scoped to the benchmark module is empty. One test flipped
pass to fail with no reachable code change.

Raising the number was not an option and neither was deleting the check. What I did instead is
the move this repository has already made twice: measure the quantity the assertion always
meant. #14930 turned an absolute RSS ceiling into an RSS delta; #13162 replaced two clock
races in this same file with the properties they were proxying (cache identity, peak
concurrency). The clock budgets had never had that treatment.

Two defects were hiding behind one symptom, and CI proved the second one.

1. The unit was wrong. A millisecond is a property of the runner. Every budget is now a
   ratio against a work unit — the cost of a fixed, deterministic slice of pure-Python work
   measured in the same process, in the same moment. A runner that is uniformly slow inflates
   numerator and denominator alike and the ratio holds.
2. The startup test was not measuring startup, and the constant had no headroom. Each
   component is now constructed once and discarded, then measured on a second construction, so
   the number excludes the one-time lazy import torch that lands on whichever test constructs
   first. My first commit asserted that this one-time import was most of the 500ms. CI
   falsified that: run 33156790797 put the WARM construction at 466.351ms. The constructor
   genuinely costs that on every instantiation, so the 500ms constant sat about 7% above
   the true per-construction cost — and a budget with 7% of headroom does not need runner
   weather to become a coin toss, it only needs a little. Raising the constant would have bought
   a green run or two and nothing more.

The 466ms is recorded as suspect, not blessed: #15054 has VisionProcessor's CLIP load raising
TypeError on the pinned transformers, so that constructor is timed on an error path. Its
budget is documented as a ceiling to ratchet down once #15054 lands.

What Changed

autobot_shared/perf_work_budget.py (new) — the unit, the calibration, the assertion and
the doctrine, following autobot_shared/live_service_probe.py, the test-support module #14930
added for the same suite. Shared rather than local so performance_benchmarks_test.py can adopt
it. assert_within_work_budget fails on three distinct conditions: over budget, nothing
measured, and calibration measured nothing.

autobot_shared/perf_work_budget_test.py (new, 8 tests) — guards on the harness itself,
deliberately unmarked so they run in the normal gate rather than the marker-excluded one.

autobot-backend/system_benchmarks_performance_test.py — all 20 sites converted. Final
budgets and the units CI reported on run 33161132835:

#	site	was	budget	CI units	margin
1	Config access	< 1.0ms	0.2	0.004	50x
2	Bulk config access (100 keys)	< 50.0ms	1.0	0.075	13x
3	Config section retrieval	< 2.0ms	0.6	0.005	120x
4	Cached config access	< 5.0ms	0.2	0.003	67x
5	Single multimodal processing	< 100.0ms	50.0	10.424	4.8x
6	Concurrent processing (10 inputs)	< 500.0ms	12.0	0.574	21x
7	Config validation	< 10.0ms	0.3	0.018	17x
8	Config validation (mean of 5)	< 15.0ms	0.2	0.009	22x
9	Env value parsing (per case, x6)	< 1.0ms	0.25	0.004–0.005	~55x
10	Env value parsing (all cases)	< 5.0ms	1.2	0.026	46x
11	Config manager startup	< 100.0ms	8.0	1.118	7.2x
12	Multimodal processor startup (:311, the flake)	< 500.0ms	600.0	114.306	see below
13	Memory manager startup	< 200.0ms	1.0	0.022	45x
14	Large config file loading	< 1000.0ms	420.0	71.212	5.9x
15	Statistics tracking (100 updates)	< 10.0ms	0.8	0.056	14x
16	Deep config access	< 5.0ms	0.2	0.002	100x
17	Bulk deep config access (50 keys)	< 25.0ms	0.6	0.031	19x
18	Setting 1000 config keys	< 500.0ms	6.0	0.478	13x
19	Reading keys with many configs	< 100.0ms	1.0	0.070	14x
20	Scaling to 50 concurrent requests	< 2000.0ms	30.0	1.644	18x

Not one hardcoded millisecond budget remains. Budgets were derived from observation, not
comfort: five local runs gave a per-site high, run 33156790797 measured the local-to-CI
conversion (Config manager startup 0.459 local vs 1.474 CI, so ~3.2x), and after the first
green run I ratcheted them down to 8x the highest observation (3x for the large steady
sites) — Memory manager startup came down from a 20-unit first-observation ceiling to 1.0,
Large config file loading from 1300 to 420, Single multimodal processing from 150 to 50.

Site 12 is the one deliberately wide budget, and the reason is in the file: it read 315.602,
177.909 and 114.306 units across three CI runs — a 2.76x spread in the ratio, not merely in
the milliseconds. Calibration cannot normalise it, because the cost is model-file and HF-cache
disk I/O and a CPU-bound yardstick does not track a disk-bound numerator. 600 is ~1.9x the worst
reading. That limitation is stated rather than hidden.

Also:

• measure_execution_time / measure_async_execution_time use time.perf_counter() instead of
  time.time() — a duration needs the monotonic clock; a wall-clock delta can go backwards
  across an NTP step.
• Every site records its measurement on every run, green ones included. The first commit
  only printed, and this selection runs -n auto, where xdist captures worker stdout, so a green
  run reported nothing — precisely how the old constants went years without being re-derived. An
  autouse fixture routes each site through record_property into the --junitxml file
  marker-tests.yml already uploads as marker-suite-reports. That artifact is where the CI
  column above comes from.
• .github/workflows/marker-tests.yml — the blocker note claimed test(perf): startup benchmarks assert absolute wall-clock budgets and fail on runner load — passed and failed on the same code minutes apart #15055 is why the cron is
  withheld; corrected to leave bug(ml): every transformers from_pretrained call passes resume_download, removed in the pinned v5 — vision and audio model loading raises TypeError #15054 and the ten-green-runs requirement. MARKER_MIN_PASSED
  deliberately left at 34: a floor is raised to what a run reports, not to a prediction.
  Editing this file is also what makes the marker suite run on this PR.

Verification

Three consecutive green marker-suite runs on the real runner, the job that produced every
recorded failure of this test:

run	head	result
33156790797	de7384457	failure — the new assertion firing at 315.602 units, which is how the 466ms was discovered
33158382218	512ba0bca	success
33160785258	a7e945a9d	success
33161132835	eac410d72	success

Regression-sensitivity mutation — the criterion that separates a fix from a disabled test.
Three genuine slowdowns injected into the code under test, each run against the final
budgets:

A — 30M iterations of real work added to MultiModalProcessor.__init__, the exact path :311
times:

[perf #15055] Multimodal processor startup: 1412.751 work units (budget 600.0) = 1567.686ms / 1.110ms unit
AssertionError: ... This is a load-invariant ratio, not a wall-clock ceiling — investigate the code, do not raise it.
1 failed

B — 200k iterations (~12.8ms, a realistic eager scan) added to ConfigManager.__init__:

[perf #15055] Config manager startup: 8.080 work units (budget 8.0) = 12.810ms / 1.585ms unit
1 failed

C — 30k iterations (~1.8ms) added to MemoryManager.__init__:

[perf #15055] Memory manager startup: 1.162 work units (budget 1.0) = 1.813ms / 1.560ms unit
1 failed

C is the important one: after the ratchet-down, a 1.8ms addition to a constructor fails,
against a clean baseline of 0.022 units. These are not order-of-magnitude-only detectors. All
three mutations were reverted by restoring pristine copies and confirming with diff -q, and
git status --porcelain was empty before committing.

Load tolerance — measured, not asserted. 25 sites x 5 full local runs while the machine was
already busy, comparing the coefficient of variation of the raw milliseconds against the
calibrated units:

site	cv of raw ms	cv of units
Concurrent processing (10 inputs)	37.9%	2.1%
Single multimodal processing	27.4%	3.8%
Scaling to 50 concurrent requests	16.6%	4.0%
Statistics tracking (100 updates)	19.5%	4.9%
Large config file loading	31.2%	10.1%
Bulk config access (100 keys)	51.6%	11.3%

No artificial load was generated — the machine's own concurrent work supplied it. Two honest
limits. For sites measuring single-digit microseconds the units are no steadier than the
milliseconds, because timer granularity rather than contention sets their spread; their
headroom is ~50x, so neither noise source can reach the budget. And site 12 is not normalised at
all, for the disk-I/O reason given above — that is why it, alone, keeps a wide budget.

Vacuity probe. The assertion cannot pass by measuring nothing, and this is a permanent test
rather than a one-off demonstration —
autobot_shared/perf_work_budget_test.py, 8 passing:

test_zero_elapsed_cannot_pass                test_budget_still_fails_when_exceeded
test_negative_elapsed_cannot_pass            test_work_unit_is_measurable_and_positive
test_absent_measurement_cannot_pass          test_reference_workload_is_deterministic
test_measurements_reach_the_recorder         test_recorder_is_cleared_after_the_block

A zero, negative or absent elapsed_ms fails with "the operation under test did not run"; a
zero calibration fails rather than being divided by; test_budget_still_fails_when_exceeded is
the counterpart guard proving the assertion is not unconditionally green; and
test_measurements_reach_the_recorder guards the reporting path the budgets are re-derived from.

What I ran locally: the benchmark module plus the harness tests — 22 passed, repeatedly;
scripts/check_python_file_size.py --audit-ceilings clean; black --line-length=120 --check
and isort --profile=black clean.

What a local pass could not tell me: this interpreter is Python 3.10.12 and pytest reports
90 of 206 declared versions unsatisfied — notably torch and sentence-transformers are
absent, so the MultiModalProcessor paths CI exercises do not run here. Every figure in the
CI column above therefore comes from the uploaded junit artifact, not from this machine. I also
could not reproduce a loaded-runner failure on demand, since generating artificial load here is
off-limits while live CI runs.

Scope kept. #15054 is still open, so the multimodal constructor remains on the
resume_download TypeError branch. This change is orthogonal to it: a ratio against a same-run
baseline is correct whichever branch the constructor takes, and the budget for that site is
explicitly labelled as a ceiling to lower once #15054 lands.

Model Used

Claude Opus 5 (1M context)

Refs #15055 — two of the issue's four acceptance criteria are outside this PR: #15054 is not
fixed here, and the "ten consecutive scheduled runs" criterion needs the cron, which is
deliberately still withheld. Four marker-suite runs are cited above.

[github-actions]

✅ SSOT Configuration Compliance: Passing

🎉 No new hardcoded values of either class — ssot and other both block.

Known backlog in pipeline-scripts/hardcoded_values_baseline.txt is suppressed and tracked in #14371.