Repository navigation
test(perf): three calibrated budgets are drifting or disk-bound — the startup margin is 1.4x, not the 1.9x its note records #15235
Description
Activity
- added a commit that references this issue
on Sep 13, 2026 What landed today (#16404, e04f1d6)
Checked against
origin/mainat b688b3e. #16404 referenced this issue (Refs); it closed #15055, not this one. File:autobot-backend/system_benchmarks_performance_test.py.AC Evidence on origin/mainStatus 1. Re-derive Multimodal processor startup's budget from CI readings, with a matching note:44-48: the site now has no work budget ("disk-bound … a CPU work-unit ratio can't track it"). The note cites three CI readings (2327/4348/3743 units) that failed the re-based 2200. A deterministic check replaced it (:382, warm construction imports nothing).⚠️ not ticked: the budget was removed, not re-derived. Owner call whether that satisfies AC12. Decide Large config file loading(drop the disk, or record that it is disk-bound):426assert_within_work_budget(load_time, 420.0, "Large config file loading"), with no note about disk or the budget❌ 3. Decide whether test_memory_manager_performancekeeps a liveMemoryManager:269-271still constructsMemoryManager(), with no recorded reason❌ 4. A budget that stays wide carries its reason at the constant, and an issue owns its reduction :426's 420.0 has neither❌ 5. A recorded procedure for re-deriving budgets from CI history autobot_shared/perf_work_budget.py:36-49("HOW TO CHANGE A BUDGET"): every run writesperf_work_unitsto the junit XML, whichmarker-tests.ymluploads asmarker-suite-reports. "Read the number off a run, then move the budget …"⚠️ not ticked: it is a per-run procedure, with no step for collecting readings across a run historyWhat remains: AC2 to AC4, and owner calls on AC1 and AC5. Also, the
:179comment still says a disk-bound numerator is "the limitationMultimodal processor startuprecords at its budget", but that site no longer has one (:44).Branch-freeze re-verification against current
origin/main(2026-09-20). Both tracked branches (train/2026-09-13-batch1,issue-15055-startup-benchmark) are already gone from origin.Site 1,
Multimodal processor startup— resolved, but by a better fix than the AC literally asked for. Not "re-derived and lowered": the module docstring (system_benchmarks_performance_test.py:36-45) records that a re-based 2200-unit budget (#15342) still failed three later CI runs with no code change (run 33368810264/33364129751/33579228276: 2327/4348/3743 units), and concludes "a baseline ratio can't replace it either (it would time its own numerator as the baseline)." The budget assertion was removed entirely for this site and replaced with a deterministic check (test_system_startup_performance's warm-construction/no-new-imports assertion, visible at line 383) that verifies the thing actually worth verifying without trying to time something admittedly untimeable. This satisfies the AC's intent — margin no longer overstated, because there's no numeric margin claim left to overstate — better than a re-derived number would have, and it's issue-owned (#15055, #15235, #15342 all cited inline).Site 2,
Large config file loading— not done. Stillassert_within_work_budget(load_time, 420.0, "Large config file loading")at line 427, still no comment anywhere near it recording that it's disk-bound or why 420.0 is the number. AC2 and (for this site) AC4 are both unmet — this is exactly the "silence is the defect" case the issue calls out by name.Site 3,
test_memory_manager_performance— not done. Still constructs a realMemoryManager()and writes 50 records throughcreate_task_record()under an RSS-delta assertion; no comment recording the keep-it-live decision or why, despite the rest of the module having gone storage-free.AC5 (recorded re-derivation procedure) — partially: the module docstring documents the original derivation methodology (five local runs, ~3.2x CI conversion factor, headroom multipliers) in detail, which is genuinely useful, but doesn't describe an ongoing cadence for re-checking as CI drifts — closer to "how these numbers were set" than "how to catch the next drift by looking."
1 of 5 fully resolved (via a superior approach), 2 genuinely untouched, 1 partial. Leaving open — sites 2 and 3 are the real remaining work.
Duplicate pair: #15235 and #15055 describe one defect. Found during the v0.10.0 full-coverage triage (#17863), by reading bodies — a title-similarity pass over all 44,253 pairs in that milestone returned a maximum of 0.357 and found this pair, and five others, not at all.
Both concern calibrated wall-clock budgets drifting in the same file; #15235 is the follow-up to #15055's rework. Only #15055 is done — its budgets are now runner-calibrated work units via
autobot_shared.perf_work_budget, while #15235's remaining items (the 420.0 budget with no note, and the real-MemoryManagerwrite-through) still hold. This is the pair that proves closing on duplication alone would close live work.Evidence both resolve to:
autobot-backend/system_benchmarks_performance_test.py—:44(no-budget note),:270,:427Neither issue is being closed, deliberately. A duplicate pair is not a closure argument — closure needs the acceptance criteria checked per issue, which is a separate pass. In at least one of the six pairs only one member's work is actually done, so closing on duplication alone would close a live defect.
On the relationship type: GitHub's native "duplicate" marker requires closing the issue, so it is unavailable here. This comment creates a bidirectional cross-reference instead, which is a native timeline link discoverable from both issues rather than prose in a third place. If a hierarchy is wanted, a sub-issue link can be added where one member genuinely subsumes the other.
Found while fixing #15232. Three sites in the same file, measured on CI, that will produce the next red for reasons unrelated to any diff.
1.
Multimodal processor startup— the header's stated margin is out of dateautobot-backend/system_benchmarks_performance_test.py:397, budget 600.0.PR #15207 left this one deliberately wide after measuring a 2.76× spread in the ratio, recording a worst observation of 315.602 units and describing the budget as roughly 1.9× that. Two dispatched runs since read 372.629 and 428.260.
So the real margin is now about 1.4×, not 1.9×, and the recorded worst case is no longer the worst case. This is the site most likely to go red next, and when it does it will look like a regression rather than drift.
2.
Large config file loading— disk-bound with no note saying sosystem_benchmarks_performance_test.py:426, budget 420.0.ConfigManagerreads a roughly 2000-key YAML, so the numerator is disk-bound and the CPU work unit cannot normalise it — exactly #15232's mechanism.Its sibling at
:397carries a note recording that calibration cannot normalise it. This one carries none, so the next person to see it flake has no way to know it is expected. Readings: 93.274 units on CI, 132.481 locally — a 1.4× spread between two environments on the same code.3.
test_memory_manager_performance— the module's remaining live storage pathsystem_benchmarks_performance_test.py:259writes 50 records through a realMemoryManager. Its assertion is an RSS delta, so it cannot flake on disk timing — but under-n autoit adds SQLite contention that the sibling budgets have to absorb. It is the last live storage path in a module whose other sites were just made storage-free.Why file these together
They are one question: the calibrated budgets in this file were set from a small number of observations, and some of the sites they cover are not CPU-shaped. #15232 fixed one by removing the disk from the measurement. These three need the same decision made deliberately rather than discovered through a red check on an unrelated PR — which is how #15232 surfaced, blocking a PR that had nothing to do with it.
Acceptance criteria
Multimodal processor startup's budget from accumulated CI readings rather than the two the header records, and update the note so the stated margin matches the observed one. Lower it if the data allows; do not raise it.Large config file loadingthe way test(perf): the multimodal processing benchmark measures disk contention — it is the only one of three not mocking _store_result, so calibration cannot normalise it #15232 was decided: remove the disk from the measurement, or record explicitly that it is disk-bound and why the budget is what it is. Silence is the defect.test_memory_manager_performanceshould keep a liveMemoryManager. If yes, record why, given the rest of the module no longer touches storage.:397.Refs #15232, #15055, PR #15207, PR #15233.