fix(test): retry a starved heartbeat window instead of failing or skipping it (#15266)

[mrveiss]

Thinking Path

_MIN_TICKS_PER_WINDOW = 10 in process_offload_test.py exists to reject a
measurement window too short to hold a median — a sound precondition. But it
rejected by failing, which reports a contended CI runner as a code
regression: exactly the confusion #15221 was filed about, one layer down
(#15221's reopening comment traces this).

Both obvious branches are wrong. Skipping is how a real regression hides —
pytest.skip in this same file was already found unreachable once and fixed
in #15248, and reintroducing a skip path would undo that. Failing calls a
busy runner a regression, which is the defect under repair. A measurement
that could not be taken is neither a pass nor a failure — it is a retry.
Retake the window a small bounded number of times, and fail only if every
attempt starves. That distinguishes "this runner was momentarily busy" from
"this event loop is persistently starved", and only the second deserves a red.

The retry is scoped as narrowly as possible: it wraps only the "could this
window be measured at all" precondition inside _tick_latency_during_scan,
never the ratio assertions in the test body. Those run outside the retry
call entirely, so a genuine regression is scored on its first and only
attempt, exactly as before.

What Changed

• autobot_shared/perf_work_budget.py: added MeasurementStarved (an
  Exception, deliberately not an AssertionError, so it can never be
  confused with — or accidentally catch — a real ratio-budget failure) and
  measure_with_starvation_retry(measure, *, max_attempts, what), a general
  async retry helper for any contention site whose measurement window can
  legitimately come up empty. It catches only MeasurementStarved; on
  persistent starvation it raises AssertionError with a message naming
  starvation explicitly, distinct from a budget-breach message.
• autobot-backend/code_intelligence/shared/process_offload_test.py:
  _tick_latency_during_scan now raises MeasurementStarved instead of
  asserting when a window under-fills. The test wraps each of its two
  measurement calls (force_thread=True/False) in
  measure_with_starvation_retry with a new bounded constant,
  _MAX_STARVATION_RETRIES = 3. The test's own docstring states why this is
  a retry and not a skip or a pass, per test(perf): the process-offload heartbeat asserts a 7.5ms absolute ceiling, so runner load and a real regression are indistinguishable #15221's reopening comment.
• autobot_shared/perf_work_budget_test.py: added TestStarvationRetry,
  covering a clean first attempt (no retry), an eventually-successful retry,
  persistent starvation reaching the bounded ceiling and still failing, and
  — the mutation argument — a plain AssertionError (what a completed
  measurement raises on a real regression, worded like the ~1.994x reading a
  forced in-process scan produces against the 1.5 budget) propagating on
  its first attempt, never retried or masked.

_TICK_BUDGET_VS_IDLE = 1.5 is untouched. _MIN_TICKS_PER_WINDOW = 10 is
untouched — the precondition itself is correct; only what happens when it
isn't met has changed.

Verification

Read, not run, per the hard constraint (this is a timing test on the CI
runner itself):

• _tick_latency_during_scan still performs its cleanup
  (shutdown_scan_pool, beat.cancel(), resetting
  _pool_unavailable_reason) before raising MeasurementStarved, so each
  retry attempt starts from clean state — unchanged from the prior
  assert's position in the function.
• measure_with_starvation_retry catches only MeasurementStarved. The
  ratio assertions (in_process.busy_ms < in_thread.busy_ms * 0.75 and
  assert_within_baseline_ratio(...)) execute in the test body, entirely
  outside any retry call — a regression there raises a plain
  AssertionError on the only attempt taken, with no retry loop in scope to
  catch or repeat it.
• Mutation argument (forcing the scan in-process previously produced
  1.994x its own idle baseline against the 1.5 budget, per test(perf): the process-offload heartbeat asserts a 7.5ms absolute ceiling, so runner load and a real regression are indistinguishable #15221's
  closure comment): that failure happens inside assert_within_baseline_ratio,
  called directly in the test body — never wrapped by
  measure_with_starvation_retry — so it still fails on first measurement,
  unaffected by this change. TestStarvationRetry::test_a_measured_regression_is_never_retried_or_masked
  encodes this directly with a synthetic AssertionError worded the same way.
• Persistent starvation: TestStarvationRetry::test_persistent_starvation_fails_after_the_bounded_ceiling
  proves a measure that always raises MeasurementStarved is called
  exactly max_attempts times and then raises AssertionError naming
  "starved on all N attempts" — bounded, not silently swallowed, not a skip.
• The pre-push hook ran pytest on both touched test files against this
  branch and reported all relevant tests passing (see push output on this
  branch).
• ruff check and black --check both pass clean on all three touched
  files; every touched file is well under the 600-line cap
  (perf_work_budget.py 281, perf_work_budget_test.py 143,
  process_offload_test.py 423).

Closes #15221 pending confirmation on the reopener's own criteria (a
deliberately-starved run reaching a verdict rather than erroring on the
precondition, and the retry-vs-skip-vs-pass distinction stated in the test
itself) — leaving that issue for the reopener to close by hand per repo
convention, since Closes keywords do not fire on Dev_new_gui.

Model Used

Claude Sonnet 5

Closes #15266
Refs #15221 — this is the fix that its reopening comment set as the condition for closing again.

[github-actions]

✅ SSOT Configuration Compliance: Passing

🎉 No new hardcoded values of either class — ssot and other both block.

Known backlog in pipeline-scripts/hardcoded_values_baseline.txt is suppressed and tracked in #14371.

[mrveiss]

Fixed: total-blockage regressions no longer retried as contention

Commit ec67f9c24. The two windows already measured (idle bracketing the
busy one) now discriminate WHY a window came up short:

• busy starved, idle healthy -> nothing external explains it, the scan
  itself stopped the loop. Raises a plain, non-retried AssertionError
  (_raise_if_a_window_starved, process_offload_test.py:224), scored on
  attempt 1, same as the ratio assertions.
• Either window short with no such asymmetry (both starved, or idle alone)
  -> ordinary contention, raises MeasurementStarved, retried to
  _MAX_STARVATION_RETRIES as before.

measure_with_starvation_retry's docstring (perf_work_budget.py:264-289)
now states the guarantee it cannot itself enforce -- that the caller must
discriminate operation-caused starvation from environment-caused starvation
before raising MeasurementStarved -- naming _raise_if_a_window_starved
as the worked example, matching the honesty pattern at
assert_within_baseline_ratio:177-189.

New tests pin the discrimination directly (process_offload_test.py:396-464):
test_scan_only_starvation_fails_on_the_first_attempt_without_retrying
(asserts calls == 1) and test_both_windows_starved_retries_to_the_bounded_ceiling
(asserts calls == _MAX_STARVATION_RETRIES), plus four unit-level tests on
_raise_if_a_window_starved covering all four (busy, idle) starvation
combinations.

Extracted _raise_if_a_window_starved out of _tick_latency_during_scan to
keep both functions under the 30-line guideline (30 and 30 body lines
respectively). File sizes: process_offload_test.py 533 lines,
perf_work_budget.py 303 lines -- both well under the 600-line cap.

Pre-push hook ran pytest on both touched test files against this branch and
reported all passing.