fix(http): balance _active_requests in response-consuming helpers (#12981)

[mrveiss]

Thinking Path

HTTPClientManager.request() increments _active_requests on every call but only decrements it on the failure path, delegating the success-path decrement to the caller. Repo-wide, exactly one production caller honoured that contract (autobot-backend/chat_workflow/manager.py:2147, a streaming path) against ~151 shared-client call sites. get_json() / post_json() consume the response internally and never decremented at all, so the counter only ever grew.

The counter is not cosmetic — it drives two live mechanisms:

1. Pool auto-sizing — _adjust_pool_size() computes utilization = _active_requests / _current_pool_size. A monotonically rising numerator pins utilisation above the 0.7 grow threshold permanently, so the pool ratchets toward _pool_max (200) regardless of real concurrency, and the < 0.2 shrink branch becomes unreachable.
2. Deferred pool recreation — _handle_pool_recreation() defers recreation while _active_requests > 0, and only decrement_active() reaching 0 applies it. With a counter that never returns to zero, a pending pool resize is never applied.

It also makes get_stats()["active_requests"] and utilization wrong for anything reading them.

The fix had to avoid adding more caller obligations — a contract that ~151 call sites already ignore will keep being ignored. So the decrement moved into the helper that owns the full lifecycle, where it cannot be forgotten, rather than being spread across call sites. The raw-response path (request() / get() / post()) stays caller-owned exactly as documented, because streaming callers legitimately need the slot held past the initial request (#680).

What Changed

Scope is autobot_shared/http_client.py + its test file. No call sites were converted.

• New tracked_request() async context manager — issues the request, yields the response inside async with response, and decrements in a finally. It owns request + response body + counter as one unit. Named for symmetry with the existing tracked_session().
• get_json() / post_json() now use it, so both balance the counter automatically. These were the two internal response-consuming helpers; get()/post() return raw responses and remain caller-owned.
• Docstrings made explicit on request(), get(), and post(): the returned response is caller-owned on the success path and requires decrement_active(), with tracked_request() called out as the forget-proof alternative.
• Fixed the in-file example_usage() streaming sample, which demonstrated the unbalanced pattern (async with await http_client.get(...) with no decrement) and was effectively teaching the bug.

No double-decrement is possible: if request() itself raises, it decrements on its own error path and the try in tracked_request() is never entered.

Verification

New tests fail on the old code and pass on the new — confirmed by temporarily restoring the old get_json() body:

FAILED test_get_json_returns_counter_to_baseline
FAILED test_raise_for_status_failure_returns_counter_to_baseline   assert 2 == 1
FAILED test_sequential_requests_do_not_accumulate                  assert 7 == 2
3 failed, 2 passed

assert 7 == 2 is the reported regression exactly: 5 sequential get_json() calls left the counter 5 higher than baseline.

After the fix:

$ python3 -m pytest autobot_shared/http_client_test.py -q
11 passed in 0.85s

Seven tests added, covering every case: baseline restored after a successful get_json(), after a successful post_json(), after a transport failure, after a raise_for_status() failure mid-block, and after 10 sequential requests (baseline, not baseline+10). Plus tracked_request() exception-safety.

Streaming contract asserted unchanged — test_streaming_contract_unchanged_single_decrement() verifies request() still hands back an un-decremented slot (baseline + 1 while held, so mid-stream pool recreation is still correctly deferred) and that one caller decrement_active() returns it to baseline.

Full suite, no regressions:

$ python3 -m pytest autobot_shared/ -q
1144 passed in 69.22s

# baseline, excluding the 7 new tests
1137 passed, 7 deselected in 49.28s

1137 baseline + 7 new = 1144. No pre-existing test changed state.

$ python3 -m flake8 autobot_shared/http_client.py autobot_shared/http_client_test.py
flake8 exit=0

The sole production decrement_active() caller (chat_workflow/manager.py:2147) is a streaming Ollama path built on raw post() — untouched by this change, and not at risk of double-decrementing.

Model Used

claude-opus-5

Closes #12981

[github-actions]

✅ SSOT Configuration Compliance: Passing

🎉 No hardcoded values detected that have SSOT config equivalents!