tech-debt(backend): 51 of 95 health/status handlers await unbounded — #12751's guard test cannot land until they are triaged #12892

Summary

#12751 fixed two health endpoints that hung (/api/web-research/status, /api/entities/extract/health) and bounded the run-JWT denylist lookup behind them. Its third recommendation — "add a smoke test asserting every */health and */status endpoint responds within a small budget" — was never delivered, because the rule it would enforce is broken across a much wider surface than those two endpoints.

An AST scan of every route handler in autobot-backend/ whose path ends in /health or /status:

health/status endpoints: 95; with at least one await NOT wrapped in asyncio.wait_for/asyncio.timeout: 51

The two endpoints #12751 named are now clean (entity_extraction_health awaits=0; get_research_status awaits=2, both bounded). The other 51 are not, so the guard test cannot simply be added — it would fail immediately.

The heuristic over-matches — this needs triage, not a blanket sweep

Matching on the path suffix pulls in handlers that are not health probes and legitimately do work:

• api/git_mcp.py:572 git_status_mcp — runs a git status operation
• api/agent_org.py:373 update_delegation_status — a mutation, not a probe

Wrapping those in a health-probe budget would be wrong. So the fix is not "wrap all 51": it is to separate genuine liveness/readiness probes (which must be fast, bounded and non-blocking) from action endpoints that merely happen to end in /status, and bound only the former.

Sample of the genuine-probe end of the list:

api/advanced_control.py:496   get_system_health        (3 unbounded)
api/agent_config.py:1281      check_agent_health       (1 unbounded)
api/knowledge_ai_stack.py:669 knowledge_health         (2 unbounded)
api/knowledge_connectors.py:516 connectors_health      (2 unbounded)
api/analytics_code.py:98      get_code_analysis_status (3 unbounded)

Why it matters

A health check that blocks on the very dependency it is probing defeats its purpose: callers (UI health panels, monitoring) stall instead of receiving a clear degraded signal. That is the exact failure #12751 reported, and #12316 before it — the pattern recurs because nothing enforces it.

Suggested approach

1. Triage the 51 into probes vs action endpoints that end in /status. Only probes are in scope.
2. Bound each probe's dependency calls with the shared HEALTH_PROBE_TIMEOUT_S (already the single source of truth in api/system_health.py, used by the health endpoints hang: /api/web-research/status and /api/entities/extract/health time out (8s+, no response) — health check blocks on its own dependency; should bound + return degraded status #12751 fix) and return a structured degraded response on timeout, matching get_research_status.
3. Then add the guard the original recommendation asked for, so a new probe cannot ship with an unbounded await. Add it only once the set is clean — a baselined allow-list would mask exactly what it is meant to catch.

Reproduce

The scan is a small AST walk over autobot-backend/**/*.py: collect functions decorated with a router method whose first argument ends in /health or /status, then report any ast.Await whose value is not a call to wait_for/timeout.

Related: #12751 (the two endpoints, fixed), #12316 (health endpoint crashes), #12341 (slow endpoints).