tech-debt(api): fastapi>=0.139 defers include_router() — the repo knows this in four places and guards it in none #15093

Verdict

Zero sites are genuinely broken today. A full sweep of every .routes read in the repo (134 hits across 60 files) found no site that reads a router which actually includes children and expects the flattened result — except the two that already handle it. The blast radius of the FastAPI 0.141.1 deferred-inclusion change is, right now, nil.

That is luck, not design. The knowledge that the change exists is already written down in four separate places, each with a private workaround, and there is no shared helper and no guard. The next site to read a parent router's .routes will get a silently wrong answer, and nothing in CI will say so.

This issue records the class, the sweep that bounds it today, and asks for the guard that keeps the bound.

The behaviour

Under fastapi>=0.139 (CI resolves 0.141.1 / starlette 1.6.0), parent.include_router(child) no longer flattens the child's routes into parent.routes. It appends a single deferred _IncludedRouter entry with no .path and no .methods; the child's real routes materialise only when an app mounts the router, and are otherwise reachable only via included.original_router.routes.

Consequences at a call site:

• an unguarded route.path walk raises AttributeError: '_IncludedRouter' object has no attribute 'path'
• the same walk guarded with hasattr/getattr silently finds nothing — the strictly worse outcome
• len(router.routes) counts one entry per include_router() call, not per endpoint

How it was proven

CI job 98088603289 (python-suite shard 12/12, run 32939657631, conclusion failure) on PR #15085 dumped the assembled router's top level as:

[APIWebSocketRoute /ws/{session_id}, APIWebSocketRoute /ws/ssh/{host_id}, _IncludedRouter path=null]

— three entries, one of them a deferred wrapper with a null path. The same code run locally produced 26 routes. Fixed on that PR by mounting a FastAPI() app and enumerating app.routes instead of walking the router.

The local/CI split is itself tracked as #15091: this development box resolves fastapi 0.135.2 / starlette 1.0.0, below the declared floor of fastapi>=0.141.1 (autobot-backend/requirements.txt, autobot-slm-backend/requirements.txt) and below the CI pin fastapi==0.141.1 (requirements-ci/framework.txt). A local pass carries no information about this behaviour. Everything below is reasoned from the documented semantics plus the CI evidence; it was not reproduced locally, and it cannot be until #15091 lands.

Sweep and classification

Method: git grep -n '\.routes' -- '*.py' → 134 hits / 60 files. Import lines (from .routes import router) discarded. For every remaining site the router object was resolved back to its defining module and that module checked for include_router() calls against that same router.

The decisive fact: the set of routers that include children is disjoint from the set of routers whose .routes anyone reads — with exactly two exceptions, both already handled.

Routers that call include_router() on themselves:
autobot-backend/api/{knowledge,terminal,agent_terminal,transcriber,chat,voice_bundle_admin,analytics}.py, autobot-backend/api/codebase_analytics/router.py, autobot-backend/llc/api/__init__.py (37 children), autobot-backend/middleware/builtin/transcriber_extension.py (combined).

Of those, only llc/api/__init__.py and codebase_analytics/router.py have their .routes read anywhere — and both readers already traverse .original_router.

Already handled (2)

Handled by sidestepping the router entirely (3)

Degraded — passes, but verifies or reports less than it claims (3)

Latent — correct today only because the router happens to have no children (3)

Not affected (all remaining sites)

Every other reader resolves to a router built purely from @router.get/post/put/delete/websocket decorators, which register eagerly on every FastAPI version, or reads an app rather than a router. Named in full:

autobot-backend/api/agents_self_improvement_test.py:29,58 · autobot-backend/api/marketplace_sources_routes_test.py:18 (+ the plugin_manager router) · autobot-backend/api/presence_ws_router_test.py:31,34,40 · autobot-backend/api/settings_telemetry_test.py:58 · autobot-backend/api/telegram_bot_test.py:43 · autobot-backend/api/whatsapp_test.py:37 · autobot-backend/backend_startup_e2e_test.py:33,53 · autobot-backend/tests/api/test_web_research_settings_guard.py:40 · autobot-backend/tests/test_diagnostics_router_registration.py:51,57 · autobot-slm-backend/tests/api/test_prometheus_scrape_is_unauthenticated_14339.py:66 · autobot-backend/llc/tests/test_agent_hires_auth.py:99 · autobot-backend/llc/tests/test_contact_directory.py:300 · autobot-backend/llc/tests/test_findings_api.py:395 · autobot-backend/tests/api/test_onboarding_auth.py:153,178 · autobot-backend/tests/integration/test_mobile_device_me.py:109 · autobot-backend/tests/initialization/test_analytics_cost_router.py:28,42,51,59,83,94,97 · autobot-backend/tests/initialization/test_analytics_export_router.py:28,42,51,59,76 · autobot-backend/tests/initialization/test_knowledge_grounding_router.py:28,36,45,53,68 · autobot-backend/tests/initialization/test_health_celery_dead_letter.py:27 · autobot-backend/utils/operation_timeout_integration_test.py:76 · autobot-backend/api/self_capabilities_test.py:51,264 (synthetic fixture) · autobot-infrastructure/shared/scripts/utilities/validate_chat_knowledge.py:155 · autobot-infrastructure/shared/scripts/utilities/verify_backend_config.py:198,200 · autobot-backend/api/codebase_analytics/api_endpoint_scanner.py (static AST/regex scan, no runtime introspection).

Two of these deserve a note because they look dangerous and are not:

• autobot-backend/api/long_running_operations.py:153, 220, 273, 325 — await manager.router.routes[0].endpoint(create_request), in production code (api.long_running_operations is registered at initialization/router_registry/feature_routers.py:418). manager.router is OperationIntegrationManager.router (autobot-backend/utils/operation_timeout_integration.py:112); that file contains zero include_router calls, and _setup_routes() (:318) calls _setup_crud_routes() first, whose first decorator is @self.router.post("/create") → create_operation (:327-328). So routes[0].endpoint is still create_operation under 0.141.1. Neither an AttributeError nor a wrong endpoint. It remains a fragile idiom — reordering _setup_crud_routes silently redirects four production endpoints — but that is a separate robustness concern, not this class.
• autobot-backend/tests/initialization/test_analytics_cost_router.py / test_analytics_export_router.py — these read analytics_cost.router / analytics_export.router, which are the children included by autobot-backend/api/analytics.py:402,404. Reading a child is unaffected; only reading analytics.router would break.

Counts

The real finding: the knowledge exists and did not spread

Four places in the repo already know about this. Each solved it privately. None produced a shared helper, and none left anything that would catch a fifth site:

git grep -n 'original_router' -- '*.py' returns two files. There is no helper in autobot_shared/. Meanwhile autobot_shared/api_routing/router_prefixes.py already exposes INCLUDE_ROUTER_NAME_RE (:74) and _INCLUDE_ROUTER_TARGET_RE (:66) — precisely the static machinery a guard needs — and repo_tests/router_prefix_convergence_test.py is the precedent for a repo-wide source-parsing guard built on it.

Caveat: one handler is version-locked the other way

autobot-backend/api/codebase_analytics/endpoints/impact_endpoint_test.py:164 does:

for sub in getattr(included, "original_router", None).routes

No None fallback. On an eager-inclusion FastAPI (<0.139, i.e. what the dev box in #15091 actually resolves), router.routes holds real APIRoute objects with no original_router, so getattr(...) returns None and the line raises AttributeError: 'NoneType' object has no attribute 'routes'. Correct in CI, crashes locally. test_roles_routes_registered.py:66 gets this right with subroutes = original.routes if original is not None else [included]; that is the form a shared helper should take. Sharpens the case for #15091.

Acceptance criteria

1. A shared traversal helper exists in autobot_shared/ that, given an APIRouter, yields its effective routes correctly on both eager and deferred inclusion (the test_roles_routes_registered.py:66 form, recursing through nested inclusion). The four existing private workarounds are migrated to it; git grep 'original_router' afterwards points at the helper and its tests, not at scattered call sites.

2. A guard fails when a new site reads <router>.routes expecting flattened children. A static repo-wide check (host: repo_tests/, alongside router_prefix_convergence_test.py; reuse autobot_shared/api_routing/router_prefixes.py for include_router detection rather than a new regex — core rule 2). It must:

   • scan every tracked *.py for reads of <name>.routes where <name> resolves to an APIRouter (not a FastAPI app),
   • resolve <name> to its defining module and determine whether that router receives include_router() calls,
   • fail on any such site that does not go through the shared helper, get_openapi(), or a mounted app,
   • carry an explicit allowlist for the sidestepping sites (self_capabilities.py, audit_api_wiring.py), each entry annotated with why it is exempt.

3. The guard is proven to bite — contrast mutation. A test adds a fixture module to a temp tree containing a parent router that does parent.include_router(child) plus a site doing for r in parent.routes: r.path, runs the guard's scanner over that tree, and asserts it is reported. Then the same fixture with the include_router line removed (decorator-only router, everything else byte-identical) must not be reported. Both halves required: the first proves detection, the second proves the guard is not simply flagging every .routes read.

4. Non-vacuity — an empty sweep fails loudly. The guard asserts its scanner found a non-zero number of candidate sites (routers-with-children whose .routes are read, allowlisted ones included) before evaluating any of them. A path-resolution or import bug that makes the scan match nothing must turn the check red, not green. Precedent: scripts/audit_api_wiring_test.py:139-143 (test_dump_openapi_websocket_union_survives_empty_runtime_walk) pairs a positive assertion with _runtime_websocket_paths_returns_empty(fake_app) for exactly this reason, and autobot-backend/llc/tests/test_roles_routes_registered.py:101-105 records that the first draft of that file "passed" while verifying nothing. This property caught a real defect on fix(terminal): stop the router-level admin Depends 500ing both terminal WS handshakes #15085 — it is not decoration.

5. The degraded sites are resolved, each either fixed or its claim corrected:

   • self_capabilities.py:178,189 — cache key derived from something that actually changes with the route table (e.g. the get_openapi() path count already computed at :190), or the docstrings at :167 and :218 amended to stop claiming route-change invalidation.
   • presence_ws_router_test.py:65 — assert a real mounted path instead of len(app.routes) > 0.
   • test_real_startup.py:82 — count through a version-correct view or drop the line.

6. impact_endpoint_test.py:164 gets the None fallback (via the helper from AC 1), so it stops being correct only on >=0.139.

7. No fixes are in scope for chore(devenv): local verification runs fastapi 0.135.2 against a declared floor of >=0.141.1 — a local pass carries no information about CI #15091. This issue must not be closed by making the dev box match CI; the guard has to be version-independent and pass on both.

Not in scope

• autobot-backend/api/batch_jobs.py — tracked in bug(api): POST /api/batch-jobs/load is dead — imports a nonexistent module, and its route lookup is blind on fastapi>=0.139 #13558.
• autobot-backend/api/developer.py register_router dead wiring — noted above, needs its own issue if not folded in.
• The routes[0] positional-index fragility in long_running_operations.py — correct today, separate robustness concern.

Related

• chore(devenv): local verification runs fastapi 0.135.2 against a declared floor of >=0.141.1 — a local pass carries no information about CI #15091 — this box runs fastapi 0.135.2 against a declared floor of >=0.141.1; local verification cannot observe any of this. Blocks reproducing the behaviour outside CI.
• bug(terminal): both terminal WebSocket routes 500 on every handshake — a router-level Depends cannot resolve in a WS scope #14998 — the terminal WS handshake bug whose fix (fix(terminal): stop the router-level admin Depends 500ing both terminal WS handshakes #15085) surfaced the CI evidence cited above.
• fix(terminal): stop the router-level admin Depends 500ing both terminal WS handshakes #15085 — where the _IncludedRouter dump and the 3-vs-26 route-count split were captured (job 98088603289).
• bug(api): POST /api/batch-jobs/load is dead — imports a nonexistent module, and its route lookup is blind on fastapi>=0.139 #13558 — the one production call site with this premise, already filed.
• test: 13 unrelated singleton failures remain in the Python suite (#13162 tail) #13551 — where self_capabilities_integration_test.py first hit AttributeError: '_IncludedRouter' object has no attribute 'path'.
• audit_api_wiring: resolve SLM-backend (:8000) getBackendUrl calls so they aren't false-unwired #12381, audit_api_wiring: WS-route enumeration misses sub-router parametrized WebSocket routes (advanced-control/ws/*, overseer/ws/*) #12432 — where audit_api_wiring.py worked around the same change with a static scan.