Repository navigation
test(slm): whole-backend sweep RUN phase — 105 pre-existing runtime failures from sys.modules order-rot and never-run files (post-#11794) #11798
Description
Observed gap
After #11794, the whole slm-backend sweep (cd autobot-slm-backend && python3 -m pytest -q) collects clean (1220 tests, 0 errors) and runs end-to-end. The RUN phase, however, surfaces pre-existing runtime failures that were previously hidden behind the collection interrupt:
= 105 failed, 1103 passed, 1 skipped, 36 errors in 93.88s =
Baseline comparison (base c7afcc7b6, sweeping with the 8 then-uncollectable files --ignored): 94 failed, 999 passed, 11 errors. The #11794 fix introduced zero regressions — the delta reconciles exactly: +46 bad outcomes from the newly-collectable files, −10 improvements (test_sso_auth 5, test_pg_hba_multiuser 4, jwks_verifier 1 now pass in-sweep).
Inventory (whole-sweep run, grouped by root cause)
A. Never-run test files newly made collectable (45 bad outcomes)
user_management/services/sso_secrets_test.py— 11 F + 13 Euser_management/services/sso_e2e_test.py— 9 F + 12 E
Fixtures call create_async_engine against the root-conftest sqlalchemy MagicMock → TypeError: object MagicMock can't be used in 'await' expression. Fix = the #11737 real-load pattern (tests/services/code_version_test.py): snapshot/clear sqlalchemy*, real-import sqlalchemy + sqlalchemy.dialects.sqlite.aiosqlite (aiosqlite is installed), real-load models/database.py, services/encryption.py (needs SLM_ENCRYPTION_KEY env), user_management/services/sso_secrets.py, restore.
B. Runtime-phase sys.modules clobbering (order-dependent; all pass per-file)
tests/api/test_sso_auth.pyassignssys.modules["fastapi"] = MagicMock()(and 6 more keys) inside test functions, permanently — later starletteTestClientusers break:tests/test_prometheus_registry_endpoint.py(6 F),tests/test_api_request_counter.py(4 F). Verified pre-existing: the pairtest_sso_auth.py + test_saml_slo.pyfails identically on base.tests/test_claude_api_monitor.py(14 F in sweep, passes alone):ModuleNotFoundError: 'monitoring' is not a package— a baretypes.ModuleType("monitoring")squat (installed bytests/test_api_request_counter.py) shadows the real package for the runtime importfrom monitoring.claude_api_monitor import UsageTracker.tests/services/test_saml_slo.py::TestBuildSsoLogoutUrl::test_saml_link_calls_initiate_saml_slo(1 F in sweep): itsuser_management.models.ssostub is guarded bynot in sys.modules;tests/api/test_scim.pyleaves that key populated, so the real-enum stand-in is skipped. Reproducer (fails on base too):pytest tests/api/test_scim.py tests/services/test_saml_slo.py.
C. Pre-existing per-file / sweep-order failures (identical counts on base)
| file | sweep | note |
|---|---|---|
api/code_source_test.py |
12 F | fails alone on base too |
tests/api/test_drift_resolve.py |
12 F | fails in any paired run on base (builds requests from conftest MagicMock schemas) |
services/jwks_verifier_test.py |
10 F | sweep-order only (passes in services/-only run) |
services/drift_checker_test.py |
9 F | fails alone on base too |
main_ensure_local_node_test.py |
7 E | sweep-order |
tests/api/test_sso_health.py |
4 F + 4 E | fails alone on base too |
slm/agent/health_collector_state_change_test.py (+ ansible copy) |
2 + 2 F | sweep-order |
tests/test_websocket_auth_smoke.py |
2 F | fails alone on base too |
tests/test_code_sync_topology.py |
2 F | 1 E alone on base |
tests/api/test_fleet_node_update_11511.py |
2 F | sweep-order (present on base sweep) |
tests/api/test_component_resolve_job.py |
2 F | sweep-order (present on base sweep) |
tests/api/test_require_permission.py |
1 F | fails alone on base too |
Fix direction per #11794/#11478: polluters snapshot/restore what they stub (module-level AND test-function-level writes); victims real-load their deps where the conftest mocks can never satisfy them.
This is the runtime counterpart of #11794 (collection) and the same class #11737/#11795 fixed for tests/services/.
Activity
mrveiss commentedon Jul 16, 2026
Cross-refs (same defect class, keep fixes consistent): #11796 — identical module-scope sys.modules stubbing on autobot-backend (~128 collection errors, found via #11791); #11248 — older scoped inventory of tests/services failures that overlaps this run-phase list (several of its 10 files were likely fixed by #11737/PR #11795 — re-verify against post-c7afcc7b6 base before working it); umbrella #9924 (test-infra / namespace fragility) is the parent theme. Established fix patterns: polluter-side sys.modules snapshot/restore (#11737 code_version sqlalchemy*-prefix sandbox), real-load via _real_load_and_bind / spec-load (#11661/#11729), conftest path repair (PR #11797).
fix(test): repair slm sweep run-phase order-rot + never-run files (#1…
mrveiss commentedon Jul 17, 2026
Closed — merged to Dev_new_gui. Commit: f540fff (PR #11831). Whole slm sweep: 105 failed/1111 passed/36 errors → 1226 passed / 3 justified skips / 0 failed / 0 errors (independent re-run identical); collection 1229/0 kept; tests/services 321/1 intact. Product bugs fixed with test evidence: sso_secrets._legacy_retrieve strict-decrypt (corruption no longer masked as 'no secret'), oidc_token_cache Redis-down contract (_redis_or_none, 9 TypeErrors → 0). Discovery filed: #11830 (connection-manager singleton poisoning under stubbed config). Cross-refs: collection phase was #11794; backend sibling campaign #11796.