autobot-slm-backend: 56 failing tests + 11 errors revealed by #13300 — never executed in CI before

What happened

python-suite step 6 ran two pytest invocations under bash -e. The first exits non-zero on the standing backend failures, so the step aborted and the second never executed. #13300 (PR #13304, merged as adf289c60) split them into separate steps.

The first run with the fix produced two sessions instead of one:

= 252 failed, 19214 passed, 2386 skipped, 2 xfailed, 26 errors in 3551.29s   <- autobot-backend
=  56 failed,  1434 passed,    3 skipped, 1 xfailed, 11 errors in  775.46s   <- autobot-slm-backend, FIRST EVER RUN

These 56 are not regressions. They are pre-existing defects in code that CI has never executed. 1,434 slm-backend tests were silently skipped on every PR for as long as the backend suite has been red.

Owner decision (2026-08-02)

Merge and re-baseline the ratchet at ~308 (252 backend + 56 slm). The gate blocks anything above that; these 56 become tracked work rather than a merge blocker. Chosen over fixing them first because #13304 also fixes the CI-parking failure (#12823) that was stalling every open PR.

The 56, grouped by file

     18 migrations/migrate_sso_secrets_test.py
      6 tests/test_service_management_gate.py
      6 tests/test_prometheus_registry_endpoint.py
      5 tests/services/version_checker_test.py
      5 tests/api/test_sso_auth.py
      4 tests/test_api_request_counter.py
      3 api/code_source_test.py
      2 tests/test_code_sync_topology.py
      2 tests/api/test_nodes_list_timeout_10913.py
      2 services/jwks_verifier_test.py
      1 tests/api/test_role_migration_extra_vars.py
      1 tests/api/test_node_code_status_read_derivation_12428.py
      1 tests/api/test_auth_logout.py

Natural clusters for batching into PRs:

SSO secret migration (migrations/migrate_sso_secrets_test.py) — the largest single group; encryption-at-rest for provider secrets
SSO auth / callback URL (tests/api/test_sso_auth.py) — host validation, including an invalid-host rejection test
JWKS / token verification (services/jwks_verifier_test.py, tests/api/test_auth_logout.py) — RS256 vs legacy HS256
Service-management RBAC gate (tests/test_service_management_gate.py) — admin/operator permission enum
Prometheus registry endpoint (tests/test_prometheus_registry_endpoint.py)
Nodes / code-sync (tests/api/test_nodes_list_timeout_10913.py, tests/test_code_sync_topology.py, api/code_source_test.py)
API request counter middleware (tests/test_api_request_counter.py)

Several are security-relevant (secret encryption, JWKS verification, an RBAC gate), which is the part worth reading first — those have had no CI signal at all.

python-suite aborts after the first pytest: autobot-slm-backend tests and the 70% coverage gate never run #13300 — the abort that hid this (fixed)
ci: Python unit-test suite is not a required check — red tests merged to Dev_new_gui twice #13162 — the ratchet gate this re-baselines
Marker-excluded tests (slow/integration/performance/distributed) run in no workflow at all #13286 — marker-excluded tests, a different never-executed category
ci: backend unit tests not enforced — stale assertion merged via #10623 undetected #10691 — 6-way sharding, which now has slm timing data to split on for the first time

chore(tests): claim worktree for slm-backend slow test investigation (#…

fix(slm-tests): stop code-sync unit tests doing live health polls and…

test(slm): assert the live-health-poll guard aborts instead of dead-w…

fix(slm-tests): keep sqlalchemy.exc importable in the xdist controller (

fix(slm-tests): guard as a runtest hook, block real subprocesses, bin…

test(slm): assert the subprocess guard blocks a real systemctl shell-…

docs(slm-tests): scope the ansible shell-out note to a real repo root (…

chore: claim worktree for issue-13312-sso (#13312)

fix(tests): stop RBAC-middleware test bootstrap from leaking a broken…

fix(tests): update JWT logout test to patch get_async_redis_client, t…

fix(tests): make RBAC-middleware test stubs behave like the real depe…

security(sso): reject non-http(s) callback scheme and stop a bad port…

chore: claim worktree for issue-13312-sso (#13312)

fix(tests): stop RBAC-middleware test bootstrap from leaking a broken…

fix(tests): update JWT logout test to patch get_async_redis_client, t…

fix(tests): make RBAC-middleware test stubs behave like the real depe…

security(sso): reject non-http(s) callback scheme and stop a bad port…

fix(slm-tests): code-sync unit tests dead-waited 180s on live health …

Progress from PR #13320 (merged as `cf3a1e75c`)

Not a closure — this addresses the runtime half plus two live-I/O defects found underneath it. The 56 failures are tracked separately (PR #13324 covers 33 of them).

The four ~180s tests were waiting on a retry chain, not doing work

api/code_sync.py:_wait_component_healthy probes a health URL, swallows every per-attempt error (except Exception: pass), sleeps 2s and retries until the timeout expires — roughly 90 real httpx GETs. The 180s window (rather than the 60s fast one) was selected by accident: _run_post_sync_steps passes _ensure_venv_python's return value as slow_start, and the tests patched it with a bare AsyncMock(), which returns a truthy MagicMock.

It only reproduces where nothing answers the health port — on a workstation running a real backend the first probe returns 200 and the poll exits in 0.03s. Reproduced under a network namespace with nothing listening.

test	before	after
`deploy_bugs::…pip_ok_true_on_npm_success`	182.01s	0.02s
`deploy_bugs::…pip_ok_true_on_success`	181.15s	0.02s
`deploy_bugs::…calls_deploy_repo_root_requirements`	180.64s	0.02s
`symlink_restore::…restores_both_backends`	180.19s	0.02s

Whole slm invocation 665s → ~105s, with the FAILED set byte-identical (md5 742f52fb51b6d1be6a40a2c5991d957c both sides).

Unit tests were spawning real subprocesses against the live install

_snapshot_component was unmocked, so these tests ran a real archive-and-mirror sync into the live /opt/autobot/snapshots tree. The source was a stubbed MagicMock repr so no live data was read or removed — but the sync tool creates the destination before failing (rc=23), so on any host where the test user can write there, every run littered the live snapshot store with empty directories. Defused on this workstation only because that directory is not group-writable. Introduced db441315f (2026-07-09) — roughly 3.5 weeks of exposure.

Also unmocked: _is_systemd_unit_failed spawning a real systemctl.

Both are now blocked structurally rather than per-test: an asyncio.create_subprocess_exec guard alongside the httpx one, package-wide, with zero hits for either guard message across all 1465 tests.

A whole-session CI crash fixed in passing

autobot-slm-backend/conftest.py stubs sqlalchemy as a MagicMock in every process including the xdist controller. The controller rebuilds worker warnings by importing the warning's module, so a worker SAWarning made it import_module('sqlalchemy.exc') against a non-package stub → node down → INTERNALERROR → the entire session's results lost. Confirmed A/B: 0/2 crashes at base, 2/2 after the speedup, because --dist loadscope assignment is timing-dependent. Would have started eating whole CI runs.

Correction recorded

The PR originally attributed 9 new teardown errors to base. Review disproved that by bisecting on an unmodified tree with single-purpose plugins — the guard fixture caused them by taking monkeypatch as a fixture parameter, which registers its finalizer for every test in the package. Restructured as a pytest_runtest_call hookwrapper (no fixture, no finalizer). Error count back to the baseline 10.

That fragility is now tracked as #13329 — the db fixture in test_slm_endpoints_12515.py shields a non-awaitable, so any added fixture finalizer trips it. It also emits the SAWarning behind the controller crash above, making it load-bearing for two separate defects.

Remaining

52 of the 56 tracked failures. PR #13324 covers 33; 23 have no reproducible local failure and no known cause.

fix(tests): SSO/JWKS/RBAC-gate security tests — stop cross-file autob…

Re-measured before closing — all 56 failures + 11 errors are already drained

Re-ran the accounting this issue asked for rather than trusting the 2026-08-02 body, per the instruction that this backlog's premises go stale. They have here: every item this issue tracked has already landed, across four merged PRs, none of which needed a new PR from this pass.

The 56 + 11, traced to their fixes

Group (from the issue body)	Count	Fixed by	Commit
SSO auth / callback URL (`test_sso_auth.py`)	5	#13324	`452d076523`
Service-management RBAC gate	6	#13324	`452d076523`
JWKS / token verification	2	#13324	`452d076523`
`test_auth_logout.py` (stale patch target)	1	#13324	`452d076523`
Nodes / code-sync (`code_source_test.py`, `test_code_sync_topology.py`, `test_nodes_list_timeout_10913.py`)	7	#13324 (side effect of the stub-leak fix)	`452d076523`
Prometheus registry endpoint	6	#13324 (side effect)	`452d076523`
API request counter middleware	4	#13324 (side effect)	`452d076523`
`test_role_migration_extra_vars.py`, `test_node_code_status_read_derivation_12428.py`	2	#13324 (side effect)	`452d076523`
2 RBAC collection errors (of the 11)	2	#13324 (`autobot_shared` stub-leak root cause)	`452d076523`
9 `db`-fixture teardown errors (of the 11, tracked separately as #13329)	9	#13359	`3c33711335`
`tests/services/version_checker_test.py`	5	#13371	`17db271188`
`migrations/migrate_sso_secrets_test.py`	18	#13371 — not a real bug: reproduces only when `autobot-backend` and `autobot-slm-backend` are collected in one pytest session (both define a top-level `migrations` package); CI already runs them as separate invocations (#13084), so this was never actually red in CI	`17db271188`

33 + 6 + 5 + 18 = 62 accounted for against the issue's 56 failed + 11 errors (some errors overlap the failure groups in the original count; #13324's own PR body reconciles 33 failed + 10 errors against a fresh sweep, matching #13312 exactly before it landed).

Runtime-only fixes not part of the 56/11 count but from the same root cause: #13320 (cf3a1e75c9) cut the four ~180s dead-waits to ~0.02s and fixed two live-I/O defects (unmocked _snapshot_component/_is_systemd_unit_failed subprocess calls hitting the real install).

Current CI evidence (the only judge)

Latest completed Dev_new_gui run, https://github.com/mrveiss/AutoBot-AI/actions/runs/33301413716 (run #13814, head c58089324667a6319b1166ba4015d5b37d7033c1, 2026-08-30): all 12 python-suite shards green, both pytest invocations inside every shard. slm-backend per shard, zero failed / zero errors throughout:

shard  1: 149 passed
shard  2: 133 passed, 1 skipped
shard  3: 138 passed
shard  4: 876 passed, 1 skipped
shard  5: 138 passed
shard  6: 144 passed
shard  7: 140 passed
shard  8: 185 passed
shard  9: 160 passed
shard 10: 149 passed
shard 11: 125 passed
shard 12: 221 passed, 2 skipped

~2558 slm-backend tests passing (up from the 1434 that first ran under #13300), 0 failed, 0 errors, across the actual sharded invocation the merge gate runs.

Dangling-reference check

grep -rn "#13312" across the tree: every hit is a historical explanatory comment ("fixed because of #13312") in already-merged code — conftest.py guards, test_rbac_middleware_cache.py, test_sso_auth.py, api/sso_auth.py, etc. No forward-tracking reference (TODO(#13312), tracked in #13312, deferred to #13312) anywhere in the tree.

Why no new PR

I opened a worktree for this (issue-13312) expecting to group and fix remaining failures per the task brief, but found nothing left to fix: the four PRs above already cover 100% of the enumerated 56 failures and 11 errors, and current CI confirms zero regressions since. Opening a PR with no code change would be pure noise against a suite whose entire point is to gate on real content. No worktree changes were made; the branch is being dropped.

Closing — every item this issue enumerated is either fixed in the codebase (with a regression test) or shown not to have been a real failure (the 18 migrations/ collection-collision cases), and CI is green.

Note for #13162 (the "make python-suite required" issue this backlog also depends on): the python-suite job itself is fully green right now, but branch protection on Dev_new_gui does not yet list python-suite in required_status_checks.contexts — that gate is a separate, still-open action item on #13162, not blocked on anything from this issue.

fix(tests): bypass the #13312 subprocess guard for real git calls in …

Navigation Menu

Repository navigation

autobot-slm-backend: 56 failing tests + 11 errors revealed by #13300 — never executed in CI before #13312

What happened

Owner decision (2026-08-02)

The 56, grouped by file

Related

Progress from PR #13320 (merged as `cf3a1e75c`)

The four ~180s tests were waiting on a retry chain, not doing work

Unit tests were spawning real subprocesses against the live install

A whole-session CI crash fixed in passing

Correction recorded

Remaining

Re-measured before closing — all 56 failures + 11 errors are already drained

The 56 + 11, traced to their fixes

Current CI evidence (the only judge)

Dangling-reference check

Why no new PR

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

autobot-slm-backend: 56 failing tests + 11 errors revealed by #13300 — never executed in CI before #13312

Description

What happened

Owner decision (2026-08-02)

The 56, grouped by file

Related

Activity

mrveiss commented on Aug 2, 2026

Progress from PR #13320 (merged as cf3a1e75c)

The four ~180s tests were waiting on a retry chain, not doing work

Unit tests were spawning real subprocesses against the live install

A whole-session CI crash fixed in passing

Correction recorded

Remaining

mrveiss commented on Aug 30, 2026

Re-measured before closing — all 56 failures + 11 errors are already drained

The 56 + 11, traced to their fixes

Current CI evidence (the only judge)

Dangling-reference check

Why no new PR

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Progress from PR #13320 (merged as `cf3a1e75c`)