Repository navigation
test-remediation: 10 tests/services files fail scoped (8 real assertion failures + 2 conftest-stub artifacts) #11248
Description
Body repaired 2026-07-15: the original body was accidentally posted as a literal scratchpad file path (
@/tmp/...) from a prior session and its content is lost. Reconstructed from the title; findings below re-verified 2026-07-15.
Original scope (from title)
10 autobot-slm-backend/tests/services files fail when run scoped (not via the full repo-wide pytest invocation): 8 real assertion failures + 2 conftest-stub artifacts.
Re-verified 2026-07-15 (while implementing #11717)
The conftest-stub layer is the structural cause of the artifact class:
autobot-slm-backend/conftest.pyunconditionally MagicMock-stubssqlalchemy/models.databasewithout attempting a real import first (unlikeautobot-backend/conftest.py, which real-imports light modules — see the Pre-existing: in-package agent_loop/ tests fail when co-collected with stubbing conftest (14) #11153/ci: backend unit tests not enforced — stale assertion merged via #10623 undetected #10691 pattern). Any test doingfrom models.database import Nodeand expecting real ORM attribute behavior only passes when a previously-collected module happens to real-import sqlalchemy first — test outcomes depend on collection order.- Separately, 9 files under
autobot-slm-backend/tests/api/fail collection standalone withModuleNotFoundError: services.ansible_secrets; 'services' is not a package. - Workaround used by new tests (e.g.
tests/services/test_node_ansible_target_11717.py): swap in the realsqlalchemy/models.database, run, restore stubs — this should be a shared conftest helper, not per-file boilerplate (overlaps consolidate(test): merge conftest _load_real_mod + _real_load_service into one always-binds helper #11661 consolidation).
ACs
- slm-backend conftest attempts real import before stubbing (light-module real-load pattern), or provides a canonical real-load fixture
-
tests/servicesandtests/apipass when run scoped, independent of collection order - The 8 real assertion failures triaged/fixed or filed individually
Activity
mrveiss commentedon Jul 8, 2026
Triage progress (via a py3.12 venv — closer to target py3.14 than local py3.10):
ClientResponseError(None, None, status=400)(request_info=None) makesstr(exc)raise inside the except-blocklogger.error(..., exc), crashing before the service's real re-raise. Built a well-formedRequestInfo; now 45 passed on both versions. Service code was correct.Continuing the triage.
test(notification): build well-formed ClientResponseError so 4xx re-r…
mrveiss commentedon Jul 8, 2026
Triage update (py3.12 venv):
✅✅ notification_service_test + test_conflict_resolver — FIXED & merged (PR test(#11248): fix notification 4xx mock + conflict_resolver stale redis attr #11273, squash d912048). Both clean test-rot (malformed
ClientResponseErrormock; stale_redis_client→_redisafter theAsyncRedisClientMixinmigration). Verified on both py3.10 and py3.12.chain_depth=4, confidence=0.85) correctly yields base=CRITICAL, but_assess_severity(causal_inference_engine.py:670-674) intentionally downgrades CRITICAL→DEGRADED when a high-confidence fix exists (intervention.predicted_success_rate≥0.8 AND confidence≥0.8; the test's intervention is 0.9/0.9), per the explicit comment "High-confidence fix available, not critical." So the code behaves as documented and returns DEGRADED. Decision needed: is that downgrade intended (severity = urgency of manual attention → fixable errors are less urgent) — in which case fix the test (lower the intervention confidence so it verifies the true CRITICAL path, or rename to expect DEGRADED) — or should a critical error stay CRITICAL regardless of fix availability (then the downgrade at :671 is the bug)? Not resolving blind since it changes severity semantics used by alerting/prioritisation.❓ concurrent_limiter / grounded_agent / rag_service_events — still inconclusive: my scratch py3.12 venv is dep-incomplete for these (collection error / mismatched pass-counts vs py3.10), so I can't cleanly separate real bugs from env noise yet. Need a complete backend venv (ideally py3.14) to classify.
Net: 2 of the 8 fixed; 1 escalated as a design decision; 3 need a complete target-env venv; 2 are #10691 stub-class.
mrveiss commentedon Jul 8, 2026
More triage (py3.10 reliable baseline):
✅ rag_service_events_test — 37 of 38 FIXED (PR test(rag): patch real publish_event (was stale publish_live_event) — fixes 37/38 (#11248) #11282). Clean test-rot: the test patched
services.rag_service.publish_live_eventbut the code callspublish_event(fromevents.bus; nopublish_live_eventexists) → every patch raised AttributeError at setup. Renamed 14 patch targets + a stale docstring. 1 residual anomaly:test_calls_publish_eventreports 0 mock calls while its 7 identical siblings intercept fine and the code demonstrably publishes (debug log) — a per-test mock-interception oddity, pre-existing, flagged for follow-up._reconstruct_response(grounded_agent.py:527) does case-sensitiveif claim_text in reconstructed. The test's claim"Latency increased"(capitalised) doesn't match the response"the latency increased", so no annotation is inserted → assertion fails. LLM-extracted claims routinely differ in case from the source text, so annotations silently drop today. Fix direction is case-insensitive matching — but the test also asserts the claim's capitalisation appears in the output, so a fix must decide whether the annotation preserves the response's original case or rewrites to the claim's case. Flagging rather than blind-fixing since it changes user-visible annotation output.#11248 running tally: notification ✅ · conflict_resolver ✅ · rag 37/38 ✅ · causal⚠️ (severity-downgrade design Q) · grounded ⚠️ (case-match code Q) · concurrent_limiter ❓(needs complete venv) · tool_output_filter + llm_api_key_service = #10691 stub-class. 3 clean-fixed, 2 flagged-for-decision, 1 needs-env, 2 systemic.
test(rag): patch real publish_event (was stale publish_live_event) — …
fix(grounded): case-insensitive claim annotation + verify causal CRIT…
mrveiss commentedon Jul 8, 2026
Progress — 2026-07-08 remediation loop
Real assertion failures: 7 of 8 resolved.
notification_service_test::test_raises_on_4xx_responseClientResponseError(#11273, merged)test_conflict_resolver::test_init_lazy_loads_redis_redis_client→_redisrename (#11273, merged)rag_service_events_test::test_calls_publish_live_eventpublish_live_event→publish_event(#11282, merged)test_causal_inference_engine::test_critical_severitytest_grounded_agent::test_reconstruct_response_with_annotationtest_llm_cost_tracker::test_model_has_pricing[claude-opus-4-20250514]ANTHROPIC_CLAUDE_OPUS4(claude-opus-4-20250514) IS present inMODEL_PRICING({'input':15.0,'output':75.0}, ssot_constants.py:280). The py3.10 triage was a module-import artifact (deps missing → whole file errored, all params reported failed). Verified passing via direct import.test_scheduler_registry::test_all_scheduler_files_registeredllc/scheduler/heartbeat_scheduler.py,routine_scheduler.py— both started ininitialization/lifespan), excludedmigrations/false positive. 5/5 pass.test_concurrent_limiter::test_queue_still_waitsasyncio.Eventartifact; needs py3.14 verification before touching concurrency logic.Conftest-stub artifacts (2) —
test_tool_output_filter,test_llm_api_key_service— remain folded into the #10691/#11153 conftest-reconciliation effort (not logic bugs).After #11288 merges, only
test_concurrent_limiter(py3.14 verify) + the 2 #11153 stub artifacts remain.fix(schedulers): register LLC heartbeat/routine schedulers; exclude m…
test(conftest): real-load services.tool_output_filter so _strip_ansi …
mrveiss commentedon Jul 8, 2026
Progress — strip_ansi artifact fixed (PR #11329)
test_tool_output_filter::test_strip_ansi_*(conftest-stub artifact) — FIXED via test(conftest): real-load services.tool_output_filter so _strip_ansi tests pass (#11248) #11329:conftest.pynow real-loadsservices.tool_output_filter(lightweight) after theservicespackage stubs, so the tests exercise the real_strip_ansi. 82 passed, zero regressions (blast-radius checked vs baseline).Remaining in this umbrella:
test_llm_api_key_service::test_parse_key_id_from_bearer_valid— real-loadingllm_api_key_servicefixes it and cuts the file's failures 88→14, BUT the remainingmodel_allowed/budget/auth tests then showLLMApiKeyRecord-appears-mock behavior even in isolation (not resolved by the real-load, not test-order pollution). Held back pending root-cause — didn't want to ship a partial/not-understood change.test_concurrent_limiter::test_queue_still_waits— needs a py3.14 env to verify (suspected asyncio.Event artifact).test_llm_cost_trackerwas a py3.10 import artifact (key present).mrveiss commentedon Jul 8, 2026
Progress — llm_api_key_service artifact fixed (PR #11332)
test_llm_api_key_service(conftest-stub artifact) — FIXED via #11332 (Closes #11331):conftest.pynow real-loadsservices.llm_api_key_servicevia apytest_configurehook (runs after all module-level stubs). Real-loading surfaced two masked bugs, both fixed in the same PR: (1)_r()ignored the injectedself._redisthe tests rely on; (2)test_issue_keymockedpipeline()as async when redis-py's is synchronous. 8 failed/10 passed → 18 passed, zero regressions (full-suite collection identical to baseline).Remaining in this umbrella: only
test_concurrent_limiter::test_queue_still_waits— code reads correct; suspected py3.10/3.12asyncio.Eventartifact, needs a py3.14 env to verify before touching concurrency logic. Both #11153 conftest-stub artifacts (tool_output_filter #11329, llm_api_key_service #11332) are now resolved.mrveiss commentedon Jul 9, 2026
Correction —
test_concurrent_limiteris NOT a py3.14 asyncio artifactRoot-caused on py3.12: the test fails at collection, not logic. Import chain
services.workflow_automation.concurrent_limiter→workflow_automation/__init__→manager→orchestratorhitImportError: cannot import name 'GemmaClassificationAgent' from 'orchestration.orchestrator_stubs'— a missing import fallback (fixed via #11392/#11390). It never reached the test body, so the 'py3.10/3.14 asyncio.Event' hypothesis was never actually exercised.After #11392, collection advances to the next blocker:
knowledge/base.py's unguardedfrom llama_index.core import ...(llama_index is in no requirements file) — tracked as #11391.test_concurrent_limiterneeds that chain resolved (guarded imports or conftest stubs), not a py3.14 environment.mrveiss commentedon Jul 9, 2026
Correction to my earlier note: the second
test_concurrent_limitercollection blocker I attributed to a 'latent llama_index issue (#11391)' was a mistake —llama-indexis a fully-declared prod dependency (requirements.txt:65-69), imported at startup. The localModuleNotFoundErroris just this dev venv missing ~54 deps, not a codebase gap. So in CI's full-dep env,test_concurrent_limitercollects fine after #11390 (the GemmaClassificationAgent fallback). Whether its logic (test_queue_still_waits) passes there is the only remaining open question — and backend pytest is not a gated CI job. #11391 closed as invalid.mrveiss commentedon Jul 15, 2026
Additional scoped-run failure found 2026-07-15 (while implementing #11719):
tests/api/test_require_permission.py::test_services_auth_imports_shared_permissionhas a brittle string-match assertion against theservices/auth.pyimport line — fails on a clean checkout even standalone, independent of the conftest-stub artifacts above. Should be fixed alongside this remediation (assert on behavior/import success, not source text).mrveiss commentedon Jul 15, 2026
One more scoped-run artifact found 2026-07-15 (while fixing #11748):
autobot-slm-backend/slm/.../health_collector_state_change_test.pyfails withModuleNotFoundError: No module named 'services.notification_service'— pre-existing, reproduces on clean base; same conftest/path-resolution family as the rest of this issue.mrveiss commentedon Jul 16, 2026
Cross-ref/update: #11737 (PR #11795, merged c7afcc7) repaired 33 tests/services runtime failures incl. sso_security and sync_orchestrator clusters, and #11794 (PR #11799, in CI) fixes the sweep collection errors — several of this issue's 10 files are likely now green. Re-verify against current base before working; remaining run-phase inventory lives in #11798.
mrveiss commentedon Jul 17, 2026
Filed #11833 with a measured instance of this family: collecting any autobot-slm-backend test alongside
autobot-backend/api/yields 44 collection errors (user_management.models.base ... not a package), because slm-backend's conftest pre-populates sys.modules session-wide with hollow stubs. The repo's existing workaround is the separate-process convention documented only inco-located-smoke.yml:95. Per-test isolation was attempted and discarded — the leak just moves to the next stub; the fix belongs in the conftest scoping this issue tracks.test: remediate scoped-failing services tests (#11248) (#12113)
github-actions commentedon Jul 22, 2026
PR #12113 (merged to
Dev_new_gui) references this issue with a close keyword.If this issue is fully resolved, close it manually. If work remains, no action is needed.
mrveiss commentedon Jul 22, 2026
Closed — merged via PR #12113 (
609901168). 3 files remediated (Redis-hang autouse mocks + conftest inert-patch real-load); 396 passed, 0 hangs. Test-infra follow-up #12114.