Repository navigation
bug(tests): 2 pre-existing failures in test_claim_classifier — duplicate module identity breaks isinstance + MagicMock awaited #12839
Description
Problem
Two tests in autobot-backend/tests/services/test_claim_classifier.py fail on Dev_new_gui, independent of any PR:
FAILED TestIndividualClassification::test_classify_claim_basic
FAILED TestBatchClassification::test_batch_classify_single_claim
2 failed, 41 passed
Verified pre-existing by checking out the untouched file from origin/Dev_new_gui and re-running — same two failures.
Two distinct defects
1. Dual module identity — isinstance fails against a matching object
tests/services/test_claim_classifier.py:332: assert isinstance(result, Claim)
E AssertionError: assert False
E + where False = isinstance(Claim(claim_text='The service is running', claim_type=<ClaimType.FACTUAL: 'factual'>, ...), Claim)
The repr is a Claim, but isinstance says no — so services.knowledge_grounding_models has been loaded twice under two module identities, producing two distinct classes of the same name. Suspected interaction with the services package stub in conftest.py:~265, which swaps the real package for a stub and then re-points __path__ at the real directory; a module reachable both through the stub and through a direct file load ends up duplicated.
2. Sync MagicMock awaited
ERROR claim_classifier:claim_classifier.py:384 KB search failed for claim
'The service is running': object MagicMock can't be used in 'await' expression
The KB search mock is a MagicMock where the code awaits it. The classifier catches the exception and returns early, so the test's real assertion never gets a chance — the failure is masked into a wrong-type result rather than an obvious error. AsyncMock is imported in the same test file already, so this looks like an oversight.
Why it was not caught
_real_load_and_bind / the services stub were introduced to work around services/__init__.py being import-heavy — the exact problem fixed in #12830. With re-exports now lazy, the stub's original rationale is gone, so narrowing or removing it is worth evaluating as the real fix for defect 1 rather than patching the symptom.
Scope
Deliberately not folded into #12830: conftest.py is loaded by the entire backend suite, so changing the stub has repo-wide blast radius and needs its own verification pass.
Done when
- Both tests pass without weakening their assertions (
isinstancecheck stays). - The KB-search mock is awaitable, so a genuine classifier failure surfaces as a failure rather than a type mismatch.
- Whatever causes the duplicate module identity is fixed at the source, with a note on whether the
servicesconftest stub can now be narrowed post-bug(services): services/__init__ eagerly imports aiohttp via ai_stack_client #12830.
Activity
fix(tests): conftest re-executed already-loaded modules, breaking isi…
github-actions commentedon Jul 28, 2026
PR #12851 (merged to
Dev_new_gui) references this issue with a close keyword.If this issue is fully resolved, close it manually. If work remains, no action is needed.
mrveiss commentedon Jul 28, 2026
Fixed in PR #12851 — squash-merged to
Dev_new_guias8dcf6f03aa554e5b575062434192ccdcef766aba.Two independent defects, not one. Fixing the obvious one left both tests still red, which forced the second investigation.
1. The KB mock lied about its own interface.
ClaimClassifier._search_kbbranches onhasattr(self.kb, "search_async"), and the fixture was a bareMagicMock()— which auto-creates every attribute. So the classifier took the async branch and awaited a plain MagicMock ("object MagicMock can't be used in 'await' expression"), caught it, and returned no results. The test was asserting against a degraded error path instead of the one it meant to exercise.MagicMock(spec=["search"])makes thehasattrprobe answer honestly.2. The same module was executed twice. A stack probe during a real pytest run showed both executions:
services/claim_classifier.py:31importing.knowledge_grounding_modelsat collection, thenconftest.py:1141 pytest_configure -> _real_load_light_services()re-executing the same file via_real_load_and_bind. That built a second set of class objects and swapped them intosys.moduleswhileclaim_classifierkept referencing the first — henceisinstance(x, Claim)returning False for an object whose repr says it IS aClaim, with only onesys.modulesentry to inspect.Re-execution exists only to replace a MagicMock stub, so a module already real-loaded from that same path is now left alone. The parent bind is still re-applied on that skip path —
patch("pkg.mod.NAME")resolves viagetattr(pkg, "mod"), and dropping it would resurrect the inert-patch bug from #11532/#11618. That case has its own test.Verification —
conftest.pyis loaded by every backend test, so this was measured against a pristineorigin/Dev_new_guiworktree with an identical command:The failure sets were diffed, not just the counts — the remaining 12 are a strict subset of the baseline's 14 (gitlab/gitea connector tests needing a live Redis, etc.). The +6 passes are the 2 fixes plus 4 new tests pinning the loader invariant.
Note on the first verification attempt: running
tests/ llc/tests/together reported "Interrupted: 49 errors during collection", which looked like this change breaking the suite. The identical command on the pristine baseline produced exactly the same 49 errors — pre-existing, triggered by combining those paths, and it meant nothing had executed. The numbers above come from a subset that collects cleanly on both trees.fix(tests): drop the code_generation conftest stub so llm_code_genera…