bug(ci): sys.modules leak guard fails on a clean Dev_new_gui — test_transcriber_pipeline leaks 4 transformers keys #14004

Activity

1. mrveiss

   added

   bugSomething isn't workingSomething isn't working

   ci

   on Aug 11, 2026
2. github-actions

   added

   infrastructure

   on Aug 11, 2026
3. mrveiss commented on Aug 30, 2026

   

   mrveiss

   on Aug 30, 2026

   OwnerAuthor

   More actions

   The premise has changed shape: it no longer fails, it warns — and that is the more interesting problem

   This issue reports the guard "fails on a clean Dev_new_gui". On current base it does not fail; it warns. Base CI is fully green right now (12/12 python-suite shards), while shard logs still carry:

   repo_tests/sys_modules_leak_guard.py:774: _SysModulesLeakWarning:
   sys.modules['chromadb'] was synthetic by autobot-backend/conftest.py and is STILL
   INSTALLED while pytest handles the collection of autobot-infrastructure
   

   So leaks are still happening and the suite is green anyway.

   Why, established from the source rather than inferred

   • The guard's only reporting path is _warn_once_per_owner → warnings.warn(..., _SysModulesLeakWarning) at :774. There is no raise, no assert, no pytest.fail on the leak path.
   • _SysModulesLeakWarning is never escalated to an error: it appears in no .ini, .toml, .cfg or workflow, and pytest.ini declares no filterwarnings section at all.

   So a new leak cannot fail a build, by construction.

   What the guard does enforce, which is the half that works

   repo_tests/sys_modules_leak_guard_session_test.py:370 asserts a stale baseline entry fails the run (returncode == 1). That is the down-only ratchet: an owner that stops leaking must leave the baseline in the same change, so the tolerance it held cannot be silently re-spent.

   The resulting asymmetry is worth stating plainly:

   situation	outcome
   baselined owner still leaking	tolerated (known)
   baselined owner stopped leaking, entry left behind	fails
   new, unbaselined owner leaks	warns only

   The baseline shrinks under pressure and cannot grow through the front door — but nothing stops a new leak arriving through the side. test_transcriber_pipeline is not in the baseline (25 entries; no transformers or test_transcriber_pipeline line), which is consistent: it never needed to be, because warning is the whole consequence.

   Criteria, re-read against this

   • AC1 — "the reproduction command exits clean on the base": not verifiable as written any more. The command will not exit non-zero regardless of whether the leak is fixed, so a clean exit would prove nothing. This AC needs restating as "emits no _SysModulesLeakWarning", or the guard needs to fail.
   • AC2 — "the baseline has not grown": holds. 25 entries; last touched by cc10f3e610 (fix(test): stop leaking empty package stubs into sys.modules (#14240) #14348), which removed leaks rather than adding tolerance.
   • AC3 — "a test proves the guard still catches a genuine leak — mutate to show it": partly served by the session test's stale-entry assertion, which is a real mutation rather than a re-run. It proves the ratchet catches a stale entry; nothing proves it catches a new leak, because catching is not what happens.

   The decision this needs

   Fixing the transformers leak closes the instance. It does not change that the next one warns. Escalating _SysModulesLeakWarning to an error is one line of filterwarnings — but it would turn every currently-warning leak into a failure at once, so it needs the same treatment the baseline already implements: escalate, and baseline the population that exists on the day it lands, down-only from there.

   That is a bigger change than this issue's title suggests, which is why I am recording it rather than doing it. Worth noting the cost of not doing it: this guard exists to prevent #13320's INTERNALERROR class of failure, and it currently reports that risk into a warnings summary nobody is required to read.
4. mrveiss

   added

   area: test-suite-healthWave 0 · cluster Q — Test suite healthWave 0 · cluster Q — Test suite health

   testing

   on Sep 1, 2026
5. mrveiss commented on Sep 2, 2026

   

   mrveiss

   on Sep 2, 2026

   OwnerAuthor

   More actions

   Correcting the 2026-08-30 triage comment on this issue — its central claim is wrong, and the conclusions drawn from it fall with it.

   That comment states: "The guard's only reporting path is _warn_once_per_owner → warnings.warn at :774. There is no raise, no assert, no pytest.fail on the leak path… So a new leak cannot fail a build, by construction."

   It is correct that there is no raise/assert/pytest.fail. That is precisely why it missed the gate, which is not any of those:

   • repo_tests/sys_modules_leak_guard.py:1058-1059 — if _verdict().failed: session.exitstatus = pytest.ExitCode.TESTS_FAILED
   • :1012-1013 — failed returns bool(self.new_owners or self.fixed_owners)
   • :1007 — new_owners is documented in the dataclass as "leaking and unlisted — a regression, fails the run"

   An unbaselined leak does fail the run. Verified directly against base.

   Consequently these statements in that comment are also wrong:

   • the verdict-table row "new, unbaselined owner leaks → warns only"
   • the AC1 restatement, "the command will not exit non-zero regardless"
   • the AC3 assessment, "nothing proves it catches a new leak"

   AC3 is in fact met: repo_tests/sys_modules_leak_guard_session_test.py:321 test_an_unlisted_leak_fails_the_run asserts returncode == 1, with :410 repeating it for a baselined owner and :448 under xdist. These spawn real subprocess pytest sessions against a scratch repo carrying a genuinely leaking conftest — actual mutation, not a re-run, and not satisfiable by an empty match.

   The comment's one accurate observation — base CI is green while chromadb leak warnings appear — is consistent rather than contradictory: autobot-backend/conftest.py is baseline entry #2, so it resolves to known_owners and is reported only.

   The issue's own premise is now stale

   The title says the guard fails on a clean Dev_new_gui. The guard-side exemption for exactly the four reported keys has been in base since 2026-08-05 — six days before this issue was filed:

   • :622 _parent_is_genuine names transformers==5.14.1 removing tokenization_utils, tokenization_utils_fast and image_processing_utils_fast — the three was-synthetic keys
   • :298 _is_self_rebind covers the fourth, the _LazyModule rebind
   • :741 _now_exempt re-checks classification mid-import, which is what makes the exemption reachable at all

   Landed in da1d38ef07 / 0bcef05e61 for #13450/#13599.

   Corroborating: test_transcriber_pipeline.py is not on the baseline, so if it still leaked, base CI would be red rather than green.

   Where that leaves the checklist

   AC	status
   reproduction exits clean on base	UNVERIFIABLE from a diff — needs a run. The mechanism is gone; someone should run it and record the result.
   baseline has not grown	MET — 25 non-comment lines, matching the issue body byte-for-byte. The only commit touching it since filing is cc10f3e610 (2026-08-17), 1 deletion(-). Every prior commit also only deletes.
   a test proves the guard still catches a genuine leak, by mutation	MET — the three session tests above.

   Not closing: the first criterion needs an observed run, which is the one thing a diff cannot supply. Everything else is satisfied.
6. mrveiss

   modified the milestones: Backlog, v0.15.0 on Sep 12, 2026