test-infra(leak-guard): base branch red — 4 transformers keys misattributed to a test that references neither transformers nor sys.modules

Symptom

The python-suite job fails on Dev_new_gui — all 12 shards, current base head. The failure is not a test failure. Both pytest invocations in the shard are green:

======== 2070 passed, 2125 skipped, 2692 warnings in 102.50s (0:01:42) =========
====================== 497 passed, 38 warnings in 28.21s =======================

The non-zero exit comes from the leak guard:

============================ sys.modules leak guard ============================
LEAK: autobot-backend/api/multimodal_response_test.py leaves 4 sys.modules key(s)
      installed outside autobot-backend/api/ — first seen at ...
      keys: transformers.tokenization_utils_fast, transformers.tokenization_utils,
            transformers.image_processing_utils_fast, transformers...
1 file(s) above are NOT on repo_tests/sys_modules_leak_baseline.txt, so this run fails.

The accused file cannot be the leaker

autobot-backend/api/multimodal_response_test.py contains no reference to transformers and no reference to sys.modules. Its only test double is unittest.mock.MagicMock, used as a plain object:

grep -n "transformers\|sys.modules" autobot-backend/api/multimodal_response_test.py
# -> no matches for either

Run on its own it is clean and the guard says nothing:

python3 -m pytest autobot-backend/api/multimodal_response_test.py -q -p no:randomly
======================== 7 passed, 36 warnings in 1.60s ========================

So the attribution is wrong: the keys are installed by some other node, and the file that happens to own the snapshot boundary when the delta is first observed gets named.

Why this is the guard's bug, not the file's

_LeakGuard.snapshot() attributes the whole delta since the last snapshot to the node currently being visited. Any key installed while a sibling ran, but first observed at this node's boundary, lands on this owner. The module docstring already documents one instance of this failure mode — an ancestor Dir collector "stole the first seen at slot from the sibling that actually inherited the stub, discarding the actionable half of the report" — and that fix exempted ancestor directories only. This is the same defect between siblings, which that fix does not cover.

It is also composition-dependent: pytest-split re-composes shards whenever durations change, so which node owns the boundary moves between runs. That makes the gate intermittent — it will appear to "fix itself" and come back.

The wrong fix

Adding multimodal_response_test.py to repo_tests/sys_modules_leak_baseline.txt. The baseline is shrink-only and exists to record real owners; baselining a misattributed file both launders a real leak somewhere else and permanently blames an innocent module. The named file has nothing to fix.

What is actually needed

Attribute a mutation to the node that was executing when the key appeared, not to the node that owns the next snapshot boundary — or, where that is not recoverable, report the candidate set instead of picking one and asserting it.
Identify the real owner of the four transformers.* keys. autobot-backend/conftest.py is already baselined at 47 keys and stubs package parents (conftest.py:63,102,106), which makes it the first place to look.
Until attribution is trustworthy, the guard should not be able to red the base branch on a file that has no way to fix itself.

Blast radius

python-suite is not a required check (#13162), so this does not block merges — but it does mean the base-branch signal restored by #13432 reads red for a reason unrelated to any test, which is exactly the confusion #13432 set out to remove.

Guard introduced in #13398 / #13442. Related: #13361 (conftest leaks 62 keys), #13450 (_classify refinement), #13554 (orphan checker is regex-based).

Narrowing the cause — the four keys are almost certainly real transformers submodules, not stubs

Investigated further. This changes what the fix should be, so recording it before anyone starts on the attribution rework.

Nothing in the repo creates those keys. A repo-wide search for the flagged names returns nothing:

grep -rn "tokenization_utils\|image_processing_utils" --include=*.py .   # no matches

They are also absent from both stub lists in autobot-backend/conftest.py (_SIMPLE_STUBS at :137 — prometheus_client, xxhash, torch, torch.nn, torch.cuda, asyncpg, psycopg2, alembic; _PKG_STUBS at :1132 — sqlalchemy.* and alembic.*). No conftest asks for transformers.

But transformers is installed in CI — requirements-ci/ai-ml.txt:4 pins transformers==5.14.1. So these are that package's own submodules, appearing through an ordinary lazy import.

Why a real module still trips the guard

_is_synthetic is a single attribute probe:

def _is_synthetic(module: object) -> bool:
    """True for hand-built module stubs and mocks, false for real imports."""
    return getattr(module, "__spec__", None) is None

transformers builds its public surface out of _LazyModule objects rather than importing submodules eagerly. A lazily-materialised submodule registered in sys.modules without a populated __spec__ reads as __spec__ is None — which is exactly the stub signature. From there _shadows_real_code("transformers.tokenization_utils") is true (the package is installed), so the entry is reported as a leak.

That makes the docstring's premise wrong for this family: __spec__ is None does not mean "hand-built", it means "not created by the normal import machinery" — and a third-party lazy-import system is the other thing in that set. The _shadows_real_code guard was designed for the opposite case (cffi/Cython pseudo-modules that shadow nothing); here the name does resolve to something installed, so it passes.

Status: unverified. transformers is not installed in this environment and I am not adding it — optional test deps are importorskip-gated by policy, not ad-hoc installs. The one-line check for anyone who has it:

import transformers.tokenization_utils as m, sys
print(type(m), getattr(sys.modules['transformers.tokenization_utils'], '__spec__', None))

If __spec__ is None, this is confirmed and the misattribution in the original report is a symptom, not the cause.

What this changes about the fix

The attribution rework described above is still worth doing — blaming a file that contains neither transformers nor sys.modules is bad either way. But it would not fix this failure: correct attribution would simply blame whichever module first touched transformers, and that module is not leaking anything. The guard would still red the base branch.

The narrower fix is in _classify / _is_synthetic: treat a module as synthetic only when it is also not a real importable package under that exact name, or exempt entries whose object is a ModuleType subclass originating from an installed distribution. #13450 is already open on _classify refinement and needs a decision — this is concrete input for it.

Cross-referencing #13450 and #13361.

Persisting across every base run — and not caused by the day's merges

Re-checked on a later base head (203c69de6, run 30951203696) after four more PRs landed. Identical failure, same file, same four keys:

2070 passed, 2125 skipped   in 92.64s
 497 passed                 in 27.13s
FAILED test count: 0
LEAK: autobot-backend/api/multimodal_response_test.py leaves 4 sys.modules key(s) installed outside autobot-backend/api/

Three consecutive AutoBot CI/CD Pipeline runs on Dev_new_gui have now failed this way (cc4c9f585, 203c69de6, and the current head still running). Zero test failures in every one — the non-zero exit is the guard alone. So the base-branch signal that #13432 restored reads red for a reason unrelated to any test, which is precisely the confusion #13432 existed to remove.

Why I have not just fixed it

The obvious patch — stop treating a __spec__-less entry as a stub when the name resolves to an installed package — would break the guard's core purpose. _shadows_real_code deliberately requires exactly that condition: a stub is dangerous because it shadows something real. The three historical leaks it was built to catch (sqlalchemy, autobot_shared, agents) all shadow installed packages, and every one of them would stop being detected.

So the correct fix has to separate two things that currently look identical:

a test that installs a MagicMock over an installed package and leaves it behind — must stay caught
an installed package's own lazy-import machinery registering a __spec__-less submodule of itself — must be ignored

A plausible discriminator is ownership: a real package's lazy submodule is an object whose parent package is the genuine distribution, whereas a leaked stub is not. That is a design decision with real consequences either way, which is why this belongs on #13450 rather than being patched unilaterally at 1am.

What is needed from you

One decision on #13450, then the fix is small. Until then the base branch stays red on every merge, and python-suite remains unusable as a signal — which also blocks measuring #13162.

The hypothesis in my previous comment (transformers _LazyModule submodules carrying __spec__ is None) is still unverified — transformers is not installed here and optional test deps are importorskip-gated by policy rather than installed ad hoc. The one-line check is in that comment.

Hypothesis CONFIRMED, and the cause is worse than "lazy import"

Found transformers==5.14.1 already installed in a local CI-shaped venv, so this is now measured rather than guessed — no ad-hoc install involved.

transformers 5.14.1
transformers.tokenization_utils:            type=module __spec__=False
transformers.tokenization_utils_fast:       type=module __spec__=False
transformers.image_processing_utils_fast:   type=module __spec__=False

So _is_synthetic (__spec__ is None) answers True for all three, and _shadows_real_code passes because the names are importable — hence the leak report.

But they are not lazy-loaded real modules. The files do not exist at all:

site-packages/transformers/tokenization_utils.py            MISSING
site-packages/transformers/tokenization_utils_fast.py       MISSING
site-packages/transformers/image_processing_utils_fast.py   MISSING

transformers 5.x removed these modules and installs synthetic backward-compatibility shims into sys.modules under the old names. They are, by every property the guard can inspect, exactly what a hand-built stub is — because that is essentially what they are, just authored by the library instead of by a test.

No attribute can separate them — I tried

transformers (parent)          spec=True   file_on_disk=True   type=_LazyModule
transformers.tokenization_utils spec=False  file_on_disk=False  type=module
MagicMock stub                  spec=False  file_on_disk=False  type=MagicMock
hand-built ModuleType stub      spec=False  file_on_disk=False  type=module

__file__, __loader__, __spec__ and object type all fail to discriminate: the shim is byte-for-byte indistinguishable from types.ModuleType('x'). Any attribute-based patch either keeps the false positive or stops catching real stubs.

A discriminator that does work: is the parent genuine?

A library's own shim always sits under a real, spec-carrying parent package. A leaked stub replaces the package itself.

def parent_is_genuine(name: str) -> bool:
    parent, _, _leaf = name.rpartition(".")
    if not parent:
        return False
    p = sys.modules.get(parent)
    return p is not None and getattr(p, "__spec__", None) is not None

Measured against both populations:

entry	parent_genuine	outcome
`transformers.tokenization_utils`	True	ignored ✓ (the false positive)
`transformers.tokenization_utils_fast`	True	ignored ✓
`transformers.image_processing_utils_fast`	True	ignored ✓
`sqlalchemy`	False	still caught ✓
`sqlalchemy.orm`	False	still caught ✓
`agents`	False	still caught ✓
`agents.core`	False	still caught ✓

All three historical leaks stay detected, because a test stubbing a package replaces the top-level entry, which drops the parent's __spec__.

The blind spot, stated plainly

This trades away one case — a test that stubs only a submodule while leaving the real parent intact:

json.decoder  parent_genuine=True  -> would be IGNORED

That is a real leak the guard would stop catching. Whether it is an acceptable trade depends on whether anything in this repo stubs a child without its parent; the two stub lists in autobot-backend/conftest.py do not (_PKG_STUBS always installs the parent alongside sqlalchemy.ext, sqlalchemy.orm, …), so on current evidence the exchange costs nothing real and buys back a green base branch.

That is the decision for #13450 — I am not making it unilaterally. But it is now a concrete either/or with measurements on both sides rather than an open question.

fix(test-infra): stop the leak guard reporting a real package's own s…

PR #13620 (merged to Dev_new_gui) references this issue with a close keyword.

fix(test-infra): stop the leak guard reporting a real package's own shims (#13450, #13599)

If this issue is fully resolved, close it manually. If work remains, no action is needed.

Fixed — landed as 0bcef05e6 (PR #13620).

The four transformers.* keys are no longer reported. Root cause, measured: transformers==5.14.1 removed tokenization_utils, tokenization_utils_fast and image_processing_utils_fast and installs spec-less compatibility objects under the old names — the files are not on disk. So _is_synthetic was correct about them; they are genuinely synthetic, just authored by the library rather than a test.

No attribute of the entry could separate the two cases (__spec__, __file__, __loader__ and object type are identical to types.ModuleType("x")), which is why the obvious __file__ patch would not have worked. Ownership of the parent does separate them, and the owner chose that fix from four options.

Verified against both populations with real transformers installed:

transformers.tokenization_utils       exempt=True
transformers.tokenization_utils_fast  exempt=True
sqlalchemy / sqlalchemy.orm           exempt=False
agents / agents.core                  exempt=False
autobot_shared                        exempt=False

The misattribution this issue reported is a symptom of the same cause and goes with it. The accepted trade — a test stubbing only a submodule while leaving the real parent intact is no longer caught — is recorded in the code and pinned by a renamed test, with a new sibling test asserting that a stubbed parent still drags its children into the report.

Correction — my fix did not resolve this. Reopening.

I closed this on PR #13620 saying the four keys would no longer be reported. They still are. Base run 31001085706 on d94f032f9, which contains 0bcef05e6:

============================ sys.modules leak guard ============================
LEAK: autobot-backend/api/multimodal_response_test.py leaves 4 sys.modules key(s) …
      keys: transformers.tokenization_utils_fast, transformers.tokenization_utils,
            transformers.image_processing_utils_fast, transformers
2070 passed, 2125 skipped … FAILED tests: 0

What I got wrong

transformers itself is one of the four keys. I read the earlier report as three submodules and did not notice the top-level entry in the list. That single fact invalidates the diagnosis:

My exemption requires the parent to be untracked. Here the parent is tracked, so _parent_is_genuine correctly returns False and the exemption never applies — by design.
The "library's own lazy shim" story cannot hold either. requirements-ci.txt:14 pulls in requirements-ci/ai-ml.txt, which pins transformers==5.14.1, so transformers is genuinely installed in this job. A real installed package does not appear as a suspicious top-level mutation.

So something replaces or synthesises transformers itself during the shard, and the three submodules follow it. That is a real leak of exactly the kind this guard exists to catch — not a false positive.

My verification was the flaw: I tested _parent_is_genuine in isolation against a cleanly imported transformers and confirmed the predicate did what I intended. I never reproduced the CI classification path, where the parent is tracked and the entries reach _classify differently. Testing the predicate is not testing the fix.

Is #13620 still correct?

The change itself stands on its own terms — a new spec-less child of a genuinely untouched package should not be reported, both new tests pass, and no existing detection was lost. It simply does not address this failure. I am not reverting it; I am reopening this issue because the base branch is still red for the original reason.

Identify what installs a transformers entry during shard 1 — the two _make_transformers_mock() helpers in llm_shared/optimization/ pass mocks as function arguments and never touch sys.modules, so it is not those. Once found, the fix is the ordinary one this guard asks for: install and remove the stub in the same try/finally, scoped to the nodes that need it.

Root cause — reproduced locally in 2.4s, and it is not a repo leak

pytest autobot-backend/tests/test_mmr_diversity.py autobot-backend/api/multimodal_response_test.py
LEAK: …multimodal_response_test.py leaves 4 sys.modules key(s) …
      keys: transformers.tokenization_utils_fast, transformers.tokenization_utils,
            transformers.image_processing_utils_fast, transformers
31 passed

Inspecting the entries after that test module imports:

transformers:                    type=_LazyModule  spec=True   file=True
transformers.tokenization_utils: type=module       spec=False  file=False

The parent is real — a spec-carrying _LazyModule, and importing it registers 371 transformers.* entries. Only four are reported, which is the tell.

Ruled out by measurement, not assumption:

autobot-backend/conftest.py does not stub transformers — imported it directly and the key is absent afterwards.
The two _make_transformers_mock() helpers in llm_shared/optimization/ pass mocks as function arguments; neither touches sys.modules.
requirements-ci.txt:14 pulls ai-ml.txt, so transformers==5.14.1 is genuinely installed in this job.

So the four break down as:

key	why the guard tracks it
`transformers`	replaced — the package rebinds its own `sys.modules` entry to a `_LazyModule` during init, which is a different object than the one the import machinery first installed
the three children	synthetic — spec-less compat shims for modules 5.x removed, under a parent that is now tracked

That is why #13620 did not help: my exemption requires an untracked parent, and the self-replacement puts the parent in _tracked first.

What the fix has to be

Exempt a replacement whose new object is a genuine module re-registering itself — a real ModuleType carrying a __spec__ whose __name__ equals the key. A library doing its own lazy-init rebind satisfies that; a test swapping in a MagicMock does not, because the mock has no valid __spec__. Checked against the three historical leaks: autobot_shared → bare MagicMock, sqlalchemy → MagicMock, agents → types.ModuleType with no spec — all still caught.

With the parent no longer tracked, the child exemption from #13620 then applies to the three shims on its own.

I will implement and verify against the local reproduction above rather than against the predicate in isolation — that isolation test is exactly what let the last attempt through.

Progress, and a regression in my own fix — not pushed

The original failure is fixed locally. The 2.4s reproduction is clean:

pytest autobot-backend/tests/test_mmr_diversity.py autobot-backend/api/multimodal_response_test.py
31 passed          # no LEAK line — previously 4 keys reported

Two changes were needed, and the second is the one I had missed twice:

_is_self_rebind — a package re-registering itself under its own name (real ModuleType, populated __spec__, __name__ == key) is not a swap. This clears the transformers parent.
A report-time re-check. Classification runs at the snapshot that first sees a key, and for a transformers.* shim that is mid-import, before the parent has bound its _LazyModule. So the parent-genuine exemption could not apply at classification time, and a tracked mutation was never revisited. That is why fixing _classify alone moved the count 4 → 3 and no further.

The regression: re-checking _is_synthetic(module) and parent_is_genuine(key) at report time also silences leaks that were previously reported. Measured on the pair from #13457:

before: 2 known leaking file(s), 4 sys.modules key(s)   (reported as known/baselined)
after:  nothing reported

Those are baselined owners, so they were not failing — but they were being reported, and the shrink-only baseline depends on that reporting to know when an owner is fixed. Silencing them would eventually fail the removal check and, worse, hide the real leaks the guard exists for.

So the report-time re-check is too broad as written. The next step is to establish exactly which keys those two owners leak and why they satisfy parent_is_genuine, then narrow the re-check — most likely to the self-rebind case plus a check tied to the specific parent that rebound itself, rather than any genuine parent.

Nothing is pushed. The branch issue-13599 holds the work in progress; Dev_new_gui is unchanged and still red for the original reason.

fix(test-infra): scope the leak-guard exemptions to third-party packa…

PR #13633 (merged to Dev_new_gui) references this issue with a close keyword.

fix(test-infra): scope the leak-guard exemptions to third-party packages (#13599)

If this issue is fully resolved, close it manually. If work remains, no action is needed.

test-infra: drop the slm api conftest from the leak baseline, it no l…

Fixed and verified — closing with evidence

Fixed across PR #13620, PR #13633 and PR #13636, all squash-merged to Dev_new_gui. Closed manually because Closes #N does not fire on this branch.

The reproduction this issue specified now passes on clean base (6fa09d97f), in a worktree with nothing of mine applied:

python3 -m pytest -q -p repo_tests.sys_modules_leak_guard \
  autobot-backend/tests/test_mmr_diversity.py autobot-backend/api/multimodal_response_test.py
31 passed

No guard report, no misattribution. And the base branch's own CI confirms it — the latest run's python-suite shard 3/12 log contains zero mentions of transformers, where this issue's four keys used to appear.

What the fix turned out to be

The diagnosis took three attempts, and the first two were wrong in ways worth recording:

Exempt spec-less children of a genuine parent — never fired, because transformers itself was one of the four tracked keys, so the parent was never "genuine" by that test.
Re-check at report time — went 4 keys → 3, but silenced two baselined owners that had to stay reported. Not shipped.
What landed: classify by file location. transformers 5.14.1 replaces its own sys.modules entry with a _LazyModule during import and registers spec-less compatibility shims for submodules 5.x removed. No attribute distinguishes those from a test stub — __spec__, __file__, __loader__ and object type all fail identically. Only "loaded from outside this repository" separates them, which is the correct rule anyway: a guard on this repo's stubs has no business policing a third-party package's own bookkeeping.

_is_self_rebind() additionally requires the previous object to have been a real module, so repo code re-importing a real module over a real module is not mistaken for a library self-rebind.

#13636 then removed autobot-slm-backend/api/conftest.py from the baseline, which had stopped leaking — the baseline shrinks, never grows.

Not closed by this

The base is still red on python-suite shard 6/12, but for a different cause: three files under autobot-backend/services/knowledge/ leak utils.chromadb_client / utils.async_chromadb_client. That is #13651, not this issue — the keys, the owners and the mechanism all differ. #13450 remains open for the five owners the two CI path shapes hide.

Navigation Menu

Repository navigation

test-infra(leak-guard): base branch red — 4 transformers keys misattributed to a test that references neither transformers nor sys.modules #13599

Symptom

The accused file cannot be the leaker

Why this is the guard's bug, not the file's

The wrong fix

What is actually needed

Blast radius

Narrowing the cause — the four keys are almost certainly real transformers submodules, not stubs

Why a real module still trips the guard

What this changes about the fix

Persisting across every base run — and not caused by the day's merges

Why I have not just fixed it

What is needed from you

Hypothesis CONFIRMED, and the cause is worse than "lazy import"

No attribute can separate them — I tried

A discriminator that does work: is the parent genuine?

The blind spot, stated plainly

Correction — my fix did not resolve this. Reopening.

What I got wrong

Is #13620 still correct?

Next

Root cause — reproduced locally in 2.4s, and it is not a repo leak

What the fix has to be

Progress, and a regression in my own fix — not pushed

Fixed and verified — closing with evidence

What the fix turned out to be

Not closed by this

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

test-infra(leak-guard): base branch red — 4 transformers keys misattributed to a test that references neither transformers nor sys.modules #13599

Description

Symptom

The accused file cannot be the leaker

Why this is the guard's bug, not the file's

The wrong fix

What is actually needed

Blast radius

Activity

mrveiss commented on Aug 4, 2026

Narrowing the cause — the four keys are almost certainly real transformers submodules, not stubs

Why a real module still trips the guard

What this changes about the fix

mrveiss commented on Aug 4, 2026

Persisting across every base run — and not caused by the day's merges

Why I have not just fixed it

What is needed from you

mrveiss commented on Aug 4, 2026

Hypothesis CONFIRMED, and the cause is worse than "lazy import"

No attribute can separate them — I tried

A discriminator that does work: is the parent genuine?

The blind spot, stated plainly

github-actions commented on Aug 5, 2026

mrveiss commented on Aug 5, 2026

mrveiss commented on Aug 5, 2026

Correction — my fix did not resolve this. Reopening.

What I got wrong

Is #13620 still correct?

Next

mrveiss commented on Aug 5, 2026

Root cause — reproduced locally in 2.4s, and it is not a repo leak

What the fix has to be

mrveiss commented on Aug 5, 2026

Progress, and a regression in my own fix — not pushed

github-actions commented on Aug 5, 2026

mrveiss commented on Aug 6, 2026

Fixed and verified — closing with evidence

What the fix turned out to be

Not closed by this

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions