tech-debt(ci): the connectors package __init__ imports every connector eagerly, so each one's third-party dep lands in the migration-gate install list

Problem

knowledge/connectors/__init__.py imports every connector module at package import time. Anything
that touches knowledge.connectors.credential_store therefore pulls in all connectors and all of
their third-party imports, whether or not the caller uses them.

The migration-gate job installs a deliberately minimal, hand-maintained pip list. Each time a
migration test reaches that import chain, one more connector's dependency is discovered the hard
way — as a red CI run on an unrelated PR:

Discovered on	Missing package	Needed by
#16444	`aiohttp`	`knowledge/connectors/gdrive.py:32`
#17134	`defusedxml`	`knowledge/connectors/nextcloud.py:32`

Both were fixed by appending to the install list at .github/workflows/migration-gate.yml. That
keeps working only until the next connector gains a dependency, and the failure mode is a red check
on whichever PR happens to be in flight — not on the change that introduced the coupling.

Root cause

The eager package __init__. A migration test that wants _create_system_vault_secret has no way
to avoid importing WebDAV XML parsing and Google Drive HTTP.

Fix direction

Make the connector imports lazy so importing credential_store does not import every connector. A
registry that resolves a connector on first use, rather than at package import, is the shape the
rest of the codebase already uses for detectors.

Acceptance criteria

Importing knowledge.connectors.credential_store does not import any concrete connector
module, proved by a test that asserts the connector modules are absent from sys.modules
after the import.
The migration-gate install list no longer needs connector-specific packages; aiohttp and
defusedxml are removed from it and the job stays green.
A guard fails if a new eager import is added back to knowledge/connectors/__init__.py.

Notes

Found while fixing #17134's red migration-tests. The immediate unblock (adding defusedxml)
shipped on that PR; this issue is the recurrence, not that one instance.

ci(migration-gate): install defusedxml, which the eager connector pac…

ci(migration-gate): add jsonschema and prometheus_client, found by wa…

Third package found on this path tonight: jsonschema, and a fourth predicted and pre-empted: prometheus_client.

defusedxml (the connectors chain) was not the last one. The next red run failed on jsonschema, reached through a completely separate eager chain — middleware/__init__.py → plugin_sdk/__init__.py → loader.py. So this is not specific to knowledge/connectors; it is the general pattern of eager package __init__ files, and the issue title understates it.

Rather than keep appending one package per ~6-minute CI cycle with no bound on how many remain, the import graph was walked statically from api/secrets.py, following module-level imports only, resolving relative imports, and importing every ancestor package __init__ the way Python does. That reports exactly three unguarded third-party names on this path: jsonschema, prometheus_client, and starlette (already installed transitively with fastapi).

Worth recording, because two earlier versions of that walk were confidently wrong:

following imports nested inside functions reported 25 gaps, nearly all false — a function-level import does not run at import time;
skipping relative imports missed jsonschema itself, the very failure it was written to find, and reported a clean two-gap result.

Both failure modes produced a plausible, specific-looking answer. Anyone re-running this check should verify it reproduces a known failure before trusting its negatives.

Scope of that result is api/secrets.py's import graph, not the whole test suite. Other migration tests may reach other chains.

chore(guards): re-pin hooks-path-override floor 6481 -> 6484 (#17138)

Merge pull request #17147 from mrveiss/vehicle-v090-2026-09-20-batch

Verified against merged main (8830293, post-#17147 consolidation).

.github/workflows/migration-gate.yml pip install list, lines ~156-159:

"alembic>=1.13" "sqlalchemy[asyncio]>=2.0" asyncpg aiosqlite \
pydantic pydantic-settings \
bcrypt pyjwt redis httpx aiofiles python-dotenv pyyaml \
cryptography fastapi structlog aiohttp prometheus_client \
pytest pytest-asyncio

defusedxml — absent (was only needed by nextcloud.py's now-lazy import, chore(vehicle): security consolidation train — #17113, #16444, #17088, #17095 in one CI run (#17048) #17134).
jsonschema — absent (was only needed by plugin_sdk/loader.py's now-lazy Draft202012Validator binding).
aiohttp, prometheus_client — still present, correctly: the accompanying comment (lines 130-146, also intact on main) documents these are reached via a different, untouched eager-import chain (autobot_shared.security→ssrf_guard and autobot_shared.monitoring→prometheus_metrics), tracked separately as tech-debt(ci): autobot_shared.security and autobot_shared.monitoring eager __init__ still pull aiohttp/prometheus_client into migration-gate #17143 rather than silently dropped by this fix's scope.

Consolidation did not revert or alter this list. Criterion verified by reading merged source directly (git show origin/main:...), not by a local run.

Correction: an earlier version of this comment additionally cited a local pytest re-run as though it were part of this criterion's evidence. This project's standing rule treats a local run from a checkout as a design aid only, never evidence in a closing comment (the environment differs from CI/deployment). Retracted — the migration-gate.yml content check above is file-read evidence and stands on its own.

fix(ci): migration-gate installs cachetools — the import-graph walk t…

refactor(forks): retire fifteen forks across eight issues, each onto …

Navigation Menu

Repository navigation

tech-debt(ci): the connectors package init imports every connector eagerly, so each one's third-party dep lands in the migration-gate install list #17138

Problem

Root cause

Fix direction

Acceptance criteria

Notes

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

tech-debt(ci): the connectors package __init__ imports every connector eagerly, so each one's third-party dep lands in the migration-gate install list #17138

Description

Problem

Root cause

Fix direction

Acceptance criteria

Notes

Activity

mrveiss commented on Sep 20, 2026

mrveiss commented on Sep 20, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

tech-debt(ci): the connectors package init imports every connector eagerly, so each one's third-party dep lands in the migration-gate install list #17138