bug(deps): SQLAlchemy 2.1.0 breaks the SLM backend — an unbounded floor took a minor bump and 27 tests fail with InternalTraversal

Verdict

SQLAlchemy 2.1.0 released, and the tree's >=2.0.54 floors let it in alongside a ==2.0.54 pin. Two versions are installed into the same job, the second overwriting the first, and the result is 27 test failures with TypeError: 'InternalTraversal' object is not callable.

This is the same upstream event as #17432/#17433's missing greenlet — one SQLAlchemy release, two distinct defects.

Evidence

From python-suite shard 10/12, both versions installed in one job:

01:35:26  Using cached sqlalchemy-2.0.54-cp314-...whl (3.4 MB)
01:37:02  Successfully installed ... (first install completes)
01:37:18  Downloading sqlalchemy-2.1.0-cp314-...whl.metadata (10 kB)
01:37:22  Downloading sqlalchemy-2.1.0-cp314-...whl (4.6 MB)
01:37:33  Successfully installed ... (second install completes)

The declared constraints disagree:

autobot-backend/requirements.txt:27                          sqlalchemy>=2.0.54
autobot-slm-backend/requirements.txt:31                      sqlalchemy>=2.0.54
autobot-infrastructure/.../ai-stack/requirements-ai.txt:81   sqlalchemy>=2.0.54
requirements-ci/storage.txt:4                                sqlalchemy==2.0.54   <- pinned

Three floors admit 2.1.0; one pin demands 2.0.54. Sequential pip install -r runs resolve independently, so the later one upgrades what the earlier one pinned. InternalTraversal is SQLAlchemy internals — the signature of a partially-replaced install, not of a test bug.

Why it surfaced only now, and why #17433 looks like the cause

It is not #17433's regression. Compare the same shard:

main        d9881afa1   shard 10/12:   71 passed, 17 errors,  0 InternalTraversal
PR #17433   856f6f508   shard 10/12:  123 passed, 27 failed, 27 InternalTraversal

On main, greenlet is missing, so those tests die at collection — 17 errors, and the SQLAlchemy conflict is never reached. #17433 supplies greenlet, collection succeeds, and the tests get far enough to hit the version conflict.

#17433 did not introduce this. It unmasked it — the greenlet fix converted 17 silent collection errors into 27 real failures that were always going to be there.

main is independently red on python-suite at d9881afa1, which confirms the base was already broken before #17433 existed.

Acceptance criteria

AC1 Establish which install step pulls 2.1.0 and which pulls 2.0.54. The log shows both; I did not trace each to its originating requirements file, and the fix should not be chosen before that is known.
AC2 One SQLAlchemy version resolves per environment. Whether that means raising the CI pin to 2.1.0, capping the service floors below it, or making the pin authoritative is an owner-level call — 2.1.0 is a minor-version bump with its own migration surface.
AC3 python-suite green on main, all 12 shards, with the InternalTraversal count at zero.
AC4 A guard that fails when two requirements files that are installed into the same environment declare incompatible constraints for one package. This is the general defect; the SQLAlchemy instance is one symptom.
AC5 Re-check whether 2.1.0's arrival explains the greenlet disappearance in ci(base-red): main's smoke-test fails — SLM container dies on an undeclared greenlet dependency, blocking six PRs #17432. If the two are one upstream change, say so on both issues — they are still two fixes, but a reader deserves the connection.

Scope note

The conflict is in the same requirements files #17433 edits, so per same file ⇒ one PR, one agent the fix belongs on #17433 rather than in a parallel PR. Filed separately because it is a distinct defect with its own root cause, and #17433's acceptance criteria are about a missing declaration, not a version conflict.

Related: #17432, #17433, #14551.

Correction — AC1 is answered and it overturns the mechanism in the issue body. There is no mixed install. Retitling guidance below.

I wrote that two versions install into one environment and the second overwrites the first. That is wrong. They are two separate, freshly-created virtualenvs, each internally consistent:

1055  "$VENV/bin/python"     -m pip install -r requirements-ci.txt          -> 1122  sqlalchemy-2.0.54 (cached)
1991  rm -rf "$SLM_VENV" ; python -m venv "$SLM_VENV"
2002  "$SLM_VENV/bin/python" -m pip install -r autobot-slm-backend/requirements.txt  -> 2046  sqlalchemy-2.1.0 (downloaded)
2015  SLM_VENV: /home/runner/.venv-python-suite-slm

And the failing tracebacks are unambiguous about which one runs them:

../../../.venv-python-suite-slm/lib/python3.14/site-packages/sqlalchemy/ext/asyncio/session.py:651
../../../.venv-python-suite-slm/lib/python3.14/site-packages/sqlalchemy/util/concurrency.py:199 (greenlet_spawn)
../../../.venv-python-suite-slm/lib/python3.14/site-packages/sqlalchemy/orm/session.py:2473

So the real defect is simpler and different: SQLAlchemy 2.1.0 breaks the SLM backend's code. autobot-slm-backend/requirements.txt:31 declares sqlalchemy>=2.0.54 with no upper bound, so a minor-version bump was taken silently by a fresh venv. requirements-ci/storage.txt's ==2.0.54 is not in conflict with it — it governs a different environment, and that venv is fine.

Note the traceback passes through greenlet_spawn, which confirms #17433's fix worked: greenlet is present and the failure is downstream of it, inside 2.1.0.

How I got it wrong, since it is the point of the guard in AC4: I saw two version strings in one job log and inferred a shared environment without checking. Two versions in one job and two versions in one environment are different facts, and only the second is a conflict. I asserted the mechanism rather than measuring it — the same error class this repo's measurement discipline exists to catch, and I had been correcting others on it in the same session.

Revised acceptance criteria:

AC1 Traced. $VENV ← requirements-ci.txt → 2.0.54. $SLM_VENV ← autobot-slm-backend/requirements.txt → 2.1.0. Failing tests run in $SLM_VENV.
AC2 (revised) Decide between capping autobot-slm-backend/requirements.txt below 2.1.0, and migrating the SLM backend's code to 2.1.0. A cap unblocks the base today and defers the work; the migration is the real fix and is not a base-unblocking change. Owner-level call — this is a minor-version bump with a migration surface, not a pin correction.
AC3 python-suite green on main, all 12 shards, InternalTraversal count zero.
AC4 (revised, and now the general defect) A guard that fails when a production requirements file declares an unbounded floor for a package whose major/minor bumps have broken the tree before. The original AC4 — cross-file constraint conflict — describes something that is not happening here and should not be built on this issue's evidence.
AC5 Confirm whether 2.1.0's release also explains the greenlet disappearance in ci(base-red): main's smoke-test fails — SLM container dies on an undeclared greenlet dependency, blocking six PRs #17432. Still open, still worth answering on both issues.

The title should say SQLAlchemy 2.1.0 breaks the SLM backend, not both install into one job. I will leave retitling to whoever picks it up, since the body now carries the correction either way.

fix(deps): cap SQLAlchemy below 2.1 -- 2.1.0 breaks the SLM backend (#…

AC5 answered: the two defects are one upstream change. Credit to e5, who root-caused it.

SQLAlchemy 2.0.x  declares  greenlet>=1
SQLAlchemy 2.1    does not

That is the whole mechanism. The service plane's unbounded >=2.0.54 took 2.1.0 the moment it published, and a transitive dependency stopped arriving without a single line changing in this repo.

The two CI virtualenvs make it visible:

venv	declares	resolves to	greenlet
`.venv-python-suite`	`requirements-ci/storage.txt:4` → `sqlalchemy==2.0.54`	2.0.54	installed (3.5.6)
`.venv-python-suite-slm`	`autobot-slm-backend/requirements.txt:31` → `sqlalchemy>=2.0.54`	2.1.0	absent

Pip states it outright, in the main venv only:

Collecting greenlet>=1 (from sqlalchemy==2.0.54->-r requirements-ci/storage.txt (line 4))

Both interpreters are python3.14, so it is not a platform or marker difference — a hypothesis that was raised and disproved rather than assumed away.

So #17432 and this issue are one upstream event with two symptoms, and the ordering is now established rather than inferred: 2.1.0 published → greenlet stopped arriving in the SLM venv → SLM tests died at collection → #17433's greenlet declaration let them run → they reached 2.1.0's incompatibility and failed with InternalTraversal.

A near-miss worth recording, because it is the same error class the rest of this issue is about. e5 grepped the shard log for the installed SQLAlchemy version, found sqlalchemy-2.0.54 and greenlet-3.5.6 in a Successfully installed line, and nearly filed a correction saying the diagnosis was wrong on both halves and that #17433 fixes nothing. That version string was the main venv's pinned install; the SLM venv appears later in the same log. A measurement that is correct about something other than what was asked — which is exactly how I got the mechanism wrong earlier on this issue, reading two version strings in one job as one environment. Two of us made the same mistake on the same log within an hour. The log interleaves environments and marks the boundary only with an rm -rf line neither of us read first.

This does not change the fix. The cap on #17433 stands and #17446 carries the migration. What it changes is #17448's justification: greenlet was reachable only through SQLAlchemy's dependency list, so declaring directly what the code imports directly is not style — it is the specific thing that would have prevented this.

Severity escalation: this is a latent production outage, not a CI failure. Host evidence from e5, from a provisioned node.

environment	declares	sqlalchemy	greenlet
deployed SLM service venv	`>=2.0.54`	2.0.52	3.5.3 present
deployed backend service venv	`>=2.0.54`	2.0.54	3.5.3 present
CI main venv	`==2.0.54`	2.0.54	3.5.6 present
CI SLM venv	`>=2.0.54`	2.1.0	absent

Production matches the == plane. Both deployed venvs run 2.0.x with greenlet present. The CI SLM venv at 2.1.0 is the only environment in the fleet running it — CI has been testing a resolution that exists nowhere else since 2.1 published.

The consequence, which reframes this whole issue:

greenlet is present on the deployed nodes for exactly the reason it was present in CI's main venv — SQLAlchemy 2.0.x pulls it — and for no other. Nothing declares it.

So the moment any update path resolves >=2.0.54 to 2.1.0 on a deployed node, the SLM service loses sqlalchemy.ext.asyncio at import, in production, with no code change and no warning. The same resolution that broke CI is one pip install -r away from breaking the running service.

#17433 is therefore not a CI fix that happens to be noisy. It is this defect arriving first in the environment that fails safely, and the cap plus the explicit greenlet declaration are what stop it arriving in the one that does not. That is the argument for landing it, and for not treating the cap as cosmetic debt.

This also settles #17448's open question by measurement rather than judgement: production matches the == plane, so the service plane's unbounded >= is the outlier and the direction of conformance is not a preference.

Stated limits, e5's and mine: one node read, one package measured. Nobody has established how many other packages diverge between the two planes, and nobody should quote a number until #17448's enumeration runs.

Related: #17449 — the deployed SLM venv is at 2.0.52, below its own declared floor, while the deployed backend is at 2.0.54. Whether that is a stale node or an update path that does not upgrade in place is not established, and those are a currency problem and a bug respectively.

fix(deps): declare greenlet where the async SQLAlchemy engine is used (…

fix(deps): restore the SQLAlchemy cap this bump rewrote (#17445)

The cap was rewritten by an automated bump today, and the comment did not stop it

#17706, a grouped all-dependencies PR for /autobot-slm-backend, proposed sqlalchemy>=2.1.0,<2.2 and left this issue's fifteen-line rationale in place directly above the line it contradicted — including "Do not raise this bound without running the SLM suite against 2.1.x first."

Restored in 0e41600b84 on that PR.

The cap needs enforcement, not a comment

.github/dependabot.yml already ignores sqlalchemy under update-types: ["version-update:semver-major"], which never applied: 2.0 → 2.1 is a minor. Same shape as openai (#14431) and protobuf (#14727).

Written and verified, held on branch wip-sqlalchemy-cap-guard-17445:

versions: [">=2.1"] in the SLM pip block, bounded rather than unbounded unlike ansible-core (chore(deps): exclude the ansible-core pin from dependabot — it tracks a deployment fact, not a release #15983), so 2.0.x patches including security ones still arrive.
sqlalchemy_cap_frozen_17445_test asserting both halves — the pin in requirements.txt and the config range — modelled on ansible_core_pin_frozen_15983_test, and carrying its caution that a test over config shape is evidence about the config, not about what the bot does next.

Mutation-checked: putting update-types back fails the guard with the reason. Its own contrast pair also caught a bug in my first version-comparison helper, which read >=2.1.0 as not barring 2.1 because bare tuple comparison makes (2, 1, 0) <= (2, 1) False.

Blocked at pre-push, not by anything wrong with it: touching .github/dependabot.yml selects ~35 workflow guards and overruns the budget (#17451), and the documented override is refused as a safety-bypass flag.

Until it lands, expect this bump again next Monday. That is the practical reason to prioritise either #17451 or lifting this cap properly.

chore(deps): bump the all-dependencies group in /autobot-slm-backend,…

fix(deps): restore the sqlalchemy 2.1 cap this bump reversed, raise t…

fix(deps): restore the sqlalchemy cap and hold huggingface-hub below …

chore(deps): bump the all-dependencies group in /autobot-backend with…

chore(deps): bump the all-dependencies group in /autobot-infrastructu…

Sighting: dependabot #18104 raised the cap to sqlalchemy>=2.1.3,<2.2 and python-suite shards 7, 8, 9 and 10 failed with the same TypeError: 'InternalTraversal' object is not callable from _cache_key_cy.py. The fix stays here. The autobot-slm-backend dependabot block only ignores sqlalchemy semver-major, so it will keep proposing 2.1.x until a versions: [">=2.1"] ignore is added there, citing this issue, or until this migration lands.

fix(deps): dependabot holds sqlalchemy below 2.1 in every pip block t…

Owner decision (2026-10-10): fix forward to SQLAlchemy 2.1. Root-cause the SLM InternalTraversal break and migrate. The <2.1 cap and its guard stay only until that migration lands, then both go.

test(deps): the sqlalchemy cap guard requires 2.0.x patches to stay a…

test(deps): the sqlalchemy cap guard fails on missing dependencies, c…

test(deps): the sqlalchemy <2.1 freeze is asserted in all four manife…

Progress: #18213 merged (1478ffa). repo_tests/sqlalchemy_cap_frozen_17445_test.py freezes the sqlalchemy cap in all four manifests and the dependabot versions: [">=2.1"] ignore in all four pip blocks.

It is range-based: every 2.1+ probe is excluded and 2.0.x patches stay allowed, so >=2.0 / !=2.1.0 / ==2.1.0 / ==2.0.* fail.
yaml and packaging are plain imports, so a missing dependency fails instead of skipping.
It uses a duplicate-key-rejecting YAML loader.

AC1/AC2 (version resolution) and AC4 (general guard) remain open.

Navigation Menu

Repository navigation

bug(deps): SQLAlchemy 2.1.0 breaks the SLM backend — an unbounded floor took a minor bump and 27 tests fail with InternalTraversal #17445

Verdict

Evidence

Why it surfaced only now, and why #17433 looks like the cause

Acceptance criteria

Scope note

The cap was rewritten by an automated bump today, and the comment did not stop it

The cap needs enforcement, not a comment

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

bug(deps): SQLAlchemy 2.1.0 breaks the SLM backend — an unbounded floor took a minor bump and 27 tests fail with InternalTraversal #17445

Description

Verdict

Evidence

Why it surfaced only now, and why #17433 looks like the cause

Acceptance criteria

Scope note

Activity

mrveiss commented on Sep 25, 2026

mrveiss commented on Sep 25, 2026

mrveiss commented on Sep 25, 2026

mrveiss commented on Sep 30, 2026

The cap was rewritten by an automated bump today, and the comment did not stop it

The cap needs enforcement, not a comment

mrveiss commented on Oct 9, 2026

mrveiss commented on Oct 10, 2026

mrveiss commented on Oct 11, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions