chore(devenv): reconcile the CI-parity venv instead of trusting the directory (#15130)

[mrveiss]

Thinking Path

setup-ci-parity-env.sh (#13573) exists so a local pass means something. It was idempotent by existence: a second run confirmed the directory was there and the interpreter was 3.14, then stopped. So it never reconciled, and pr-preflight.sh went on printing ok python 3.14.x from the CI-parity venv — a claim about packages backed by a check of a path.

Measured before the fix, scoped to the two requirement files the script itself installs:

	Count
Declarations read from requirements-ci.txt + requirements-ci-test.txt closure	86
Installed but below the declared version	21
Declared but not installed at all	9

The 9 are the expensive ones, and they were invisible. The runs were passing by doing less: a test module that importorskips a missing package is never collected at all, so the shard goes green having tested fewer things — no skip line, no warning, nothing to notice. The missing packages were: anthropic, datasketch, ddgs, docker, faiss-cpu, ldap3, pdfplumber, trafilatura, weasyprint. That is precisely what the docker>=7.1.0 requirement comment already warns about for the containerised-sandbox smoke. check_dependency_floors.py could not see them, because it treats an absent distribution as a neutral fact. Correct for an arbitrary box; wrong for the one venv that installs these very files.

The fix is reconciliation, not --force. Every run compares installed against declared, reusing #15125's reporter rather than growing a second comparator. Two things had to be added to it to make that reuse honest, and both are defects in their own right — see What Changed.

Repair in place, not rebuild. All 21 shortfalls were an installed version older than a satisfiable declaration, and all 9 absences were simply not installed — re-running the same two pip install -r commands fixes every one. rm -rf would re-download the whole torch stack to arrive at the same place. --recreate stays for what that cannot fix. Nothing outside the venv is touched and no sudo is used, unchanged.

Refreshing is prompted, not automatic (AC3). setup-ci-parity-env.sh repairs, because you ran it to get parity. pr-preflight.sh calls the new --check mode, which installs nothing, and downgrades its ok line to a named STALE note — reported, never gated, the same call #15091 made. Reinstalling under someone mid-review would be slow and surprising. Recorded in CLAUDE_WORKFLOW.md as a table.

Premise correction: the deferral is fastapi's, not starlette's

The issue's sharpest argument is that starlette 1.3.1 against a pinned 1.6.0 means "the parity venv can still fail to reproduce the exact divergence it exists to reproduce", because the deferred include_router() of #15093 is "a starlette 1.6.0 behaviour". It is not. Measured directly, three ways:

fastapi	starlette	parent.include_router(child, prefix="/p") yields
0.141.1	1.3.1	_IncludedRouter, path=None — defers
0.141.1	1.6.0	_IncludedRouter, path=None — defers
0.135.2	1.6.0	APIRoute, path=/p/leaf — copies eagerly

The parity venv already satisfied fastapi>=0.141.1, so it was reproducing the deferral the whole time; the starlette version neither reproduces nor suppresses it. A starlette floor is not the lever anyone should reach for. That correction is now written into CLAUDE_WORKFLOW.md, where the previous wording named both packages and invited the same wrong inference.

This does not weaken the issue — it relocates its evidence. 9 missing packages silently shrinking a test run is a worse and better-supported fault than one stale starlette.

What Changed

• scripts/setup-ci-parity-env.sh — reconciles on every run instead of short-circuiting on the directory existing. New --check mode reports and exits 1 without installing anything. On drift it re-runs the same two installs and re-checks, and only then asks for --recreate. The two requirement files are named once, in one array that feeds both the install and the check, so the set installed and the set judged cannot diverge. A reporter exit of 2 (the check itself broke) is separated from exit 1 (stale), so nobody is sent into a reinstall to fix a broken checker.
• scripts/pr-preflight.sh — the ok … from the CI-parity venv line is now conditional on --check. A stale venv gets a named note and the remedy instead of an unqualified pass sitting next to a floor report that contradicted it.
• pipeline-scripts/check_dependency_floors.py — reused, not duplicated, plus two fixes:
  • --roots scopes the sweep to a caller's own entry points. Judged against all four roots the parity venv looks permanently short (44 of 206) because the backend files describe components neither it nor CI's python suite installs — a number nobody can act on.
  • --require-present makes absence a shortfall for the one caller that installs the files. Off by default; the existing "absent says nothing" behaviour is unchanged for every other caller.
  • Bug fix, found here: a PEP 440 local segment demoted a release, so torch 2.13.0+cpu — the build this script installs on purpose, from the CPU index — was permanently and falsely reported below its own ==2.13.0 pin. The default report drops from 6 shortfalls to 5 on a correct environment.
• docs/developer/CLAUDE_WORKFLOW.md — the prompted-vs-automatic decision as a per-caller table, the before/after numbers, and the corrected fastapi attribution.
• Tests: repo_tests/parity_venv_reconciles_15130_test.py (new) and pipeline-scripts/check_dependency_floors_test.py (extended).

Verification

Run against the final head of this branch (04de0c4), after the rebase onto the post-squash base.

Forensic — AC4, the count for a reconciled venv. Running the new script against the real parity venv took it from 21 of 86 to dependency floors: 86 declarations checked, all satisfied — with --require-present on, so the 9 absences are included in that zero. It installed all 9 missing packages and upgraded all 21, starlette among them. Against all four roots the same venv went from 44 of 206 to 5 of 206; the 5 that remain are backend declarations this venv never installs (protobuf ×2, xxhash, langchain-community, websockets), which is the "components CI never installs" the issue asked to have stated rather than left as noise.

Forensic — the deferral probe. Three throwaway venvs, the table above. Confirms the parity venv reproduced include_router's deferral before this PR as well as after.

Unit — 80 passed across pipeline-scripts/check_dependency_floors_test.py (62), repo_tests/parity_venv_reconciles_15130_test.py (9) and repo_tests/dependency_floor_banner_test.py (9, untouched, still green).

Test file	Shard (12-way, repo_tests/stable_shard.py)
repo_tests/parity_venv_reconciles_15130_test.py (new)	10
pipeline-scripts/check_dependency_floors_test.py (extended)	7

The new repo test drives the real script in --check mode against a structurally valid but package-starved venv, and asserts it installs nothing while doing so. It skips on any interpreter that is not 3.14, because the script refuses to build against one — CI's suite is 3.14, so the skip never fires there.

Contrast mutations — five, each re-run against this final head (04de0c4) after the rebase

Mutation	Observed failure
Restore existence-idempotency (skip the check when the venv is healthy)	E AssertionError: assert 'STALE' in 'base interpreter: ... checking ... against requirements-ci.txt requirements-ci-test.txt\n\n'
Drop --require-present from the check	E AssertionError: ... 'dependency floors: 86 declarations checked, all satisfied' — a venv with none of the packages reports clean
Drop --roots, unscoping the check	E AssertionError: install and check must both expand the list, found 1 / assert 1 >= 2
Revert the PEP 440 local-segment fix	E AssertionError: assert False is True where satisfies('2.13.0+cpu', '==', '2.13.0')
Restore the unconditional preflight ok	E assert 'setup-ci-parity-env.sh --check' in '#!/usr/bin/env bash ...'

Static — bash -n on both scripts; black / isort / flake8 clean on all touched Python.

Risks

• Stacked on chore(devenv): report installed packages below a declared floor, where a local pass is read (#15091) #15125 — resolved. chore(devenv): report installed packages below a declared floor, where a local pass is read (#15091) #15125 merged (squashed as c81ded2) and this branch was rebased with git rebase --onto origin/Dev_new_gui <old-parent>, which dropped both of its commits cleanly. The diff is now exactly the six files below, and neither reporter bug is duplicated: the merged base still carries both, and this branch is what fixes them.
• --require-present makes the parity script stricter. On a box where one of the 9 genuinely cannot install, the script now exits 1 where it used to say nothing. That is the intended trade — a silent skip is worse — and --recreate plus the named package is the path out.
• The reconcile is a real pip install, so a run on a stale venv now takes minutes where it used to take a second. --check exists for anyone who wants the answer without the wait, and it is what preflight uses.
• Reconciling the parity venv upgraded starlette 1.3.1 → 1.6.0 on this box. Local runs there now see the declared framework versions, which is the point, but it is a behaviour change for anything that had adapted to the old ones.

Model Used

Claude Opus 5 (claude-opus-5[1m])

Issue Link

Closes #15130

Changelog fragment

• N/A — developer tooling, no user-visible behaviour.

Checklist

• Code follows AutoBot patterns from CLAUDE.md
• Tests added or updated
• Documentation updated if behavior changed
• Pre-commit hooks pass
• PR targets Dev_new_gui
• No secrets or credentials in the diff

[github-actions]

✅ SSOT Configuration Compliance: Passing

🎉 No new hardcoded values of either class — ssot and other both block.

Known backlog in pipeline-scripts/hardcoded_values_baseline.txt is suppressed and tracked in #14371.