Skip to content

CRITICAL(update): the venv provenance marker survives a pip upgrade and bricks the package — 12 husk dist-info dirs already block deploys #17332

Closed
Closed
CRITICAL(update): the venv provenance marker survives a pip upgrade and bricks the package — 12 husk dist-info dirs already block deploys#17332
Labels
bugSomething isn't workingdeploymentinfrastructurepriority: critical

Description

@mrveiss
mrveiss
opened on Sep 23, 2026

The #15067 install-provenance marker is written into a distribution's own dist-info
directory and is not listed in that distribution's RECORD. pip only deletes what RECORD
lists, so on the next upgrade the old dist-info directory survives as a husk containing one
file: the marker. Python then sees a second, contentless distribution for that package with
version None, and every later pip run that must uninstall it aborts.

The venv cannot be repaired by re-running the deploy — the failing step is the deploy.

Evidence — a provisioning run from this evening

TASK [backend : Install filtered backend requirements]
fatal: [<manager-node>]: FAILED! => {"changed": false, "cmd": [".../venv/bin/pip3", "install", "-r", ".../filtered-requirements.txt"],
  "msg": "... Installing collected packages: cachetools, sqlalchemy, sentence-transformers,
          pypdf, faiss-cpu, langchain-core, langchain, botocore
          Attempting uninstall: cachetools
            Found existing installation: cachetools None
          :stderr: error: uninstall-no-record-file
          x Cannot uninstall cachetools None
          '-> The package's contents are unknown: no RECORD file was found for cachetools.
          hint: You might be able to recover from this via:
                pip install --ignore-installed --no-deps cachetools==7.1.8"

On-disk state in that venv:

cachetools/                      <- the real package
cachetools-7.1.8.dist-info/      <- 1 file:  AUTOBOT_PROVENANCE
cachetools-7.2.0.dist-info/      <- 6 files: AUTOBOT_PROVENANCE, INSTALLER, METADATA, RECORD, REQUESTED, WHEEL

cachetools-7.1.8.dist-info/AUTOBOT_PROVENANCE:

{"tool": "autobot-venv-reconcile", "component": "autobot-backend", "recorded_at": "2026-09-20T11:36:20+00:00"}

This is not one package. A sweep of that venv's site-packages found 12 dist-info
directories holding exactly one file:

anthropic-1.5.0        boto3-1.43.93          botocore-1.43.93       cachetools-7.1.8
faiss_cpu-1.15.0       langchain-1.4.0        langchain_core-1.6.2   protobuf-7.36.1
pypdf-6.18.1           sentence_transformers-6.0.1  sqlalchemy-2.0.52  uvicorn-0.52.4

Every one of those packages is now un-upgradable by pip on that node. pip stops at the first,
so a run surfaces them one at a time — 12 consecutive failed deploys to discover them all.

Root cause

autobot-slm-backend/venv_provenance.py:115 (write_provenance_marker) writes
AUTOBOT_PROVENANCE inside the target's dist-info directory. Nothing adds it to RECORD, and
nothing removes it before an upgrade.

The module's own docstring states the assumption that fails
(autobot-slm-backend/venv_provenance.py:35): a reinstall "replaces the dist-info directory and
erases the marker". pip does not replace the directory — it unlinks the paths named in RECORD
and removes the directory only if nothing else is left in it. An unrecorded extra file makes that
removal a no-op, which is the whole defect: the marker is designed to be erased by an upgrade and
is instead what survives one.

The marker was chosen to be dist-info-adjacent precisely so it could not outlive the install it
describes (venv_provenance.py:21-25). On this evidence it outlives it in the one case that
matters.

Acceptance criteria

  • The provenance marker can no longer leave a husk: either it moves out of dist-info to a
    location keyed by package+version that the reconciler owns end to end, or the writer records
    it in RECORD so pip removes it with the rest of the distribution.
  • The reconciler detects and clears an existing husk — a dist-info directory with no
    METADATA/RECORD — before installing, so the 12 already on disk are recovered rather than
    needing 12 manual --ignore-installed runs. Per #17038, removal of stored state is
    proposed and approved, never silently performed; a husk is tool-authored and carries the
    tool's own marker, so state its classification explicitly in the implementation.
  • A test installs a package, stamps it, upgrades it, and asserts no contentless dist-info
    directory remains — the sequence that produced this, which the #15067 tests do not cover.
  • has_tool_provenance is not fooled by a husk: a husk currently answers "ours" for a
    distribution that is not installed.

Related

  • #15067 — the feature that introduced the marker (CLOSED)
  • #15063 — the builtin update path cannot remove a dependency, the problem #15067 addressed
  • #17331 — the other failure in the same run, on the non-manager node

Activity

  1. mrveiss
    added
    bugSomething isn't working
    priority: critical
    deployment
    infrastructure
    on Sep 23, 2026
  2. mrveiss
    added 2 commits that reference this issue on Sep 23, 2026

    fix(update): stop the venv provenance marker outliving its install, a…

    03fde8d

    test(slm): scope the multipart import stub so it stops leaking into l…

    a4b2e7a

  3. mrveiss commented on Sep 23, 2026

    @mrveiss
    mrveiss
    on Sep 23, 2026
    OwnerAuthor

    Fix pushed in #17338 (the provenance-husk half), head a4b2e7a0e5.

    Not closed, and the reason matters more than the fix. The last criterion here is a multi-host provisioning run, which this session cannot perform. The failure was observed on one, the cause is read from the code and from pip's own source, but the fixed run has not happened. Closure needs that run's output, not this merge.

  4. mrveiss commented on Sep 23, 2026

    @mrveiss
    mrveiss
    on Sep 23, 2026
    OwnerAuthor

    Correction: 13 husks across two venvs, not 12 in one

    The count in this issue was measured in autobot-backend/venv alone and written as "12 packages on this host" — a one-venv measurement reported with host-wide language. A reader greps that venv, confirms 12, and stops.

    Found by autobot-ai-f8, who swept every venv rather than the one this issue named. Re-measured here independently before correcting anything, read-only, using the same predicate is_provenance_husk applies (a .dist-info whose only entry is AUTOBOT_PROVENANCE):

    venv husks
    autobot-backend/venv 12
    autobot-slm-backend/venv 1 — psycopg2_binary-2.9.12.dist-info
    ai-stack, browser-worker, npu-worker, slm-agent, tts-worker 0

    The thirteenth matters beyond the arithmetic. It is in the SLM's own venv — the component that runs the builtin updater, which is the only path clear_provenance_husks is wired into (#17339). Its live sibling psycopg2_binary-2.9.13.dist-info carries a 46-line RECORD while the husk holds one file, so it is the same upgrade-leak mechanism, not a second defect.

    f8 also ran the predicate as a standalone dry-run over every *.dist-info under the install tree: 13 matched, 0 near-misses. Nothing on this host sits near the boundary, so the narrow predicate costs no coverage here while keeping the deletion defensible.

    Corrected in the branch (e70493c24f) in both docstrings that carried the number, and in #17338's body. The code is unchanged — the defect and the fix are the same; only the measurement was wrong.

  5. mrveiss
    added 10 commits that reference this issue on Sep 23, 2026

    docs(update): correct the husk count — 13 across two venvs, not 12 in…

    e70493c

    fix(update): stop the venv provenance marker outliving its install, a…

    ec4055f

    test(slm): scope the multipart import stub so it stops leaking into l…

    5855075

    docs(update): correct the husk count — 13 across two venvs, not 12 in…

    9ca4926

    fix(guards): re-pin two reach floors from BOTH their bounds, on this …

    adb46b4

    fix(update): stop the venv provenance marker outliving its install, a…

    bf10a37

    test(slm): scope the multipart import stub so it stops leaking into l…

    6973e07

    docs(update): correct the husk count — 13 across two venvs, not 12 in…

    ab0f0ef

    fix(guards): adopt main's reach floors on rebase and record what they…

    da45434

    fix(deploy,update): controller-only manifest paths, and a provenance …

    193b182

  6. mrveiss commented on Sep 24, 2026

    @mrveiss
    mrveiss
    on Sep 24, 2026
    OwnerAuthor

    All four criteria met. PR #17338 merged 193b182c79 at 02:10. Verified against merged code with git show origin/main:, not against the PR diff.

    • The marker can no longer leave a husk. venv_provenance.py:171 _record_marker_in_record appends the marker to the distribution's own RECORD, so pip's uninstall removes it and can then rmdir the directory. The append format was checked against pip's own convention on a real installed distribution — PyNaCl-1.5.0.dist-info/RECORD,,, i.e. dist-info-name/filename with empty hash and size — so pip's iter_declared_entries() will pick it up rather than skip a malformed line.
    • The reconciler detects and clears an existing husk. is_provenance_husk and clear_provenance_husks are both on main. The predicate is conjunctive — directory, no METADATA, no RECORD, iterdir() succeeds, and contents are exactly [AUTOBOT_PROVENANCE] — so anything carrying another file fails it and is left alone. OSError on iterdir fails closed; OSError on unlink/rmdir is logged and skipped rather than retried.
    • A test installs, stamps, upgrades, and asserts no contentless dist-info. test_a_stamped_install_leaves_no_husk_when_it_is_upgraded (:109), with test_the_old_unrecorded_marker_reproduces_the_husk (:92) as its positive control — the pair is what makes it meaningful, since the second proves the harness can actually produce the defect the first asserts is absent.
    • has_tool_provenance is not fooled by a husk. venv_provenance.py:158-168 returns not is_provenance_husk(dist_info), with the reasoning in the docstring: "answering True would let a removal be authorised by the debris of an earlier one." Pinned by test_has_tool_provenance_is_false_for_a_husk (:193).

    Ten tests in test_venv_provenance_husk_17332.py, including test_a_real_installation_is_never_cleared and test_a_directory_holding_anything_else_is_left_alone — the two that make the deletion defensible rather than merely narrow.

    What this does NOT do, recorded so nobody reads the closure as broader than it is. The 13 husks measured on a live host are still on disk. The repair is reachable only from _run_pip_install (the builtin code-sync path); git grep across autobot-slm-backend/ansible/ returns zero references to it, and that host is ansible-provisioned. #17339 stays open and is the issue that repairs an already-husked host. This one closes because the defect that creates husks is fixed and existing ones are cleared on the path that can see them.

    One narrow fail-open found in review and filed as #17357, not a blocker: write_provenance_marker writes the marker before recording it, and recording silently no-ops when RECORD is absent or unwritable — recreating this defect for that one package. It is strictly narrower than main's previous behaviour, which never recorded the marker at all, and none of the 13 observed husks were of that shape. It is honestly tested as intended behaviour (test_a_distribution_without_a_record_is_still_stamped) rather than hidden, which is why it is a fast-follow and not a revert.

  7. mrveiss
    closed this as completedon Sep 24, 2026
  8. mrveiss
    added 3 commits that reference this issue on Sep 24, 2026

    fix(deploy): add the third marker-capable venv, and make both guards …

    8ffacf9

    fix(deploy): make the venv husk repair reachable from ansible provisi…

    54fec85

    fix(provisioning): four bugs in the venv/package provisioning path (#…

    5dcdb18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdeploymentinfrastructurepriority: critical

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions