Repository navigation
deploy: the builtin updater never removes files deleted from source, so deleted modules pile up on every host #16310
Description
Activity
Owner requirement, 11 Sep 2026: "We need to have drift check for all the files." Folded into this issue's scope, read as:
- Every file under each deployed component is checked. Untracked files are no longer "not drift": a file removed from source is drift, and only declared host state and expected build bundles are excluded.
- Every exclusion is named and counted in the result: the host-state set, build bundles, the venv,
node_modulesand the rest. "Clean" can never hide "did not look". - One run covers every component, plus
autobot_sharedand the infrastructure tree, not one component at a time. - Tests: a planted file removed from source, a planted never-tracked host file, and a planted modified file each get the right verdict, and a run over an empty or unreadable tree fails loudly rather than reporting no drift.
Owner decisions, 2026-09-11 (after a verification review found AC5 unreachable with the Python-side wiring). The normal Update All fires the full playbook, which restarts the SLM partway through, so Python never sees a success to hook onto. Three components also have no marker to start from, which made the pass a permanent no-op for them.
- Deletion runs in the Ansible roles. Before the update, the SLM computes a safe deletion list per component: renames and deletions between the last-deployed commit and the new one, excluding host-state files, gitignored files and build bundles, and contained to the deployed tree. Each role deletes that list right after its own file sync succeeds, then records the new commit in the pass's own marker, never
.deployed_commit. That's one mechanism for co-located and remote nodes alike, so deploy: ansible synchronize never deletes files removed from source on remote fleet nodes #16322 folds into the same PR. - A one-time cleanup where no record exists. For a component with no deletion marker and no legacy marker (backend, frontend and SLM-frontend today), the first update deletes the files the drift check proves git once tracked and no longer tracks, with the same exclusions, then starts recording. It runs at most once per component. This relies on the owner's no-manual-patches rule, which treats such a file as debris. The residual risk is a file deleted from git long ago and put back on the host by some other means; it's named in the code and the tests.
AC5 still needs host evidence from a real update through the GUI.
- Deletion runs in the Ansible roles. Before the update, the SLM computes a safe deletion list per component: renames and deletions between the last-deployed commit and the new one, excluding host-state files, gitignored files and build bundles, and contained to the deployed tree. Each role deletes that list right after its own file sync succeeds, then records the new commit in the pass's own marker, never
- added 8 commits that reference this issue
on Sep 11, 2026 18 remaining items
Evidence from the live install on 2026-09-14: the one-time bootstrap deleted nothing, because the updater's
code_sourceclone is shallow.- The first update after fix(code-sync): delete files removed from source through the updater, and check drift across every file (#16310, #16322) #16351 merged (18:32, Riga) had no marker for
autobot-backend, so it ran the bootstrap path: enumerate present files, compute the bootstrap plan, settle on the plan. No delete step ran, and it then wrote the marker. The next update (19:31) used the diff-based plan from that marker. - Twelve files deleted from source in August are still deployed under
autobot-backend/, includingllm_shared/semantic_cache.py,utils/semantic_chunker_gpu_optimized.pyandsecurity/enterprise/config_loading.py. Only the nestednpu_workers.yamlwent, through its own dedicated task. - Root cause, checked read-only:
code_sourcehasrev-parse --is-shallow-repository= true, only 1 commit reachable, and 210 shallow-boundary entries, against 13,203 commits onorigin/main._ever_added_pathsrunsgit log -M --name-status --diff-filter=ARthere and gets nothing back for these files. So_bootstrap_candidates(services/sync_deletions.py) finds no candidates and the plan comes out empty. It isn'tkept_reason: none of the files are host-state or gitignored. - Because the marker was written after that empty bootstrap, later runs use diff-based mode and never revisit these files. Without a fix they stay on the host permanently.
Added criterion:
- The bootstrap plan is computed against the full history (deepen or unshallow the updater clone for this computation, or keep
code_sourceat full depth). A bootstrap computed on a shallow clone is not recorded as complete, so the next update re-runs it, including on hosts that already hold a marker from an empty shallow bootstrap. Verified on a host: after one update, the 12 August leftovers are gone and the drift check shows 0 "removed from source" files forautobot-backend.
- The first update after fix(code-sync): delete files removed from source through the updater, and check drift across every file (#16310, #16322) #16351 merged (18:32, Riga) had no marker for
- added a commit that references this issue
on Sep 16, 2026 #16789 merged (
e49d3d077) — correctly left open, and one criterion cannot close without host evidencePR #16789 landed on
mainase49d3d077carryingRefs #16310, which is the right keyword: it
contributes to this issue without completing it. Recording where that leaves the criteria.Criterion 5 cannot be ticked from any diff: "Host evidence: after one update through the GUI, the
drift check shows 0 'removed from source' files on all four components." That is a statement about a
running system after an operation performed through the maintenance UI. No amount of reading merged
code satisfies it — it needs the update run and the drift check read afterwards. Flagging it explicitly
so nobody ticks it on the strength of the code looking right.The other five are diff-verifiable but unverified. I merged this PR and have not checked them, so
none should be ticked yet. The two worth particular care:- Criterion 1 requires the deletion set come from
git diff --name-status --diff-filter=DR <old>..<new>
and that a never-tracked path survives. The test called for is three-way — deleted file removed,
renamed file's old path removed, never-tracked file untouched — and a test that only proves the first
two would pass while leaving the dangerous case unexercised. - Criterion 6 concerns a bootstrap computed against full history, with the specific failure that a
bootstrap computed on a shallow clone must not be recorded as complete. A marker written from an empty
shallow bootstrap is indistinguishable from a successful one unless something checks depth — the same
shape as a guard that cannot tell nothing found from did not look.
Next step: whoever verifies should check each criterion against
e49d3d077and tick only what the
merged code demonstrates, leaving criterion 5 unticked with a note until a host run exists. Partial
delivery closes nothing; this issue stays open until all six hold.- Criterion 1 requires the deletion set come from
Criterion 5 host evidence: NOT MET. Collected read-only from the live install on 2026-09-19.
- Every component (backend, frontend, SLM frontend and backend, browser worker, NPU worker, AI stack) now has its own deletion marker, current as of today's source commit. The deletion pass is wired and running for all of them, including the 3 that previously had no marker.
- But 3 of the files this issue's 2026-09-14 comment listed as August leftovers are still deployed under the backend and are absent from source: the
llm_sharedsemantic-cache helper, the GPU-optimised chunker util, and the enterprise config-loading module. - Root cause, still present: the updater's source checkout is still a shallow clone (
is-shallow-repositoryreturnstrue). - Why it can't self-heal:
sync_deletionsnow correctly refuses to bootstrap on a shallow clone. Every component's marker, however, was written before that guard existed, so each one sits in diff-only mode permanently. The bootstrap sweep that would find these files never runs again.
The fix this still needs, in the builtin updater and not by hand on the host: the updater has to guarantee full history for its source checkout (unshallow it on fetch). It also has to re-run the bootstrap for any component whose marker predates the shallow-clone guard, or was written on a shallow clone.
Chunking triage — a proposal, not an assignment
- Proposed priority:
priority: high— not applied. Setting it is the milestone owner's call. - triage(v0.9.0): release criticality of the 123 open issues — 26 blocking, 36 small, 18 umbrellas, 41 misfiled, 2 undetermined #17639 criterion met: a broken install or update path — the builtin updater left deleted modules importable on every host
- triage(v0.9.0): release criticality of the 123 open issues — 26 blocking, 36 small, 18 umbrellas, 41 misfiled, 2 undetermined #17639 bucket: 1 — genuinely blocking a v0.9.0 release
- Scope:
deploy (secondary updater) - Primary files named by the issue:
ansible/tests/inventory/group_vars/all.yml,autobot-backend/autobot-backend/config/npu_workers.yaml,code_sync.py,config/npu_workers.yaml - Umbrella / container: YES — open children deploy: consolidate slm_manager's delete:true synchronize into the git-aware deletion pass #16338 deploy: a host whose sync_deletions marker came from an empty shallow bootstrap never re-bootstraps #16787 (OPEN; deploy: ansible synchronize never deletes files removed from source on remote fleet nodes #16322 closed). A container is never "finished today", so it must not sit in a numbered chunk; the chunk would never close.
- Pre-filter: ⚠ a merged commit references this issue —
dcc111fdae fix(updater): filter _sd_own_marker_is_legacy through | bool. A reference is not a delivery: verify AC coverage before putting it in a chunk, and consider a closure pass first. - Basis: triage(v0.9.0): release criticality of the 123 open issues — 26 blocking, 36 small, 18 umbrellas, 41 misfiled, 2 undetermined #17639's per-issue row, reused rather than re-derived — "Builtin updater left deleted modules importable on every host (shallow source checkout). ensure-full-history pre-flight is on main; the only host check (2026-09-19) predates it and failed."
Nothing was relabelled, moved or closed by this pass.
- Proposed priority:
Measured 11 Sep 2026 with the SLM's own File Drift Check on the maintainer's install, then classified by git history (read-only). No drift and no manual patches in any of the four components checked. The owner's rule is that there must never be manual patches, so every host file has to come from, and be cleaned up by, the builtin updater. The untracked files it lists show the updater never removes a file that was deleted from source.
What the drift check lists as "untracked on this host"
autobot-backendsecurity/enterprise/config_loading.py(3fc18e7) andutils/semantic_chunker_gpu_optimized.py(20036f9), both still importable on the host, plus 8 test files. 1 never tracked: a nestedautobot-backend/autobot-backend/config/npu_workers.yaml, byte-identical (same sha256) to the canonicalconfig/npu_workers.yaml. It was written on 2026-07-30 by the old cwd-relativeNPUWorkerManagerdefault, since fixed (seetests/test_npu_config_path_not_cwd_relative.py).autobot-frontendsrc/components/settings/DevicePairingSettingsPanel.vue, deleted from source 2026-08-10 (91ed3bc)autobot-slm-backendansible/roles/python314/role (de83f71, 09-03),ansible/tests/inventory/group_vars/all.yml(4b6defc, 08-21) andservices/reconciler_flap_escalation_test.py(3cf6918, 08-18)autobot-slm-frontendsrc/views/ServicesView.vue, deleted 09-01 (c815c7b). Threedist-<build-id>/bundles, which are expected:services/slm_frontend_build.pykeeps the newestSLM_FRONTEND_RELEASE_KEEP(default 3). And the legacydist.previous/(9.2 MB) from the pre-#15462 publish design, which the pruner never matches because its name lacks thedist-prefix.Why they stay: a normal update syncs without deletion. Only a drift resolve is a delete-style rsync (
code_sync.py, #13913/#13851), and that deletes every untracked file, host state and build bundles included. So neither path can safely clean this up, and deleted modules pile up on every host.Acceptance criteria
git diff --name-status --diff-filter=DR <old>..<new> -- <component>). It never touches a path git has never tracked, so runtime and host state are safe by construction. Test on a fixture repo: a deleted file is removed, a renamed file's old path is removed, and a never-tracked file survives.<commit>(the next sync removes it), build bundle (expected, with retention), or never tracked (host or runtime state, left alone). An operator can tell debris from state without reading git.dist.previous/is removed once thecurrent/previoussymlink layout is in place, through the SLM frontend publish step, not by hand.npu_workers.yamlduplicate is removed through the updater, as a leftover of the fixed cwd bug, only while it is byte-identical to the canonical file.code_sourceat full depth). A bootstrap computed on a shallow clone is not recorded as complete, so the next update re-runs it, including on hosts that already hold a marker from an empty shallow bootstrap.Relates to #13947, #13851, #13913, #15462, #12526.