chore(scripts): decide the last five unreferenced infrastructure scripts (#15127)

[mrveiss]

Thinking Path

First pass read the five remaining KNOWN_UNREFERENCED entries against the current tree rather than the issue's assumptions, since two of those assumptions turned out to be wrong on inspection: cleanup-legacy-python.sh's #1898/#1924 migration is confirmed complete (CI is Python 3.14 everywhere); enable-phase4-enterprise.sh's /api/enterprise/* API is real, not missing -- 9 live routes in autobot-backend/api/enterprise_features.py, registered under /enterprise at feature_routers.py:270 (app factory prepends /api), every jq path the script reads matched the route handlers line-for-line. fix-frontend-dependencies.sh and start_containers.sh target infrastructure (Dockerfile.dev, docker/compose/docker-compose.base.yml, redis/npu-worker/playwright compose service names) that never existed in tracked history. utilities/batch-configure-vms.sh's three subcommands are superseded by real Ansible playbooks (setup-passwordless-sudo.yml #700, patch-system-packages.yml #682, health-check.yml) and it piped a plaintext password over SSH, which CLAUDE.md's credentials rule forbids.

Review round one. A reviewer confirmed the enterprise-API finding independently and flagged that deleting repo_tests/unreferenced_shell_script_baseline.py had taken repo_tests/unreferenced_shell_script_test.py with it -- including TestNoNewUnreferencedScript (the guard that catches a new unreferenced script) and the enumeration/reach checks. Restored the test file with only the KNOWN_UNREFERENCED-specific machinery removed; every batch-regression class from the four already-landed decisions in this issue survives untouched. The reviewer also asked to fold in an independent finding: ensure-frontend-dependencies.sh only "passed" the guard because its path appears in pipeline-scripts/hardcoded_values_baseline.txt -- a hardcoded-value finding against it, not a caller. Added that file to NOT_A_REFERENCE. Checking the actual effect (not assuming) flipped 14 scripts to genuinely orphaned, not the one anticipated. Filed #15322 for a separate, wider gap surfaced along the way: docker/compose/docker-compose.hybrid.yml is missing but still referenced by the live run_agent.sh entrypoint and 12 other tracked files.

Review round two. A reviewer caught a real contradiction: two of the 14 retirements (deployment/start_npu_worker.sh, ensure-frontend-dependencies.sh) were justified in the commit message using the same missing file that #15322 says is a defect elsewhere in the tree, while check_ai_stack_health.sh -- which references that identical file -- was kept. Same evidence, opposite verdicts, inside one PR. Investigated each on its own, independent of #15322:

• ensure-frontend-dependencies.sh does not reference docker-compose.hybrid.yml at all -- that was a drafting error in the original commit message (this script and its already-retired sibling fix-frontend-dependencies.sh got conflated while writing it up). Re-read the actual script: it names ./run_agent_unified.sh --dev, deleted in [Housekeeping] Archive Cleanup - Extract Features and Delete Archived Files #567, and its whole technique -- docker exec autobot-frontend npm install "$package" --save, then restart the container to reload Vite -- is incompatible with the current autobot-frontend service (docker-compose.yml:631): read_only: true, cap_drop: [ALL], tmpfs-only mounts, a pre-built static bundle served by nginx (docker/frontend/Dockerfile, from the same March 2026 rebuild below). There is no writable node_modules and no npm at runtime to install into. Retirement stands, corrected to this evidence, unrelated to run_agent.sh (the documented 'Start full system' entrypoint) is broken: docker-compose.hybrid.yml was deliberately retired and 13 callers were never repointed #15322.
• deployment/start_npu_worker.sh genuinely does reference docker-compose.hybrid.yml, so this one needed real independent grounds. Checked the deletion history properly this time: the file was deleted and recreated across several commits, and the most recent, authoritative one is bf0a3b1af ("feat(docker): fresh containerization for current codebase", feat: Fresh Docker containerization — Dockerfiles and compose for current codebase #1809/feat(docker): fresh containerization for current codebase (#1809) #1818, 2026-03-15) -- its own message calls the old compose setup "obsolete" and explicitly did not recreate an NPU-worker Dockerfile when it rebuilt docker/backend, docker/frontend, docker/slm. git grep across every tracked *.yml confirms no compose file anywhere defines an npu-worker/autobot-npu-worker service -- every match is Ansible inventory/role config, because NPU-worker deployment moved to native/Ansible (autobot-npu-worker/ is a real, actively-maintained native package; the Ansible npu-worker role is real). The script's own header already says so: "Production uses native deployments. See Ansible roles for equivalent." This is different from run_agent.sh's Redis lines, which run_agent.sh (the documented 'Start full system' entrypoint) is broken: docker-compose.hybrid.yml was deliberately retired and 13 callers were never repointed #15322 covers and which do have a live repoint target (autobot-redis in the current docker-compose.yml) -- posted this distinction on run_agent.sh (the documented 'Start full system' entrypoint) is broken: docker-compose.hybrid.yml was deliberately retired and 13 callers were never repointed #15322 so whoever picks it up doesn't try to fix the NPU line the same way as the Redis lines.

Re-checked the other six retirements for the same defect class (support that depends only on something #15322 covers): none do. setup_model_sharing.sh and setup_windows_only_models.sh each reference different, unrelated missing compose files (docker-compose.shared-models.yml, docker-compose.full.yml, docker-compose.windows-models.yml -- none of them hybrid.yml, none of them named in #15322), and each has independent support that doesn't depend on any missing file at all: a wrong container name (autobot-ollama-shared vs the real autobot-ollama) and an output scheme (.env.model-sharing, AUTOBOT_MODEL_SHARING_STRATEGY) nothing reads, for the first; a hardcoded personal Windows username fallback ("marti") and the same unread-output pattern, for the second. run_agent_fast.sh (targets fast_main.py, deleted in #567 -- a separate, unrelated cleanup commit), run-playwright-tests.sh/test_terminal_input.sh (wrong test-file locations, a deprecated main.py entrypoint), and complete-vm-sync-templates.sh (autobot-user-backend and docker/npu_worker never existed; duplicates the ADR's actual named tools) never touched docker-compose.hybrid.yml or any Docker-modernization casualty at all.

What Changed

Retired (script + zero dangling references, confirmed via git grep), first batch:

• fix-frontend-dependencies.sh, start_containers.sh, utilities/batch-configure-vms.sh

Retired, second batch (14-script blind-spot sweep) -- each with evidence independent of #15322 (see Thinking Path for the full per-script reasoning):

• deployment/start_npu_worker.sh -- Dockerfile it builds from was deliberately retired with no replacement (bf0a3b1af, 2026-03-15, explicit "obsolete"); no npu-worker docker-compose service exists anywhere in the tree; the script's own header names the real, current, actively-maintained replacement (native/Ansible).
• ensure-frontend-dependencies.sh -- run_agent_unified.sh deleted in [Housekeeping] Archive Cleanup - Extract Features and Delete Archived Files #567; its live-npm install-inside-the-container technique is incompatible with the current read_only: true, cap_drop: [ALL], static-nginx autobot-frontend service.
• run_agent_fast.sh -- targets backend.fast_main:app, deleted in [Housekeeping] Archive Cleanup - Extract Features and Delete Archived Files #567.
• setup/models/setup_model_sharing.sh, setup_windows_only_models.sh -- wrong container name / hardcoded personal Windows username / an .env+.yml output scheme nothing reads (each independent of any missing compose file; the compose files these two name are different ones, not run_agent.sh (the documented 'Start full system' entrypoint) is broken: docker-compose.hybrid.yml was deliberately retired and 13 callers were never repointed #15322's).
• testing/run-playwright-tests.sh, test_terminal_input.sh -- target e2e specs that live under autobot-frontend/tests/e2e/, not the autobot-slm-frontend/ tree they cd into; the former also shells out to the deprecated root main.py (feat(security): Migrate services to mTLS authentication (PKI-based) #725/refactor: Reorganize repository folders by deployment role #781) with a --test-mode flag it never had.
• utilities/complete-vm-sync-templates.sh -- autobot-user-backend and docker/npu_worker don't exist; duplicates the raw-rsync workflow the canonical ADR (docs/adr/005-single-frontend-mandate.md) names sync-frontend.sh/sync-to-vm.sh as the actual tools for.

Wired in -- utilities/enable-phase4-enterprise.sh: fixed the bash-executed """ docstring, added the missing X-Internal-API-Key auth header (admin-gated router; same convention autobot-slm-backend/api/voice_proxy.py uses), dropped one dead /api/enterprise/health call. Documented with a pointer to docs/prd/enterprise-features-subsystems-PRD.md -- most enable_feature() paths are still logger-only stubs per that PRD.

Wired in, second batch:

• testing/manage_playwright.sh -- the compose file it starts is real (container_name: autobot-playwright) but was a bare relative path that only resolved from inside docker/compose/ itself; resolved from the script's own location.
• utilities/sync-npu-worker-to-windows.sh -- SOURCE_PATH was missing the autobot-npu-worker/ path segment; confirmed install.ps1/check-health.ps1 exist at the corrected path.

Recorded as manual operator tools (scripts/README.md): cleanup-legacy-python.sh (first batch); check_ai_stack_health.sh, setup/setup_tier2_research.sh (both hardened 6 days ago, #14877/#14867, against real current targets -- check_ai_stack_health.sh prints a suggested recovery command that also names the missing docker-compose.hybrid.yml, but that's one line of advice in an error path, not something the script itself depends on to run; #15322 covers it), environment/set-env-deepseek.sh (every env var it sets is live/SSOT-recognized), utilities/prevent-local-frontend.sh (content-consistent with the real sync-frontend.sh workflow -- never wired to a caller, but not broken).

Guard restored, blind spot closed: repo_tests/unreferenced_shell_script_test.py kept, simplified to the down-only end state (no more KNOWN_UNREFERENCED subtraction -- every tracked script must be referenced, full stop). NOT_A_REFERENCE now also excludes pipeline-scripts/hardcoded_values_baseline.txt. repo_tests/unreferenced_shell_script_baseline.py stays deleted (AC4); nothing reintroduces a baseline.

#15322 corrected and re-prioritized (comment, not a code change here): the deletion this issue first cited (f4d5c5b151) was superseded by a later, more authoritative one -- bf0a3b1af (2026-03-15), which deliberately replaced the whole compose architecture and explicitly did not recreate an NPU-worker Dockerfile. Documented that Redis-related callers (run_agent.sh's autobot-redis lines) have a live repoint target in the current docker-compose.yml, while NPU-worker-related callers do not -- those need the Ansible path, not a repoint. Retitled to name run_agent.sh as the documented "Start full system" entrypoint (QUICK_REFERENCE.md:29), per the request to reflect that this is a broken front door, not housekeeping.

Verification

Established statically (no code executed, per constraint):

• bash -n parses cleanly on every touched .sh file.
• A standalone analysis script (git plumbing only, mirroring the guard's own unreferenced() logic exactly) run against the final tree state confirms zero unreferenced scripts remain under both the old and the corrected NOT_A_REFERENCE.
• git grep confirms zero remaining references anywhere in the tree to every retired filename.
• git log --all --diff-filter=D for every "deleted"/"never existed" claim above, read to identify the most recent deletion commit and its stated rationale, not just the first one found -- this is what caught the bf0a3b1af vs f4d5c5b151 distinction on review.
• git grep across every tracked *.yml for an npu-worker/autobot-npu-worker compose service definition: none exists; every match is Ansible config.
• Line-by-line comparison of enable-phase4-enterprise.sh's curl targets/jq paths against autobot-backend/api/enterprise_features.py's route decorators and response builders.

What the restored guard covers: TestEnumeration (sweep isn't silently empty or collapsed, #15087); TestOwnBookkeepingIsNotAReference (bookkeeping files -- now just this test module plus hardcoded_values_baseline.txt -- are tracked, exist, repo-relative); TestNoNewUnreferencedScript.test_every_script_is_referenced -- the down-only end state, unconditional now that there's nothing to grandfather; four regression classes pinning every already-landed decision in this issue to the specific document/defect-fix that makes it not-debris.

Left to CI: full pytest collection, pre-commit hooks (function-length, SSOT-sourcing self-test, secrets scan, mass-deletion detector) and the rest of the suite are CI's call, not asserted here.

Review round three: two mechanical bookkeeping consequences of retiring 11 scripts, caught by CI.

• pipeline-scripts/hardcoded_values_baseline.txt had 21 entries for the 11 retired scripts (verified via git grep -- every one belonged to a file this PR deleted, none belonged to a file that still exists, none needed adding). Removed exactly those 21; the file's header requires this, it only ever shrinks.
• repo_tests/git_hooks_executable_test.py's reach-floor test dropped two of its five floors below their asserted minimum: _SHEBANGED 45 < 46 and _UTILITIES 77 < 79 (both from the two retired scripts under utilities/: batch-configure-vms.sh, complete-vm-sync-templates.sh). This guard's existing convention was floors set AT the exact measured count, zero headroom, by design -- but review asked for real headroom instead this time (_SHEBANGED >= 40, _UTILITIES >= 70): a floor at the exact count goes red on the next legitimate small retirement and teaches whoever hits it to lower it again without looking, where real headroom still catches an actual regression (a reader silently returning nothing, or a directory falling out of _HOOK_DIRS) while surviving the next small one. Recorded inline with the count and reasoning.

Also merged origin/Dev_new_gui (branch was 1 commit behind) via merge, not rebase.

Worth flagging for the next batch retirement in this repo: removing tracked infrastructure scripts has three bookkeeping consequences here, not one -- the unreferenced-script guard's own baseline (repo_tests/unreferenced_shell_script_baseline.py, retired in this PR once it hit zero), the hardcoded-values baseline (pipeline-scripts/hardcoded_values_baseline.txt), and the shebang/hook reach floor (repo_tests/git_hooks_executable_test.py, if any retired script sits under a swept hook directory). All three failed silently-until-CI in this PR; none of them showed up from a plain git grep for the deleted filename.

Model Used

Claude Opus 5 (1M context)

Refs #15127

[github-actions]

✅ SSOT Configuration Compliance: Passing

🎉 No new hardcoded values of either class — ssot and other both block.

Known backlog in pipeline-scripts/hardcoded_values_baseline.txt is suppressed and tracked in #14371.

[mrveiss]

Addressed the review: guard restored (`repo_tests/unreferenced_shell_script_test.py` is back, simplified to the down-only end state -- `TestNoNewUnreferencedScript` and the enumeration/reach checks survive, only the `KNOWN_UNREFERENCED`-specific machinery is gone), and `pipeline-scripts/hardcoded_values_baseline.txt` is now in `NOT_A_REFERENCE`.

Closing that blind spot exposed 14 scripts, not the one anticipated -- verified this wasn't a basename-collision artifact, then decided all 14 on the same standard as the original five. 8 retired, 2 wired in (real path/auth defects, now fixed), 4 kept and documented. Full breakdown and evidence in the updated PR body.

One of the retirals (`ensure-frontend-dependencies.sh` and `deployment/start_npu_worker.sh`) shares a root cause with a separate, larger finding: `docker/compose/docker-compose.hybrid.yml` doesn't exist but is still referenced by 13 other tracked files, including the live `run_agent.sh` entrypoint. That's out of scope for a script-reference-triage issue and `run_agent.sh` needs its own review, so filed #15322 rather than touching it here.

A standalone analysis script (git plumbing only, same logic as the guard's own `unreferenced()`) confirms zero unreferenced scripts remain under the corrected `NOT_A_REFERENCE`.

[mrveiss]

Both retirements stand, but the stated reasons were wrong or incomplete -- corrected in the PR body, not in the commit (can't amend a pushed commit; the two file deletions themselves were correct, only the write-up was fixed):

• ensure-frontend-dependencies.sh never referenced docker-compose.hybrid.yml -- that was a drafting error, this script got conflated with its already-retired sibling fix-frontend-dependencies.sh while writing the commit message. Its real, independent defect: ./run_agent_unified.sh --dev (deleted in [Housekeeping] Archive Cleanup - Extract Features and Delete Archived Files #567) as the suggested fix, and a live docker exec ... npm install --save + Vite-restart technique that cannot work against the current autobot-frontend service (read_only: true, cap_drop: [ALL], static nginx build).
• deployment/start_npu_worker.sh does reference it, so this one needed real independent grounds, which exist: the Dockerfile it builds from was deliberately retired with no replacement in a later, more authoritative commit than the one first cited for run_agent.sh (the documented 'Start full system' entrypoint) is broken: docker-compose.hybrid.yml was deliberately retired and 13 callers were never repointed #15322 (bf0a3b1af, 2026-03-15, explicit "obsolete" framing), and no compose file anywhere in the tree defines an npu-worker service -- confirmed via git grep across every tracked *.yml. NPU-worker deployment moved to native/Ansible; the script's own header already says so.

Re-checked the remaining six retirements for the same defect class: none depend on anything #15322 covers. Full per-script reasoning is in the updated PR body.

Also updated #15322 itself: the deletion commit it cited was superseded by a later one, and I added the distinction that matters for whoever picks it up -- run_agent.sh's Redis lines have a live repoint target in the current docker-compose.yml, its NPU-worker line does not (that one needs the Ansible path). Retitled to name run_agent.sh as the documented entrypoint per your request.

[mrveiss]

Both fixed, pushed on the same branch (commit 4822bdd52, on top of a merge of origin/Dev_new_gui):

• hardcoded_values_baseline.txt: removed exactly 21 entries -- verified each belonged to one of the 11 retired scripts before removing anything (grep -c per deleted file summed to 21), and confirmed nothing else in the diff moved. None belonged to a file I hadn't deleted, so no stop-and-tell-you needed.
• git_hooks_executable_test.py: two of the five reach floors were below their measured count once you also account for _UTILITIES (which the CI log didn't show, since _SHEBANGED's assert fires first in the same function and aborts before _UTILITIES runs -- but it was genuinely broken too, 77 < 79). Old floor -> new floor: _SHEBANGED 46 -> 40 (true count 45), _UTILITIES 79 -> 70 (true count 77). Both now sit meaningfully below the true count rather than at it, per your two constraints; the assertions themselves are untouched, only the two numeric floors and the inline comment explaining them.

Branch is current with Dev_new_gui -- merged (not rebased), confirmed via git merge-base --is-ancestor origin/Dev_new_gui HEAD.

Added a note to the PR body flagging the three-bookkeeping-consequences pattern (unreferenced-script baseline, hardcoded-values baseline, shebang/hook reach floor) for the next person doing a batch retirement here.