fix(infrastructure): make a missing script dependency fail loudly, and guard the class (#14041, #14867, #14518, #14172)

[mrveiss]

Thinking Path

#14041, #14172, #14518 and #14867 were filed separately but describe one defect
with one mechanism: a missing dependency is discarded instead of raised, so
the script runs in a degraded mode nobody can see. A shell script sources a
config library that is not there and continues on hardcoded literals. A Python
script imports a module that does not exist and dies before its first
statement. A shell script runs embedded Python that cannot import, throws the
traceback away, and prints a reassuring number.

Every one of them passes review, because the source text describes the
intended behaviour. Only the machine knows the difference.

So the fix has two halves, and both are required in each surface:

(a) make the missing dependency fail LOUD at the point of use — the value
of a check is entirely in the difference between "ran and was clean" and "could
not run", and discarding stderr erases exactly that difference;
(b) add a repo-wide guard so the class cannot be reintroduced silently —
shellcheck does not follow a dynamic source path, and pyflakes never checks
that an import target resolves, so nothing in the repo could see any of this.

Where a target could not be identified with confidence, the import is left
alone and tracked. Guessing would replace a loud ModuleNotFoundError with a
quiet wrong answer, which is the defect this work exists to remove.

The #14172 scope decision

#14172 carried needs-decision, asking whether the 56 source lines should
change shape or whether a repo guard alone should cover it. This was an
autonomous run, so the recommendation, the rationale and the rejected
alternative were posted to the issue
and the work proceeded rather than blocking.

Decision taken: change the call-site shape. A guard only sees this checkout
at PR time; it cannot see a bad deploy, a .gitignore regression or a rename
downstream — the three recurrence scenarios the issue itself names — because in
every one of those the repo is green and the machine is missing the file. The
guard ships too, as the second half rather than the whole. The reason for the
silent degrade also expired when #14041 built the library, and the tree already
has the fail-loud convention: lib/db-update-classify.sh and
scripts/lib/project_root.sh are both sourced bare, one of them two lines
below an ssot-config source that was guarded with || true.

Stated plainly: a node that legitimately ships only part of the tree now
gets a loud abort where it previously got a quiet wrong value. That is the
intended trade and the direction the issue asked for, but it is a behaviour
change on deployed hosts, and it is the part worth overruling on if the fleet
has such nodes. That needs host observation, which an autonomous run cannot
produce.

What Changed

(a) Fail loud at the point of use

56 shell scripts (#14041, #14172) — source X 2>/dev/null || true becomes
an explicit failure block that writes to stderr and exits non-zero.

• 51 single-path sites converted directly.
• 5 two-path sites keep 2>/dev/null on the first attempt only — that miss
  is expected, callers sit at two different depths — while the last attempt
  keeps its stderr and is fatal. The old fallback re-read .env and carried on.
• 2 already-bare sites get the same block, so one shape ships everywhere.
• The die uses return 1 2>/dev/null || exit 1, so a script that is itself
  sourced returns instead of killing the parent shell — the idiom
  ssot-config.sh already uses at its own re-entry guard.
• test_ssot_config_lib.sh asserted the old contract ("missing library
  degrades to literal"). It now asserts the inversion and executes the blocks
  as shipped, extracted with sed out of four real scripts, so reintroducing
  || true trips the test rather than passing a re-typed copy of the idiom.

39 of 44 Python imports (#14518) — rewritten to the real module plus the
sys.path insert the operator entry points in this tree already use (#14129,
#14507), with parents[N] computed per file from its own depth. aioredis is
migrated to redis.asyncio (the alias this repo already uses), adding no
dependency; docker gains the same ImportError guard its five other call
sites carry. Four further sites outside #14518's own enumeration
(utils.redis_database_manager, utils.logging_manager) were found by the
sweep and repointed at the canonical autobot_shared modules.

8 deployment scripts (#14867) — 7 of 9 embedded imports now name a real
module, with a PYTHONPATH derived from each script's own location. All 9
swallowed failures are gone: the traceback reaches stderr and the exit status
reflects it. Summary lines that printed unconditionally ("Health check
complete", "All checks passed", "Startup coordinator is ready for use") are now
gated on a failure counter. Three latent defects of the same shape surfaced and
were fixed: a ${PROJECT_ROOT} interpolated inside a quoted heredoc (so the
path was a literal string and "CLAUDE.md not found!" was guaranteed, returning
0); a PYTHONPATH pointing at a package's contents rather than its parent; and
a Python """ docstring at the top of a bash script, which bash executed as
commands on every run.

(b) Repo-wide guards

Three new files under repo_tests/ — that location is deliberate, because
autobot-infrastructure/ is in no pytest testpaths entry and in no CI pytest
invocation, so a test beside the scripts would be as dormant as the defects.

guard	watches	surface no existing check reads
shell_lib_sources_resolve_test.py	every source .../lib/*.sh resolves, and the decisive attempt in each || chain is not silenced	shellcheck does not follow a dynamic source path
infra_script_imports_resolve_test.py	imports under shared/scripts/*.py resolve	pyflakes never checks that an import target exists
deployment_script_imports_resolve_test.py	imports inside python3 -c and heredocs resolve; no check answers with a fabricated value	the Python is a shell string — no linter parses it at all

Verification

No code from the codebase was executed. Verification is static analysis plus
the CI run on this branch. Each guard was copied to a scratch location,
parameterised on the tree root, and run against both the pristine
origin/Dev_new_gui (git archive) and this branch.

Measured guarded/total — counted, not asserted

guard	violations on the pristine base	after this PR	coverage
shell lib sources	57 of 57 sites flagged, across 56 of 56 scripts	0	56/56 scripts — the exact population #14041 enumerated
shared/scripts/*.py imports	42 sites present, 34 flagged	0 unflagged	8 are the tracked residue below
inline Python in *.sh	27 sites present, 15 flagged	0 unflagged	12 are the tracked residue below

Result of running all three guards:

• this branch — 0 failures (13 tests, including 76 parametrized cases)
• pristine base — shell guard fails both aggregate assertions plus 57 of
  57 parametrized sites; the other two guards red on their own populations

Anti-fail-open measures

An empty or absent result reads as "clean", so every sweep asserts presence
of what it expects before asserting absence of failure:

• discovery floors on files walked, sites matched, inline blocks extracted,
  declared distributions found, and first-party roots still present. The
  block-count floor is load-bearing for the shell guard — an extractor that
  stopped recognising python3 -c would otherwise pass having read nothing.
• skip lists match relative paths, never absolute, so a checkout living
  under a directory named .worktrees or venv cannot skip every file and
  report clean.
• every exemption is asserted still broken AND still reported by the sweep.
  That second assertion is the positive control for each file: if a detector
  regresses, the exemption test goes red instead of the whole file passing
  vacuously.
• documentation and nested shell strings are excluded structurally (an
  unbalanced quote, or a backslash-escaped quote — neither occurs in a real
  source statement), not by naming files, so test_ssot_config_lib.sh can keep
  driving the very shapes it tests. A dedicated test asserts those exclusions
  exclude exactly what they claim.
• shell verification used bash -n (parse only) and compile(..., "exec") on
  the extracted Python payloads — neither executes anything. All 8 scripts are
  #!/bin/bash; no dash script is in the set, and no set -o pipefail was
  added.

Enforcement caveat — measured, not assumed

Stated plainly so nobody reads more into "repo-wide guard" than it delivers.
The three guards land in repo_tests/, which is collected by the Python suite
(coverage.yml) — and the Python suite is not a required status check on
this branch. The ten required contexts are No commit trailers,
No open blocks-merge issues reference this PR, Unit & Integration Tests
(the frontend job, per frontend-required-context.yml:84), api-wiring,
code-quality, migration-matrix, smoke-test, startup-import-smoke,
verify-generated-types and verify-precommit-config.

So these guards will run and go red visibly, but they will not block a merge
until #13162 is resolved. That is a property of the CI configuration, not of
the guards, and it is the same for the existing
first_party_imports_resolve_test.py this work generalises.

The related gap on the bash side is filed as #14878: the
ssot-config-lib-guard pre-commit hook is a regression suite, not a formatter,
but enforce-precommit.yml routes all hook findings to ::warning:: — only
a hook that could not execute at all fails the build. There is also no
pre-commit hook installed in the local git checkout, so the suite did not
run at commit time either.

Issues this PR closes, and the two it does not

Closes #14041, #14172. Both are fully delivered: the library existed
already, a missing bootstrap now fails loudly, and the repo check exists.

#14518 and #14867 are advanced but NOT closed, because partial delivery
never closes an issue:

• bug(infra): 44 imports under shared/scripts whose target does not resolve — F821-clean scripts that still cannot run #14518 — 39 of 44 sites fixed. Five remain, each naming a symbol that was
  deleted or never landed: SSHManager (0 hits repo-wide), add_api_routes
  (deleted; its successor contains no workflow_router, so repointing would
  make the script's assertion falsely red), and monitoring_compat
  (dissolved by Integrate: monitoring_compat.py — unregistered router with deprecated endpoints #1283). No import spelling reaches any of them. Tracked as
  bug(infra): 5 imports under shared/scripts name symbols that were deleted or never landed (#14518 residual) #14870.
• nine deployment/setup scripts embed Python importing packages that do not exist, with the failure discarded #14867 — 7 of 9 imports fixed, 9 of 9 swallowed failures fixed. Two
  remain: BackupEngine and get_agent_orchestrator, both zero hits
  repo-wide. What remains now fails loudly rather than silently. Tracked as
  bug(infra): 2 embedded imports in deployment scripts name an engine and an accessor that were never built (#14867 residual) #14875.

Filed rather than folded in silently: #14870, #14871 (langchain_redis
undeclared — its absence silently drops an arm of the comparison the scripts
exist to measure), #14872 (the docker SDK is imported by five production
backend modules and declared in no requirements file), #14873
(optimize_agents forked across two modules sharing one import name),
#14875, #14876.

Merge-order note — PR #14868 landed mid-review, branch rebased

This PR originally predicted a collision with PR #14868 (#14866). It happened,
and is resolved.

Two file conflicts, both in the scripts #14867 wrongly described as already
fixed. #14866's PYTHONPATH export is kept in full; only its
source ... 2>/dev/null || true is replaced by the #14172 fail-loud form.

One add/add conflict: #14868 added its own
repo_tests/deployment_script_imports_resolve_test.py — the filename #14867
predicted. Rather than ship a second guard, this branch's copy is merged into
the landed file (consolidate, never fork). The landed version is the survivor
because it carries a call-layer check this branch's did not: an awaited sync
Redis client is a failure that moves from import time to call time, which no
import guard can see. Three things merged in from here:

• the importing script's own directory now counts as an import root. These
  scripts export ${SCRIPT_DIR} on PYTHONPATH and import siblings; without it
  the landed guard reported a false positive on test_startup_coordinator.sh,
  whose startup_coordinator.py sits right beside it.
• test_no_inline_check_fabricates_its_own_result — nine deployment/setup scripts embed Python importing packages that do not exist, with the failure discarded #14867's second acceptance
  criterion, and the half fix(deploy): make the access-control validator and monitor import modules that exist (#14866) #14868 left undone. It removed the 2>/dev/null but
  kept || echo "{}", so a check that cannot run still reports a value
  indistinguishable from a clean one. Four such sites remain in
  access_control_monitor.sh (UNKNOWN, {}, {}, 0|0|0); filed as
  bug(monitoring): the access-control monitor still answers a failed check with UNKNOWN/{} (#14866 second half) #14880 and carried as a self-guarding exemption.
• seven of its nine _KNOWN_BROKEN entries are retired, because this branch
  fixed those imports. The two that remain are repointed at bug(infra): 2 embedded imports in deployment scripts name an engine and an accessor that were never built (#14867 residual) #14875.

The new fabrication detector was wrong on its first draft, and this is worth
recording because it is the failure mode this whole PR is about. It matched
python3 -c to end-of-line, which finds nothing for a multi-line block —
the || echo sits after the closing quote, several lines down. It reported
zero fabricated results on a file that has four. Nothing about that looked
wrong: the test passed. Its own positive control ("this exemption must still be
broken") is what turned it red. That is precisely why every exemption in these
guards asserts it is still reported by the sweep, not merely still broken.

Model Used

Claude Opus 5 (1M context)

Closes #14041, #14172

[github-actions]

✅ SSOT Configuration Compliance: Passing

🎉 No hardcoded values detected that have SSOT config equivalents!