Repository navigation
tech-debt(infrastructure): 18 sys.path bootstraps open-code the project root and default to the live install instead of the checkout #14544
Description
Activity
Raising the severity: this is a correctness hazard, not only tech debt
Filed as
tech-debt, but the failure mode deserves the sharper framing, because it is silent and it crosses an environment boundary.sys.path.insert(0, os.environ.get("AUTOBOT_PROJECT_ROOT", "/opt/autobot/code_source"))
When the variable is unset — which is the ordinary case in a fresh checkout, a worktree, or a CI runner — this does not fall back to the checkout. It points
sys.pathat the live install.So a script run from a developer's tree imports production code from
/opt/autobot/code_source. The script works, which is the problem: nothing errors, nothing warns, and the behaviour observed is the deployed system's, not the branch under test. That is the same shape as the defects #14517 just fixed — a path that resolves to somewhere plausible and wrong — except here the wrong answer is live code rather than an empty directory.Two consequences worth stating explicitly:
- A change under test can appear to work when it was never loaded. Editing a module in a worktree and running one of these 18 scripts exercises the installed copy. The repo already treats "verify against the codebase, never
/opt/autobot" as a standing rule; this inverts it by default. - It couples a checkout to a machine's install state. On a box with no
/opt/autobot, the insert is inert and imports fail confusingly; on a box with a stale install, they succeed against old code.
Why it is genuinely not a sweep, as the issue says
These lines run before
autobot_sharedis importable — that is their whole purpose — soautobot_shared.paths.project_root()cannot be the fix at that point. It needs a bootstrap that derives the root from__file__(the patternutilities/manage_system_knowledge.pyuses viaPath(__file__).resolve().parents[N], per #14129) rather than from an env var with an install-path default.That is a design decision about how these scripts locate the tree, which is exactly why splitting it out of #14517 was right.
Suggested acceptance criteria
- No
sys.pathbootstrap defaults to an absolute install path. Deriving from__file__is the default; an env var may override, never supply the fallback. - A guard asserting that — the same shape as
check_no_shell_placeholder_paths.pyfrom bug(infra): 32 unexpanded ${AUTOBOT_PROJECT_ROOT} placeholders in plain Python string constants, invisible to every flake8 code #14517: fails on any new occurrence, fails on an empty enumeration, and names any deliberate exception with a self-check so a rename cannot strand it. - Confirm the 18 sites still work when
/opt/autobotdoes not exist at all, since that is the state on a fresh clone.
Related: #14517 (the sibling defect class, fixed), #14129 (the
__file__-relative bootstrap pattern), #13149 (canonical project-root resolution), #14543 (scripts calling into a directory that exists nowhere).- A change under test can appear to work when it was never loaded. Editing a module in a worktree and running one of these 18 scripts exercises the installed copy. The repo already treats "verify against the codebase, never
Found a same-class, different-mechanism instance while working #14524 (PR #14548).
PlaybookExecutor.__init__(autobot-slm-backend/services/playbook_executor.py:229-238):if ansible_dir is None: ansible_dir = Path( os.getenv("SLM_ANSIBLE_DIR", "/opt/autobot/code_source/autobot-slm-backend/ansible") ) if not ansible_dir.exists(): ansible_dir = Path("/opt/autobot/autobot-slm-backend/ansible") self.ansible_dir = ansible_dir
Not a
sys.pathbootstrap, but the identical shape this issue already tracks: the FIRST fallback is existence-checked, the SECOND is not —self.ansible_dircan end up pointing at a directory that does not exist on the current machine, with no error raised until something later tries to use it (_run_subprocess'scwd=str(self.ansible_dir)at subprocess-spawn time, several calls removed from construction).Concretely reproducible on any machine that is not the deployed node (CI runners included — a bare
PlaybookExecutor()construction there resolves to/opt/autobot/..., which does not exist, and the resultingFileNotFoundErrornames a path the caller never chose). Two of #14524's own tests hit exactly this before being fixed to pass an explicitansible_dir.Not fixed in #14524's PR — flagging here per that PR's own review, rather than a fourth mechanism instance quietly going untracked.
Delivered by PR #14575, squash-merged as
6dc9c0721and verified inorigin/Dev_new_gui: the resolver raises, and all five runtime declarations are present.18 sites across 17 files (the issue's prose said 16; its own list carried
debug/debug_terminal_detailed.pytwice). Re-derived independently at the merge base, with three separate sweeps finding zero missed Python variants.The more valuable find was inside the fix.
autobot_shared/paths.py's own last-resort branch returnedPath(os.environ.get("AUTOBOT_BASE_DIR", "/opt/autobot"))— the canonical resolver everything was being migrated to carried the defect they were migrating away from. It now raisesProjectRootUndeterminablenaming both overrides.That raise broke the containers, and CI caught it before anything shipped.
smoke-testandhardened-smoke-testwent red with the raise firing from/app/autobot_shared/paths.py: the image has no.git, no.env, and set neither variable. So the answer to "does the deployed tree carry the markers" was no — and the old silent guess had been masking it.Five runtimes were relying on that guess coincidentally equalling
/opt/autobot, which is where each happens to run: both container images,autobot-slm-backend.service.j2,slm-agent.service.j2, andai-stack.env.j2. Each now declares its root explicitly.The ai-stack one was missed in the first round and found in review:
ai_api_server.pywas classified as not importingautobot_shared, butai_api_server.py:24→agents/base_agent.py:21→ssot_config.py:63(PROJECT_ROOT = project_root()at module scope) — and the unit template's own comment says "ai_api_server imports autobot_shared and otherwise crash-loops (#10121)". A direct-import grep is not a transitive check. Its survival had been topology-dependent: safe under the role default, broken underdeploy-backend-{local,remote}.yml'sbackend_install_dir: /opt/autobot.The npu-worker declaration was reverted rather than kept — its justification cited a module that unit does not run, and a guard test pinning a false rationale is worse than no line.
Stated rather than hidden, in the PR body:
AUTOBOT_PROJECT_ROOTis step 1 and returns before the walk, so container paths move from/opt/autobotto/app; the chicken-and-egg proof covers 7 of 17 files, not all 17; and "18/18" means the defect was fixed 18 times, not that those 18 scripts now run end-to-end — none do, which is a separate pre-existing bug.Follow-ups: #14590 (nine dead duplicate unit files, evidenced not assumed, filed rather than deleted) and #14591 (snapshot/PKI paths landing outside the hardened
tmpfs:overrides — pre-existing, not a new regression).- added 5 commits that reference this issue
on Aug 21, 2026 - added a commit that references this issue
on Sep 5, 2026
Problem
18
sys.pathbootstraps across 16 files resolve the project root by hand:This is
autobot_shared.paths.project_root()open-coded, and it is the exact form #14517 tells contributors not to write. It is not merely stylistic duplication — the two answers differ:project_root()walks up from the importing file and stops at the first ancestor that holds a.envor looks like a source checkout, then falls back to the install. From a checkout it resolves to that checkout.AUTOBOT_PROJECT_ROOTunset — the normal case in a checkout — it resolves to the live deployed install, so a script run from a working tree puts the deployed tree first onsys.pathand silently imports the deployed copy of every first-party module. That is the same "defaults to the live install" failure bug(scripts): run_agent.sh defaults to the live install and rm -rf's its node_modules from any checkout #13092 fixed for one shell script.Sites
autobot-infrastructure/shared/scripts/analysis/debug_circuit_breaker.py:18autobot-infrastructure/shared/scripts/analysis/redis_final_analysis.py:19autobot-infrastructure/shared/scripts/analysis/redis_vector_analysis.py:22autobot-infrastructure/shared/scripts/analysis/simple_kb_test.py:15autobot-infrastructure/shared/scripts/analysis/test_autobot_identity.py:15autobot-infrastructure/shared/scripts/analysis/test_data_layer_debug.py:198autobot-infrastructure/shared/scripts/analysis/test_documentation_api.py:19autobot-infrastructure/shared/scripts/analysis/test_kb_documentation.py:15autobot-infrastructure/shared/scripts/analysis/test_kb_search.py:16autobot-infrastructure/shared/scripts/analysis/test_redis_comparison.py:23autobot-infrastructure/shared/scripts/test_llm_awareness.py:18autobot-infrastructure/shared/scripts/test_long_running_operations.py:36autobot-infrastructure/shared/tests/ci_cd_integration.py:41autobot-infrastructure/shared/tests/comprehensive_test_suite.py:47autobot-infrastructure/shared/tests/performance/test_performance_optimization.py:43debug/debug_terminal.py:22debug/debug_terminal_detailed.py:25debug/debug_terminal_detailed.py:187Why this was not folded into #14517
#14517 swept the string constant form: a shell placeholder in a plain literal, which resolves to a directory that cannot exist. These lines are a different shape — they work, they just answer the question wrongly — and they need a design decision #14517 had no room for.
Specifically, this is not a straight substitution. These inserts run so that first-party packages become importable; swapping in
from autobot_shared.paths import project_rootmakes the file depend onautobot_sharedbeing importable before the line that arranges for imports to work. That ordering is fine where the deployment already putsautobot_sharedonPYTHONPATH(it does, for the backend/Celery/MCP units) but is not proven for every standalone invocation of these scripts.#14517 therefore imports
project_rootat module scope in the four files it touched that also carry one of these inserts, and deliberately left the inserts themselves alone rather than reorder a bootstrap it had no evidence about.Fix
Decide the bootstrap contract once, then apply it everywhere:
autobot_sharedis guaranteed importable for every entry point in these trees, replace all 18 withsys.path.insert(0, str(project_root()))and delete the fallback.autobot_shared.pathsa documented stdlib-only bootstrap entry point these scripts may call before anything else, and route all 18 through it.Either way the fallback-to-the-live-install behaviour must go: from a checkout it is a wrong answer that looks like a working one. Guard it afterwards with no allowlist, the way #14517's guard does for the literal form.
Related
code-qualityguard for it; this is the same class one layer downsys.path.appendsweep this pattern was introduced by