Repository navigation
SLM backend crash-loops on a deployed host: project-root raise lands where the systemd unit is stale #14624
Description
Activity
- added 6 commits that reference this issue
on Aug 19, 2026 Resolved on the host, and verified the honest way
The resolver fix (#14661) is deployed and proven:
drop-in removed, daemon-reload, restart state=active NRestarts=0 API login 200 ProjectRootUndeterminable in logs: 0 both fleet nodes online, heartbeats ~7sThe override was removed before the verifying restart. Leaving it in place would have produced an identical-looking result whether or not the resolver worked, which is not evidence.
The deployed layout satisfies the new arm on its own terms:
autobot_sharedbeside 12autobot-*component directories, no root.env, no.git.A finding that outlives this fix: self-update never re-renders the systemd unit
The original diagnosis was "this host's unit was two months stale". That was too generous. After a full self-update that pulled
code_sourcefromcd35d7ff6to7bf3a6da7and propagated the new code in 53 seconds:deployed unit AUTOBOT_PROJECT_ROOT count: 0 deployed unit mtime: 2026-06-18 19:05:37Unchanged. The unit template in the repo has carried
Environment="AUTOBOT_PROJECT_ROOT={{ slm_base_dir }}"for some time, and a full update still does not apply it.So the stale unit is not an accident on one host — updates propagate code and restart services but never refresh unit files. Any change that depends on a new unit setting will keep landing on hosts that cannot satisfy it, and the failure mode is a service that will not start, with no in-band recovery because the endpoints that deploy a fix live in the service that is down.
That is worth its own fix, independently of this one. #14661 removed the dependency on the unit for this setting; the next setting will hit the same wall.
Remaining
- 11 of 13 Python unit templates set no
AUTOBOT_PROJECT_ROOTat all and rely entirely on layout resolution — recorded and pinned by test(infra): pin which service units set the project-root override (#14624) #14662. - Why unit rendering is skipped during an update: not investigated here.
- 11 of 13 Python unit templates set no
Closing — PR #14668 referenced this with a close keyword but squash merges to
Dev_new_guinever auto-close, so it stayed open.Re-confirmed on the host just now, rather than on the merge:
autobot-slm-backend: ActiveState=active SubState=running NRestarts=0 systemctl cat autobot-slm-backend: Environment="AUTOBOT_PROJECT_ROOT=/opt/autobot" StartLimitIntervalUSec=2minThe
AUTOBOT_PROJECT_ROOTline the raise needed is present in the deployed unit, and it matches the template (autobot-slm-backend.service.j2:33). The restart counter is 0 against the 530+ recorded here.One thing worth recording, because it affected this issue after the fact: the unit-refresh path that delivers that line was itself broken between #14668 and #14693.
service_units.ymlhad two tasks needing root and neither declaredbecome, so the render failed withDestination /etc/systemd/system not writableand aborted Play 1 on every run. During that window the fix was in the repo and could not reach a host — so any node that had not already been refreshed stayed exposed to this crash-loop.StartLimitIntervalUSec=2minabove is the evidence the refresh is working again; it read the stale10sdefault until #14694 landed. Also relevant: at10swithRestartSec=5, the burst limit is unreachable, so a crash-looping unit never lands infailed— which is where #14702's rollback gate would look.
Impact
The SLM backend crash-loops on start and the API is unreachable. Restart counter passed 530.
uvicornexits 1 during module import, so nothing the service exposes is available — including the code-sync and self-update endpoints that would normally deploy a fix. The recovery path is inside the thing that is down.Cause
#14544 / PR #14575 replaced a silent project-root guess with a hard raise. That is the right direction, but on a deployed install every resolution arm fails:
AUTOBOT_PROJECT_ROOT/AUTOBOT_BASE_DIR— not set in the deployed unit.envarm —/opt/autobot/.envdoes not exist. The per-component files live one level down (autobot-backend/.env,autobot-ai-stack/.env, …). The resolver's own docstring assumes otherwise: "an install has a.envand no.git, so it matches on the.envarm". That premise does not hold for this layout.is_checkout_rootisall((path / m).exists() for m in (".git", "autobot_shared"))./opt/autobothasautobot_sharedbut no.git, soall(...)is False.The fix already exists and did not reach the host
roles/slm_manager/templates/autobot-slm-backend.service.j2:33sets:But the deployed
/etc/systemd/system/autobot-slm-backend.servicehas no such line and is dated 2026-06-18 — two months old. The template is correct in both the repo andcode_source; the unit was never re-rendered from it.So the change is only safe on a host whose systemd unit is current, and nothing verifies that before the new raise takes effect. A host that has not had its unit re-rendered gets a hard startup failure with no in-band way to recover.
What this needs
.envarm should recognise the real install layout, oris_checkout_rootshould not require.gitalongsideautobot_shared, or the deployment must be verified to carry the override before a release that can raise. A change that depends on deployed state being current should check that state, not assume it.slm_managerrenders it on every run, it should not be from June. That gap is independent of this crash and probably affects other deployed units.Recovery used
A systemd drop-in setting
AUTOBOT_PROJECT_ROOT=/opt/autobot— the override the raise itself names — mirroring exactly what the template would render. Stopgap only; removable once the unit is re-rendered.Related
project_root()this builds on