CRITICAL regression(slm): self-update fails in ~1s every run after #12775 — caller's finally unlinks the temp inventory/extra-vars out from under the now-detached playbook; API still returns success:true and status reports has_update:false #12803

Summary

After #12775 (a479aa57a1, detach self-update into a transient systemd service instead of a --scope) landed and deployed, POST /api/code-sync/self-update fails within ~1 second on every invocation. The detach itself works correctly — the transient unit is created exactly as intended — but ansible-playbook can no longer read the per-run inventory and extra-vars files, because the caller deletes them while the now-detached playbook is still starting.

Reproduced 3/3 times on a node running the post-#12775 SLM tier (d41881d97).

Evidence

The transient unit starts correctly (this part of #12775 is working):

sudo[373395]: autobot : COMMAND=/usr/bin/systemd-run --collect --wait
  --unit=autobot-selfupdate-364579-1785157402 --slice=system.slice --uid=999 --gid=999
  --working-directory=/opt/autobot/code_source/autobot-slm-backend/ansible
  --setenv=ANSIBLE_LOCAL_TEMP=/tmp/ansible_local_tmp_999
  --setenv=ANSIBLE_INVENTORY=/tmp/ansible_local_tmp_999/autobot_inv_8bltpymq.yml
  -- /bin/sh -c 'exec "$0" "$@" >> /var/log/autobot/self-update-ansible.log 2>&1'
  /usr/bin/ansible-playbook -i /tmp/ansible_local_tmp_999/autobot_inv_8bltpymq.yml
  .../update-all-nodes.yml --limit localhost,00-SLM-Manager
  -e @/tmp/ansible_local_tmp_999/autobot_evars_393ur4eq.json

…and then dies immediately:

systemd[1]: autobot-selfupdate-364579-1785157402.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: autobot-selfupdate-364579-1785157402.service: Failed with result 'exit-code'.

/var/log/autobot/self-update-ansible.log:

[WARNING]: Unable to parse /tmp/ansible_local_tmp_999/autobot_inv_8bltpymq.yml as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
ERROR! Unable to retrieve file contents
Could not find or access '/tmp/ansible_local_tmp_999/autobot_evars_393ur4eq.json' on the Ansible Controller.

Not a permissions problem — --uid=999 --gid=999 is the autobot user, and the temp dir is owned by it:

$ id autobot
uid=999(autobot) gid=999(autobot)
$ stat -c '%U:%G %a %n' /tmp/ansible_local_tmp_999
autobot:autobot 700 /tmp/ansible_local_tmp_999

It is a lifetime race. Polling the temp dir every 400 ms across a live self-update shows the contents existing only momentarily:

16:11:40.754  files=0
16:11:42.969  files=1   ansible-local-382734j3wq60r1
16:11:43.413  files=0

The unit failed at 16:11:43 — the same instant the directory emptied. autobot_inv_*.yml and autobot_evars_*.json never survived a single poll interval.

The deletion is unconditional, in a finally on the calling coroutine — services/playbook_executor.py:947-957:

finally:
    # Clean up the per-run temp inventory and extra-vars files
    if dynamic_inv_path is not None:
        try:
            dynamic_inv_path.unlink(missing_ok=True)
        ...
    if extra_vars_file is not None:
        try:
            extra_vars_file.unlink(missing_ok=True)

Before #12775 the payload ran inside a --scope, i.e. as a descendant of the caller, so the caller could not outrun it. A transient service is reparented to PID 1 and started asynchronously by systemd, so the caller's finally now wins the race and removes the two files out from under it. --wait is present in the argv but is evidently not holding the caller across the window in which the files are needed.

Impact — self-update is fully broken, and it reports success

• Every self-update attempt fails in ~1 s. The app tier cannot be updated through the builtin path at all.
• The API still returns {"success": true, "message": "Full update queued..."} — success is reported for a run that never executes. job_id is null, so there is nothing to poll.
• Observed end state on this node: SLM tier at d41881d97, app tier stranded at 6424820ce.

The status endpoint actively misreports this. After the first (pre-fix) run pulled code_source to the new commit, /api/code-sync/status returns:

{"latest_version":"d41881d97...","local_version":"d41881d97...","has_update":false,
 "outdated_nodes":2,"total_nodes":2,
 "stale_components":["autobot-ai-stack","autobot-backend","autobot-npu-worker","autobot-slm-agent","plugins"]}

has_update: false alongside five stale components and outdated_nodes: 2 in the same payload. local_version is read from code_source HEAD — which the pull advances — rather than from what is actually deployed to the app tier. So once code_source moves, the GUI reports "up to date" permanently, regardless of whether Plays 2/3 ever ran. This is the mechanism behind #12776 and it makes the failure invisible in the Updates view.

Verified by content hash rather than by the marker (per #11512):

autobot-slm-backend/api/code_sync.py               3024e40e4c0c -> a6619085396f   UPDATED
autobot-slm-backend/services/playbook_executor.py  2d640d7c7300 -> a31bff17eb31   UPDATED
autobot-backend/api/voice_bundle_user.py           3584c0432a07 -> 3584c0432a07   STALE
autobot-backend/auth_rbac.py                       8624878bb32e -> 8624878bb32e   STALE

Proposed fix

1. Do not delete files the detached run still owns. Either hand the temp files' lifetime to the transient unit (write them into a directory the unit removes itself, e.g. via RuntimeDirectory/--property=ExecStopPost=), or pass the inventory and extra-vars inline so there is no shared-file dependency, or have the caller unlink only after the unit has actually terminated. A finally on the caller is the wrong scope now that the payload outlives it by design.
2. Verify --wait semantics — it is in the argv but did not serialise the caller against the payload here; if it is being bypassed (fire-and-forget task, timeout, or an intermediate sudo), that should be made explicit rather than relied upon.
3. Stop returning success: true for a queued run that has not been verified. Populate job_id and surface the transient unit's exit status, so a run that dies in 1 s cannot present as a successful update.
4. Fix local_version to reflect the deployed app tier, not code_source HEAD — otherwise has_update:false will keep contradicting stale_components in the same response.

Related

• fix(slm): detach self-update into a transient systemd SERVICE, not a --scope (#12596) #12775 / deploy: Update-All STILL skips Play 2 — #12567 detach activates but the transient scope is killed by the Play-1 SLM restart (incomplete #12425) #12596 — the detach change this regresses from; the detach itself is correct, the temp-file lifetime was not updated to match
• deploy: nothing reads the self-update log post-hoc — Update-All can silently skip Play 2/3 and still report success #12776 — nothing reads the self-update log post-hoc; this is exactly that blind spot
• deploy/code-sync: most deployed services (ai-stack, npu/tts/browser workers, celery, slm-agent, plugins) have no builtin per-component update path or drift signal — only the broken Update-All #12450 — per-component update paths
• deploy: Update-All silently leaves co-located backend/frontend stale — #11492 detach falls back to attached-run when /var/log/autobot isn't writable by autobot #12425, CRITICAL: one-click update never redeploys the SLM control plane — self-update skip reads code_source HEAD (advanced by the pull) not the deployed commit #12202 — earlier instances of the app tier being left stale by Update-All
• discovery(code-sync): node version marker 484 commits stale while node runs the latest code — false 'outdated_nodes' status #11512 — verify deploys by ground truth, not the version marker

Verification

curl -X POST -H "Authorization: Bearer $T" http://<slm>:8000/api/code-sync/self-update
journalctl | grep autobot-selfupdate | tail
sudo tail /var/log/autobot/self-update-ansible.log
curl -H "Authorization: Bearer $T" http://<slm>:8000/api/code-sync/status