Repository navigation
bug(update): the SLM self-update fires without executing, and the status surface replays a stale verdict from an earlier run #15522
Description
What happens
The SLM self-update fires and silently does nothing, while the status surface keeps replaying a stale verdict from an earlier run — so the UI reports a failure that did not come from the run you just triggered.
Observed on the live SLM node today. Two firings:
| time | what the logs show |
|---|---|
| 11:36:04 | Firing Ansible self-update … (fire-and-forget) → inventory written → Executing Ansible playbook → Self-update run detached into transient systemd service → ansible log written until 11:38 → 1 failed task |
| 12:47:04 | Firing Ansible self-update … (fire-and-forget) → nothing at all |
For the 12:47 run there is: no playbook_executor line (the last one is 11:42), no write to the ansible log (mtime still 11:38), no transient systemd unit, and no inventory file. The run never reached the executor.
Meanwhile, every ~60 seconds since:
services.self_update_log_reader - ERROR - self-update verdict: self-update finished with 1 failed task(s)
That verdict is the 11:36 run's result, re-read from a log last written at 11:38 and re-emitted indefinitely. It is logged twice per cycle. The deployed marker confirms nothing advanced: .deployed_commit still reads the 11:36 value, while code_source moved on at 11:42.
Why it is hard to notice
- "fire-and-forget" means a launch failure is invisible. The code logs that it fired and returns. If the detached run never starts, nothing distinguishes that from a run in progress.
- The stale verdict fills the gap with a wrong answer. Because the reader re-reports the previous run's outcome, the surface is never blank — it shows a plausible-looking failure. An operator who triggers an update and sees "1 failed task" will reasonably believe that is their run's result. It is six hours old.
- Together these mean an update can appear to have run and failed, when in fact it never executed.
Acceptance criteria
- Establish why the 12:47 firing never reached
playbook_executor. The 11:36 path wrote its inventory under the deploy root and 11:42 wrote to a temp dir — determine whether the inventory directory being unwritable, or the transient-unit launch failing, is the cause. - A fired self-update that does not start must fail loudly rather than return. Fire-and-forget is acceptable for the run's duration; it is not acceptable for the launch itself.
- The verdict reader must not report a run older than the firing it is being asked about. Bind the verdict to a run identity (timestamp or run id) and report "no result yet" rather than the previous run's outcome. A stale answer is worse than no answer here, because it is indistinguishable from a fresh one.
- Establish which task failed in the 11:36 run, and whether it is the same failure that has been recurring. The ansible log is mode
0600owned by the service account, so this needs either a readable location or a surfaced summary — an operator cannot currently see which task failed. - Verified by MUTATION: prevent the transient unit from starting, fire an update, and confirm the operator-visible surface says the launch failed rather than showing the prior run's verdict.
Not in scope here
Two other errors recur at the same 60-second cadence and are filed separately — they are a different subsystem, but they share this issue's cadence in the log and will be seen alongside it.
Evidence
All from the live node: logs/slm-backend.log, logs/slm-backend-error.log, .deployed_commit, and the mtimes of self-update-ansible.log. No addresses or internal paths beyond component-relative ones are quoted, per #15516.
Activity
mrveiss commentedon Sep 2, 2026
Root cause: the self-update task is created without retaining a reference
autobot-slm-backend/api/code_sync.py:5217The task object is discarded. CPython's
asynciodocumentation states this directly: "Save a reference to the result of this function, to avoid a task disappearing mid-execution. The event loop only keeps weak references to tasks." An unreferenced task can be garbage-collected before it completes — or before it starts.This is not the file's usual pattern, which is what makes it a defect rather than a style choice. Of the eight
create_taskcalls incode_sync.py, five retain a reference:task = ..._run_component_resolve_jobtask = ..._sync_single_nodetask = ..._run_fleet_sync_jobtask = ..._run_fleet_sync_job_resume_task = ..._execute_resume_ansible_self_update_ansible_self_update_ansible_self_updateAll three unreferenced sites are the same function — the one that silently failed to execute.
Why this produces exactly the observed behaviour
add_done_callbackis attached anywhere in the file, so an exception inside_ansible_self_updatehas nowhere to surface. The caller has alreadyreturn True'd.stage.messagesays "service will restart" — the self-update restarts the very process holding the loop. A task that has not yet run dies with it, and the restart makes the disappearance look like a normal consequence of updating.Together: the caller reports success, the task may never run, and nothing anywhere records that it didn't.
Fix shape
Retain the reference for the task's lifetime — the pattern the same file already uses five times. A module-level set with an
add_done_callback(...discard)is the idiomatic form, and the callback is also where a launch failure should be logged rather than swallowed.That addresses the launch half. The stale-verdict half in the acceptance criteria above is separate and still stands: even with the task retained, a verdict reader that replays the previous run's outcome will still misreport.
Method note
Found by reading the call path rather than by re-running the update — the deployed node was left untouched. The distinction matters here because the bug is timing-dependent: a successful re-run would have proved nothing, and a failed one would still not have shown the cause.
fix(slm): retain self-update tasks, bind the verdict to its run, unbr…
github-actions commentedon Sep 2, 2026
PR #15525 (merged to
Dev_new_gui) references this issue with a close keyword.If this issue is fully resolved, close it manually. If work remains, no action is needed.
mrveiss commentedon Sep 2, 2026
PR #15525 merged as
2abc18de3b. Not closing — 2 of 5 criteria are unmet, and one new residual is worth recording.playbook_executor_ansible_self_updatecreate_tasksites discarded the task, and the loop holds only a weak reference. The caveat is that this is a third cause — neither hypothesis this AC named (inventory-directory writability, transient-unit launch failure) was tested or ruled out, and no host evidence was gathered.autobot_shared/async_compat.py:105_BACKGROUND_TASKS;:118.add(task);:119add_done_callback;:134.discard;:140logger.error(... exc_info=exc). All three firings converted atapi/code_sync.py:3460,:3533,:5215;git grep asyncio.create_task(_ansible_self_updatereturns nothing.services/self_update_log_reader.py:87_RUN_STARTED;:213_run_is_at_or_after(direction correct — the executor stamps after the fire,playbook_executor.py:745);:262returnsno self-update result for this run yet. Bound atcode_sync.py:5692.self_update_log_reader.py:35_RECAP_COUNTSmatchesfailed=/unreachable=and nothing else, so_describecan only say "finished with N failed task(s)". No permissions or location change to the0600ansible log in2abc18de3b. An operator still cannot see which task failed — which is the thing this AC exists for.The retention proof is sound — I checked the specific doubt
The implementing agent flagged that
test_the_task_survives_the_caller_dropping_its_referencestill passed under its own mutation, because a forcedgc.collect()does not deterministically reclaim in that window. That test is redundant, not load-bearing. The proof rests onasync_compat_test.py:127test_the_task_is_retained_while_pending_and_released_when_done, which asserts membership while pending and absence after completion — deterministic, and it kills both the.addand the callback's.discardunder mutation.:159asserts positively on caplog content, so a broken capture fails rather than passing vacuously.A residual this fix newly permits
test_an_unstamped_log_is_no_answer_to_a_bound_callerpins that an unstamped failing log returnsdegraded is Falseto a bound caller. The only bound consumer branches onif verdict.degraded:(code_sync.py:5693) — so "I cannot tell which run this is" is consumed as "not degraded", i.e. as success. Before this change the same log produced a FAILED stage.It is mitigated twice —
_await_self_update_completionrequires a completion newer than the firing, and_resume_verify_slm_stageverifies the deployed commit atcode_sync.py:5716before SUCCESS — so it is narrow. But it is a direction the fix opened, and the whole point of this issue is that an unknown state must not read as a good one. Worth an explicit decision rather than leaving it implicit.Remaining to close: AC4 (surface the failing task name — the reader has the recap in hand and could name it), and either satisfying AC5 on a node or restating it as something a diff can show.
mrveiss commentedon Sep 3, 2026
Your AC4 reproduced live, from the operator's side.
A self-update ran today and reported
self-update finished with 1 failed task(s). Which task failed is not obtainable, exactly as this issue predicts:/var/log/autobot/self-update-ansible.log0600, owned by the service account — unreadable as the operator userGET /api/code-sync/statusGET /api/healthstatus: healthy— carries no self-update verdict at allThat last row is the sharpest part. The health surface reported healthy immediately after a run that had a failed task. An operator looking at the obvious place sees nothing wrong.
So AC4 — "an operator cannot currently see which task failed" — is confirmed rather than inferred. It needs either a readable location or a surfaced summary, and neither exists today.
Two further host facts from the same run, both relevant to this issue's other criteria:
Both are the reason #15323 exists, and both are visible only by comparing timestamps by hand — which is itself an argument for the surfaced summary this AC asks for. A verdict that cannot say which release produced it is not much better than no verdict.
mrveiss commentedon Sep 12, 2026
Stale-issue sweep (never auto-closed — Dev_new_gui wasn't GitHub's default branch at merge time). Verified against current
origin/main; leaving open (matches this issue's own existing post-merge audit comment, independently re-confirmed here).MET: fire-and-forget scheduling via
autobot_shared/async_compat.pyfire_and_forget()at all 3_ansible_self_updatecall sites incode_sync.py(lines 3405, 3513, 5187); stale-verdict replay fixed viaself_update_log_reader.py:207-262(_run_started_at,_run_is_at_or_after), bound atcode_sync.py:861.NOT MET: surfacing which task failed (still only aggregate
failed=N/unreachable=Ncounts via the_RECAP_COUNTSregex) — reproduced live on a real node per the issue's own comment thread ("GET /api/health reported healthy immediately after a run that had a failed task"). Host-behavior AC is UNVERIFIABLE from this repo-only audit; the delivering PR (#15525) itself disclaims running any live update.mrveiss commentedon Oct 3, 2026
The log obstacle this issue names no longer exists — and a recurring failure pair is visible in it
One criterion states the blocker:
That is no longer true. On the deployed host today:
Mode 0644, world-readable, in a group-writable directory with an ACL. So the "readable location" half of that criterion is already satisfied — the file can be read without becoming the service account. Whether that was always so or changed since filing I have not established; what matters is that the obstacle as stated does not block the two
Establish…criteria any more, and they are now answerable by reading rather than by provisioning access.What the current log shows, and what it does not
Four
FAILED!entries today, from two separate runs 37 minutes apart (distinct pids), each producing the same pair:So for the criterion "whether it is the same failure that has been recurring" — yes, there is a recurring failure, and it is the same two missing marker files across both runs.
But it is not evidence about this issue's main symptom, and I will not present it as such. Both of today's runs executed — they went on to
TASK [Check SLM health],TASK [Clean up deploy archives]andTASK [Display update summary]. This issue is about a self-update that fires without executing. A run that reaches its final tasks is a different condition from one that never starts, so these four failures speak to the recurrence question and say nothing about the launch question.The
11:36and12:47firings this issue names are not in the current log — either rotated out or from a day not retained. Establishing why the 12:47 firing never reachedplaybook_executorstill needs either those rotated logs or a fresh reproduction, and it remains the open half.Cross-reference worth having
The two missing markers are not incidental to this repository's other open work:
.autobot_sync_deletions_commitis the subject of #16787 ("A way to tell, for a target already holding.autobot_sync_deletions_commit, whether the bootstrap that wrote it ran against a shallow or full-depth clone"), and the deployed-commit marker is in #16310's territory. A deploy path that fails twice in 37 minutes on two markers that two other open issues are about is plausibly one cause rather than three, though I have not established that and am recording it as a lead rather than a finding.What changes for this issue's state
It carries
blocked: observation-gated. On the evidence above, the observation that was thought to need an owner — reading the log — is a read any session can do. The remaining gated part is narrower: a reproduction of the non-executing firing, which is a mutation of the live system and genuinely an owner action. Worth re-scoping the label to that rather than to log access.