update-all Play 3 'Verify' cannot fail, so a run reports complete with the SLM down

The final verification stage of update-all-nodes.yml cannot fail the run.

- name: Check SLM health
  uri:
    url: "https://{{ slm_host }}/slm/api/health"
    validate_certs: false
  register: slm_health
  ignore_errors: true

ignore_errors: true, and the result is consumed only by the summary text:

- "  SLM:     {{ slm_health.json.status | default(Check manually) }}"

So a run whose SLM never came back prints Fleet Update Complete with SLM: Check manually and exits successfully. The stage named "Verify" performs no verification — it reports.

Play 1 does gate properly (Wait for API to be ready, until status == 200, 12 retries at 10s), so this is not the only check. But Play 3 runs after the fleet nodes have been updated, which is exactly when a regression introduced mid-run would show up, and it is the last thing that looks at the system before declaring success.

Fix: let the health check fail the play, or if it must stay advisory, rename it so it does not read as a gate. An ignore_errors check named "Verify" is worse than no check, because it produces a green run that looks verified.

Related: #14702 (no rollback when the restarted service fails health).

fix(update): deliver before installing, verify in a way that can fail (…

PR #14810 (merged to Dev_new_gui) references this issue with a close keyword.

fix(update): deliver before installing, verify in a way that can fail (#14809, #14701, #14700, #14804)

If this issue is fully resolved, close it manually. If work remains, no action is needed.

Delivered in PR #14810, squash-merged to Dev_new_gui as 84c51d71a4a9ef2c720ca4afd7e1440e329ad780.

Verified against the base tree, not the PR:

ai-stack installs from a requirements file it refreshes afterwards, so a bad one can never be fixed #14809 — Deploy requirements-ai.txt now precedes the filter and the install (task index asserted, not eyeballed), and the file is no longer copied a second time in the application-source loop
update-all Play 3 'Verify' cannot fail, so a run reports complete with the SLM down #14701 — Play 3's health check gates (ignore_errors gone) and retries rather than probing once
slm_manager: the admin-ui systemd tasks rely on caller escalation, same shape as #14693 #14700 — both slm-admin-ui systemd tasks declare become: true
a self-update finishing at a logrotate boundary is reported FAILED after the full timeout #14804 — _rotated_completion_time() present, the old boolean probe gone

CI on the merged head: 52 success, 7 skipped, zero failures; all 10 required checks green.

Each guard was mutation-checked with the target asserted present first, so a missing target fails loudly rather than reporting a false pass:

move delivery back after install -> CAUGHT
restore ignore_errors            -> CAUGHT
drop admin-ui become             -> CAUGHT
never return the rotated mtime   -> CAUGHT

Still needed for #14809 to take effect on a live host

The code fix is landed but the deployed node still carries the stale requirements-ai.txt (81 lines, md5 f246c6af, spacy>=3.8.14 at line 32) against the repo's 98-line 07e3d1b3. The next update run is the one that delivers it, because the copy now happens before the install that used to abort first. Worth confirming on that run that stale_components finally drops autobot-ai-stack, rather than assuming the ordering fix was sufficient.

Navigation Menu

Repository navigation

update-all Play 3 'Verify' cannot fail, so a run reports complete with the SLM down #14701

Still needed for #14809 to take effect on a live host

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

update-all Play 3 'Verify' cannot fail, so a run reports complete with the SLM down #14701

Description

Activity

github-actions commented on Aug 22, 2026

mrveiss commented on Aug 22, 2026

Still needed for #14809 to take effect on a live host

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions