Skip to content

self-update fails at the backend systemd unit: the Play 1 include runs without become #14693

Description

@mrveiss

The self-update play fails one task on the SLM node, which stops it before the service restart. That is what leaves the update-all job wedged (#14683 is the wedge; this is why the play failed in the first place).

Evidence

From the self-update log on the test deployment:

TASK [[PLAY 1] SLM | Refresh the backend systemd unit (#14624)] ***********
TASK [slm_manager : SLM | Deploy backend systemd service] ****************
fatal: [<slm node>]: FAILED! => {"changed": false, "msg": "Destination /etc/systemd/system not writable"}

PLAY RECAP
<slm node>   : ok=7  changed=4  unreachable=0  failed=1
localhost    : ok=17 changed=3  unreachable=0  failed=0

Cause

A regression from #14668. roles/slm_manager/tasks/service_units.yml holds two tasks that both need root — a template into /etc/systemd/system and a systemd enable with daemon_reload — and neither declares become.

That worked for its original caller: slm_manager/tasks/main.yml includes it from a context that escalates. It does not work for the caller added in #14668, the Play 1 include in playbooks/update-all-nodes.yml:

- name: "[PLAY 1] SLM | Refresh the backend systemd unit (#14624)"
  ansible.builtin.include_role:
    name: slm_manager
    tasks_from: service_units.yml

No play in update-all-nodes.yml sets a play-level become — checked all five, every task declares its own. The task immediately above this include sets become: true individually. The include did not, so the template ran as the unprivileged user and could not write the unit.

Fix

Declare become: true on the two tasks inside service_units.yml rather than on the one call site. Both genuinely require root regardless of who includes them, and a task file whose correctness depends on the caller supplying escalation is the thing that broke here — fixing only the call site leaves the next caller to rediscover it.

Impact

The failing task aborts Play 1 before the SLM restarts, so:

Activity

  1. mrveiss commented on Aug 21, 2026

    @mrveiss
    OwnerAuthor

    Fixed and merged.

    PR: #14694 — fix(ansible): escalate in service_units.yml instead of relying on the caller (#14693)
    Merge commit: 0cd371047 on Dev_new_gui
    CI: 10/10 required checks green, 12/12 python shards green at merge.

    What landed: task-level become: true on the 2 tasks in
    roles/slm_manager/tasks/service_units.yml that write
    /etc/systemd/system/… and run systemd enable/daemon-reload, plus all 6
    handlers in roles/slm_manager/handlers/main.yml — 8/8 in scope, none
    broadened to play or block level.

    Guard: autobot-slm-backend/tests/test_service_units_self_sufficient_14693.py,
    which walks pre_tasks/post_tasks/block/rescue/always as well as
    tasks, so a task added without escalation fails rather than passing on a
    partial walk.

    Root cause confirmed statically in review: update-all-nodes.yml:301-305
    includes this task file via include_role … tasks_from: service_units.yml, and
    neither that include nor its play (update-all-nodes.yml:222-223) sets
    become. A bare become: on include_role is rejected by ansible — the
    playbook's own comment at :1580-1582 records this — so the call site could
    not have fixed it. That is why the escalation belongs in the task file. The
    failure was silent because failed_when: false on the restart handler (kept,
    for the legitimate "unit not yet deployed" case) masked the unprivileged
    restart.

    Not fixed here, filed instead:

    Closing on merge evidence. The host-behaviour half — that a self-update run now
    gets past the backend systemd unit — is only observable on a real run and is
    tracked by the deploy chain, not asserted here.

  2. mrveiss commented on Aug 22, 2026

    @mrveiss
    OwnerAuthor

    Recording the closure evidence, which the closing comment did not carry.

    Delivered in PR #14694, merged to Dev_new_gui. Verified against the base tree, not the PR:

    • roles/slm_manager/tasks/service_units.yml — 2 tasks declaring become: true
    • roles/slm_manager/handlers/main.yml — 6 handlers declaring become: true

    Host evidence that it actually fixed the failure

    Before, on the test deployment:

    TASK [slm_manager : SLM | Deploy backend systemd service]
    fatal: FAILED! => "Destination /etc/systemd/system not writable"
    PLAY RECAP  <slm node> : ok=7  changed=4  failed=1
    

    After:

    PLAY RECAP
    00-SLM-Manager : ok=235  changed=47  unreachable=0  failed=0  skipped=49
    localhost      : ok=21   changed=3   unreachable=0  failed=0  skipped=2
    

    Zero fatal: lines in the run. The decisive independent signal is the systemd unit itself: StartLimitIntervalUSec moved from the stale 10s to 2min, the template's value. That render task had been failing since #14668, so the unit refresh #14624 and #14668 existed to perform had never once run on the self-update path — this is what made those fixes real rather than nominal.

    Fleet effect: outdated_nodes 2 → 0, stale_components 5 → 0, self_update_incomplete true → false.

    Follow-ups filed from this work and still open: #14699 (the config-level become default has never applied), #14700 (same shape, latent, in the admin-ui tasks).

  3. mrveiss commented on Aug 23, 2026

    @mrveiss
    OwnerAuthor

    Merge commit for the record: PR #14694 squash-merged to Dev_new_gui as 0cd371047ab0e58351ac63cb61a88fa74c3b26ab.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions