System updates drain borrowable roles; make reboot_strategy live declaration #16036

Parent: SLM method-adoption umbrella.

Method

Apply the update without dropping the workload. The reference stack achieves this in the kernel;
the portable half is the policy — a node being updated hands off its work first, so an update
never costs an in-flight run. In our field that matters more than it does for a web host: an
interrupted node is an interrupted agent workflow.

Already-exists audit

Both halves exist and are not connected:

• models/manifest.py:26 declares RebootStrategy and :138-142 ManifestSystemUpdates
  (policy, reboot_strategy) — so every role declares a reboot strategy.
• api/updates.py passes "auto_reboot": "false" (:163 and :949) to the update playbook. The
  declared reboot_strategy is therefore never read by the update path — it is dead
  declaration.
• services/blue_green.py already implements role borrowing, post_deploy_monitor_duration and
  health-gated auto_rollback — the drain-and-hand-off machinery exists, built for deployments.
• Nothing in api/updates.py references blue_green (grep: no hits).

Change

• Have the system-update path consume the manifest's reboot_strategy instead of a hardcoded
  auto_reboot: "false".
• For a node whose roles are borrowable, drain via blue_green role borrowing before applying
  updates that require a restart, and return the roles afterwards — the same path deployments
  already use.
• A role that cannot be borrowed says so, and its node's update is scheduled rather than silently
  applied under load.

Acceptance criteria

• api/updates.py reads ManifestSystemUpdates.reboot_strategy rather than hardcoding auto_reboot
• An update requiring a restart drains the node's borrowable roles first and restores them after
• A non-borrowable role produces an explicit scheduled-window outcome, not a silent in-place restart
• In-flight work is not dropped by a system update on a borrowable node — demonstrated by a test
• RebootStrategy values are all reachable; none remains dead declaration

Implementation Order

Wave: 2
Depends on: none

Concrete shape from oVirt (added after further research)

This is our own lineage — the SLM's Ansible deployment approach came from oVirt's downstream — so
the reference shape is worth following closely rather than reinventing.

Per-host cycle. Upgrading a host in Up status sets it to maintenance mode, which triggers
migration of its workloads to other hosts in the cluster; the packages are then installed through
the Ansible-driven host deploy; the host is brought back up. If workloads are still running when
the operator triggers it, they are warned before it proceeds.

Update availability is host state. GET /hosts/{id} returns
<updates_available>true</updates_available>, and the upgrade is an action on the same resource
(POST /hosts/{id}/upgrade). Our equivalent lives in side tables — models/database.py:142 Node
carries no updates-available field; UpdateInfo (:383) and UpdateJob (:411) hold it instead.
Worth aligning while this path is being touched, so "which nodes need attention" is a node-state
read.

Rolling fleet upgrade composes from the per-host cycle. Host by host — maintenance → upgrade →
activate — with per-host status visible so the operator can see how far the fleet has got.
grep -niE 'rolling|serial' autobot-slm-backend/api/updates.py autobot-slm-backend/services/deployment.py
→ no hits. We have no rolling fleet update; services/blue_green.py has the hand-off machinery
but it is wired only to deployments.

The control plane's own update is a separate mode. oVirt has a distinct global maintenance mode
for operations requiring the engine itself to stop — it does not pretend the control plane's update
is the same problem as a managed host's. Our equivalent is the self-update path
(services/local_admin_socket.py, api/code_sync.py), which should stay distinct from this work.

What does not transfer: oVirt's drain step is VM live migration. Handing off a role is not the
same problem — adopting the method means solving the hand-off for our workloads, not copying theirs.

Additional acceptance criteria

• The drain → update → return cycle is one operation on the node resource, not a sequence the caller assembles
• A rolling fleet update applies that cycle node by node with per-node status visible throughout
• Triggering an update on a node with in-flight work warns before proceeding
• Update availability is readable from node state, not only from the update side-tables
• The SLM's own self-update path remains distinct from managed-node updates