the self-update stage stops reporting progress once it detaches, so liveness is unobservable #14740

Summary

The slm_self_update stage of update-all fires its playbook fire-and-forget
(asyncio.create_task, detach=True so it survives the service restart) and
returns immediately. No further progress is stamped for the rest of that
stage's real duration.

So the one stage most likely to run long — a full-machine ansible pass over
backend, frontend, celery, scheduler, ai-stack, npu-worker and browser-worker —
is precisely the stage the staleness model cannot observe.

Surfaced in review of #14704.

Why it matters

_job_is_stale judges liveness by now - max(last_progress_at, created_at)
against SLM_UPDATE_ALL_STALE_SECONDS (default 1800s). Once the self-update
stage detaches, nothing advances that timestamp — so after 30 minutes a
genuinely running self-update and a dead one look identical.

That is also exactly the shape of the wedge reported in #14703 ("sat at
slm_self_update: running for hours"). The same signal cannot distinguish the
failure it was built to detect from the success it must not interrupt.

What #14704 does and does not solve

#14704 makes a live persisted resume plan an override: while a plan exists,
the job is never judged stale, never retired, and the C3-b guard still blocks a
second run. That closes the dangerous case — a running self-update can no
longer be reaped and double-orchestrated.

It is a proxy, not a heartbeat. It answers "did a self-update begin and not yet
finish", not "is it still making progress". A self-update that dies while a
plan is persisted is still indistinguishable from one that is working, and now
it also cannot be retired — the plan override converts that case from
"wrongly reaped" into "wedged until the plan is cleared".

That is the right trade (wedging is recoverable, double-orchestrating live
nodes is not), but it is a trade, and this issue is the way out of it.

Fix

Give the detached playbook a way to report liveness that survives the service
restart, and have staleness read it:

1. Have the detached side write a periodic heartbeat somewhere the API can read
   after restart — the same store the resume plan uses is the obvious candidate,
   since it already survives the restart by design.
2. Fold that timestamp into _job_is_stale alongside last_progress_at.
3. Then a dead self-update becomes reapable again without reintroducing the
   risk of reaping a live one, and the resume-plan override can relax from
   "never stale" to "stale only if the heartbeat also stopped".

Interval should be an env-var-backed module constant, well under
SLM_UPDATE_ALL_STALE_SECONDS, not hardcoded.

Acceptance criteria

• A running self-update advances an observable liveness signal after the
  stage detaches, and across the service restart
• _job_is_stale consults it
• A self-update that dies mid-flight is judged stale within roughly the
  configured window, proven by a test that stops the heartbeat
• A self-update that is merely slow is never judged stale, proven by a test
  that keeps it beating past the window
• The interval is env-var-backed, not hardcoded