Incident (live, 2026-07-23 12:01)
A deploy run stopped autobot-backend and never restarted it — dead from 12:01 to 12:48 (manual systemctl start). All /api/* → 502 → users could not log in; chat-delete failed with 'Network Error'. Manually recovered.
Root cause
roles/backend/tasks/main.yml: 'Stop service before env update (MVA-1633)' (line ~213, state: stopped) runs, then ~670 lines of risky work (apt install, pip install, code sync, alembic migrations, ownership fixes) execute BEFORE the service is started again (state: started at line ~883). If ANY task in that window fails — OR the ansible run is killed mid-flight (the #11492 self-restart-kills-the-run scenario, still live until #11492 is deployed under the running executor) — the backend is left STOPPED with no auto-restart. Compounding: the unit re-render (line 177) + daemon_reload (line 207) triggers systemd's 'Current command vanished from the unit file, execution won't be resumed', which stops the running backend even before the explicit stop.
Fix
Guarantee the backend ALWAYS ends started regardless of mid-deploy failure: wrap the stop→(heavy work)→start sequence in an ansible block with an always: clause that runs systemd: name=autobot-backend state=started daemon_reload=true (failed_when: false so the always never aborts). Idempotent on success; recovers the service on any block failure. (Process-KILL case is separately covered by #11492's detach once deployed; this handles task failures.) Consider also shrinking the stop→start window (stop only around the env-template deploy, not the whole role) as a follow-up. Apply the same guarantee to celery/celery-beat if they share the pattern.
Verify
An intentionally-failing task in the middle of the role must leave autobot-backend RUNNING (not dead). Part of umbrella #11820.
Incident (live, 2026-07-23 12:01)
A deploy run stopped
autobot-backendand never restarted it — dead from 12:01 to 12:48 (manualsystemctl start). All/api/*→ 502 → users could not log in; chat-delete failed with 'Network Error'. Manually recovered.Root cause
roles/backend/tasks/main.yml: 'Stop service before env update (MVA-1633)' (line ~213,state: stopped) runs, then ~670 lines of risky work (apt install, pip install, code sync, alembic migrations, ownership fixes) execute BEFORE the service is started again (state: startedat line ~883). If ANY task in that window fails — OR the ansible run is killed mid-flight (the #11492 self-restart-kills-the-run scenario, still live until #11492 is deployed under the running executor) — the backend is left STOPPED with no auto-restart. Compounding: the unit re-render (line 177) +daemon_reload(line 207) triggers systemd's 'Current command vanished from the unit file, execution won't be resumed', which stops the running backend even before the explicit stop.Fix
Guarantee the backend ALWAYS ends started regardless of mid-deploy failure: wrap the stop→(heavy work)→start sequence in an ansible
blockwith analways:clause that runssystemd: name=autobot-backend state=started daemon_reload=true(failed_when: falseso the always never aborts). Idempotent on success; recovers the service on any block failure. (Process-KILL case is separately covered by #11492's detach once deployed; this handles task failures.) Consider also shrinking the stop→start window (stop only around the env-template deploy, not the whole role) as a follow-up. Apply the same guarantee to celery/celery-beat if they share the pattern.Verify
An intentionally-failing task in the middle of the role must leave autobot-backend RUNNING (not dead). Part of umbrella #11820.