Observed on a live self-update today. A fleet node was reported FAILED by the orchestrator while its ansible run had logged no failure at all.
The reported failure is not a failure
The run's last lines are a task succeeding:
TASK [slm_agent : Node Exporter | Detect Mellanox InfiniBand directory]
[WARNING]: Skipped '/sys/class/infiniband' path ... is not a directory
ok: [<node>]
That [WARNING] is ansible's standard message for an absent glob path and the task reports ok. It is the last line before the process died, which makes it look like the cause. It is not.
Measured on the run's own process id in the deployment's ansible log:
|
|
| lines written by the fleet-node run |
234 |
fatal: lines |
0 |
PLAY RECAP lines |
0 |
failed= lines |
0 |
An ansible task failure always writes fatal: and a recap. This run wrote neither. It was killed mid-play.
What killed it
The self-update's own play restarted the SLM backend 523 ms after the fleet run's last line:
10:43:27,872 pid A ok: [<node>] <- last line from the fleet run, then nothing
10:43:28,395 pid B [PLAY 1] SLM | Restart backend service
The backend's own log shows it coming back ~10:43:36.
This is #11492's defect in a second spawner. That issue (closed 2026-07-22) established the mechanism exactly: the SLM backend unit is KillMode=control-group, KillSignal=15, so systemctl restart SIGTERMs every process in the cgroup, including ansible children spawned without session detachment.
Its fix was applied to services/playbook_executor.py. It was never applied to api/code_sync.py, which is where the fleet stage spawns:
| Spawner |
create_subprocess_exec |
start_new_session / setsid / systemd-run |
services/playbook_executor.py |
1 |
16 |
api/code_sync.py |
15 |
0 |
So the self-update's own run now survives the restart — #11492's fix works — while the fleet stage's children, spawned from the other module, still die exactly as before.
Why it is hard to see
The orchestrator reports FAILED because the child vanished, and the operator reads the last log line, which is a benign warning about InfiniBand. The real signal is an absence — no recap, no fatal — and absences do not appear in a scroll-back. Two runs are also interleaved in one log under different process ids, so the restart that killed the run looks like unrelated progress on another node.
Proposed fix
- Detach every ansible spawn in
api/code_sync.py the way playbook_executor.py already does, rather than adding a 16th private spelling — ideally by routing all 15 call sites through the existing executor so there is one spawner, not two.
- Fail loudly on a child that exits without a
PLAY RECAP: "killed before completing" is a different verdict from "a task failed", and only one of them means the node is broken.
- Order the self-node restart after the fleet stage has drained, so the window cannot open at all.
Acceptance criteria
Stated limits
I did not reproduce this under a controlled restart; the causal chain is assembled from the log (abrupt stop, zero failure markers), the 523 ms gap to the restart task, the unit's KillMode=control-group, and the zero-detachment count in the spawning module. Each link is measured, but the sequence is inferred rather than observed end to end. I also have not checked whether the other 14 spawns in that module are all on paths reachable during a fleet sync — the count is of the module, not of the fleet path specifically.
Related: #11492 (same mechanism, other spawner), #13916 (one concept, two copies, and the fix reaches only one).
Observed on a live self-update today. A fleet node was reported
FAILEDby the orchestrator while its ansible run had logged no failure at all.The reported failure is not a failure
The run's last lines are a task succeeding:
That
[WARNING]is ansible's standard message for an absent glob path and the task reportsok. It is the last line before the process died, which makes it look like the cause. It is not.Measured on the run's own process id in the deployment's ansible log:
fatal:linesPLAY RECAPlinesfailed=linesAn ansible task failure always writes
fatal:and a recap. This run wrote neither. It was killed mid-play.What killed it
The self-update's own play restarted the SLM backend 523 ms after the fleet run's last line:
The backend's own log shows it coming back ~10:43:36.
This is #11492's defect in a second spawner. That issue (closed 2026-07-22) established the mechanism exactly: the SLM backend unit is
KillMode=control-group,KillSignal=15, sosystemctl restartSIGTERMs every process in the cgroup, including ansible children spawned without session detachment.Its fix was applied to
services/playbook_executor.py. It was never applied toapi/code_sync.py, which is where the fleet stage spawns:create_subprocess_execstart_new_session/setsid/systemd-runservices/playbook_executor.pyapi/code_sync.pySo the self-update's own run now survives the restart — #11492's fix works — while the fleet stage's children, spawned from the other module, still die exactly as before.
Why it is hard to see
The orchestrator reports
FAILEDbecause the child vanished, and the operator reads the last log line, which is a benign warning about InfiniBand. The real signal is an absence — no recap, no fatal — and absences do not appear in a scroll-back. Two runs are also interleaved in one log under different process ids, so the restart that killed the run looks like unrelated progress on another node.Proposed fix
api/code_sync.pythe wayplaybook_executor.pyalready does, rather than adding a 16th private spelling — ideally by routing all 15 call sites through the existing executor so there is one spawner, not two.PLAY RECAP: "killed before completing" is a different verdict from "a task failed", and only one of them means the node is broken.Acceptance criteria
api/code_sync.pyruns inside the backend's cgroup; a backend restart leaves in-flight fleet runs alive.create_subprocess_execappears in either module without detachment — deploy: self-update ansible run is killed by the Play-1 backend restart (KillMode=control-group) — Play 1 tail + Play 2/3 never run #11492 fixed one spawner and nothing noticed the other for ~2.5 months.Stated limits
I did not reproduce this under a controlled restart; the causal chain is assembled from the log (abrupt stop, zero failure markers), the 523 ms gap to the restart task, the unit's
KillMode=control-group, and the zero-detachment count in the spawning module. Each link is measured, but the sequence is inferred rather than observed end to end. I also have not checked whether the other 14 spawns in that module are all on paths reachable during a fleet sync — the count is of the module, not of the fleet path specifically.Related: #11492 (same mechanism, other spawner), #13916 (one concept, two copies, and the fix reaches only one).