spawn terminate skips the pre-stop flush, so task status reports a terminated task as 'running' forever and its log is unrecoverable #802

spawn terminate destroys the instance without running the pre-stop hook that TTL and cost-limit kills use (added in 0.116.0, spawn#643 fixing #632/#642). Two consequences: spawn task status reports the task as running indefinitely, and command.log / completion.json are never flushed, so the run is undiagnosable afterwards.

Measured on spawn 0.126.1, us-west-2.

Reproduction

A task hung (my bug — an MPI collective on a rank-divergent path), so I terminated it rather than let it ride to TTL:

$ spawn terminate cookbook-fenicsx-r1 --region us-west-2
The instance is shutting down and will be destroyed.

Afterwards, the same CLI gives two different answers about the same task:

$ spawn task status cookbook-fenicsx-r1 --region us-west-2
Task cookbook-fenicsx-r1: running (no completion record yet)

$ spawn task diagnose cookbook-fenicsx-r1 --region us-west-2
no instance found with name: cookbook-fenicsx-r1 — cookbook-fenicsx-r1 (i-009c95ba26c26881b)
  is terminated: User initiated (2026-10-11 03:23:47 GMT)

diagnose is right (it queries EC2); status reads the S3 completion record, which was never written. It will say running forever.

And nothing was flushed

A task that completed normally minutes later, same spec, same bucket:

$ aws s3 ls s3://spawn-results-.../tasks/cookbook-fenicsx-r2/
2026-10-10 20:28:18          1 .exitcode
2026-10-10 20:28:17      12545 command.log
2026-10-10 20:28:17        327 completion.json

The terminated one:

$ aws s3 ls s3://spawn-results-.../tasks/cookbook-fenicsx-r1/
(empty — exit 1)

So the 22 minutes of output from the hung run was lost. I only have it because I fetched it live with spawn connect ... -- tail /var/log/spawn-command.log before terminating — which is the thing a user is least likely to think of, since the natural reaction to a hang is to kill it.

Why this is worth closing

This is the same failure mode #632/#642 described, and #643 fixed it for the lifecycle paths. A user-initiated stop is arguably the case where the log matters most: you terminate precisely because something is wrong and you want to know what. At the moment, killing a hung task destroys the evidence for why it hung.

It also leaves a permanently wrong record. Anything that reads task status to decide whether work is outstanding — a script, a dashboard, a person — sees running for a machine that no longer exists.

Two possible fixes, either would do

1. Run the existing pre-stop hook on spawn terminate before the EC2 call, so command.log and completion.json are flushed the same way a TTL kill flushes them. This seems closest to the fix(task): flush a completion record and the log when a lifecycle limit kills a task #643 design, and would also give a completion.json with a user_terminated state.
2. Have spawn task status fall back to EC2 when no completion record exists, and report terminated (no completion record) rather than running. Cheaper, and fixes the wrong-record half even if the flush is hard to guarantee on a manual path.

(1) is the better outcome; (2) alone would still be a clear improvement over a permanent running.

Smaller related note

spawn logs <name> on the terminated task prints only the generic trailer about "failures that happened before the workload started", with no indication that the task existed, ran for 22 minutes, and was terminated. Given diagnose can see all of that from EC2, logs saying "this instance was terminated by the user at T; no log was flushed" would save a reader the round trip.

Happy to test a fix — this is easy for me to reproduce on demand.