An abandoned step does not run its finally: I-VM-06 does not hold under step timeout or sibling failure #502

Summary

docs/runtime/EXECUTION_INVARIANTS.md I-VM-06 states, without qualification:

I-VM-06: finally blocks always execute

If a try/finally block is entered and control leaves (via return, throw, normal
exit, or exception), the finally block executes exactly once.

A workflow step that is abandoned — killed by its own timeout_ms, or dropped because
a sibling step failed — leaves its try block without running finally. The step had
entered the block. Cleanup does not happen, in precisely the circumstances cleanup exists
for.

Two independent triggers verified, so this is a property of the abandonment path rather
than of either trigger.

Reproduction

Verified against src/ at 5.0.4.

Trigger 1 — a sibling step fails

workflow cleanup {
    step a { print("  a ran"); return 1i }
    step slow after a {
        print("  slow ACQUIRED resource")
        try {
            sleep(60i)
            print("  slow FINISHED")
        } catch e {
            print("  slow CAUGHT")
        } finally {
            print("  slow RELEASED resource")
        }
        return 2i
    }
    step boom after a { sleep(10i); print("  boom throwing"); throw "boom" }
}

  a ran
  slow ACQUIRED resource
  boom throwing
steps: {"a": 1}

slow entered the try, was mid-sleep when boom threw, and was dropped.
slow RELEASED resource never printed.

Control — the identical workflow with the failing sibling replaced by a succeeding one:

    step ok after a { sleep(10i); print("  ok done"); return 9i }

  a ran
  slow ACQUIRED resource
  ok done
  slow FINISHED
  slow RELEASED resource
steps: {"a": 1, "ok": 9, "slow": 2}

So the finally runs normally and is skipped because a sibling failed.

Trigger 2 — the step's own timeout_ms

workflow tmo {
    step slow with { timeout_ms: 100 } {
        print("  slow ACQUIRED")
        try {
            sleep(2000i)
            print("  slow FINISHED")
        } catch e {
            print("  slow CAUGHT")
        } finally {
            print("  slow RELEASED")
        }
        return 1i
    }
}

  slow ACQUIRED
failed: ["slow"]

No sibling involved. The step is reported failed, and finally still does not run.

Why this matters more than a missing print

finally is the only cleanup construct the language has — there is no defer, no
context manager, and no workflow-level exit hook (STEP_OPTION_KEYS in
src/nodus/orchestration/workflow_lowering.py:61-69 is seven entries, none of them about
cleanup or failure). So a step that acquires anything — a lock, a file handle, an open
transaction, a spawned subprocess, a channel with a waiting peer — has exactly one place
to release it, and that place does not run when the step is killed.

The timeout case is the sharper one, because timeout_ms is a documented step option
whose entire purpose is to bound a step that might hang. A user who bounds a
possibly-hanging step and wraps its resource in try/finally has done everything the
documentation asks, and still leaks.

On the invariant's wording

The enumeration in I-VM-06 is "return, throw, normal exit, or exception", and
abandonment is not in that list — so this can be read as outside the invariant's scope
rather than as a violation of it. That reading is available, but it does not help a
reader: the heading says always, the body says exactly once, and nothing anywhere
mentions that a coroutine can be dropped with pending finally blocks.

So the fix is one of:

1. Honour it — unwind an abandoned step coroutine through its pending finally-gates
   before dropping it, the way return/throw already do. This is the behaviour the
   invariant advertises.
2. Qualify it — state in I-VM-06 that abandonment (timeout, sibling failure, scheduler
   shutdown) does not unwind, and say so in the guide beside timeout_ms, so the leak is
   at least predictable.

(1) is the right answer if it is affordable, because (2) leaves the language with no
cleanup mechanism that survives a timeout. If (1) is taken, it needs a bound of its own —
a finally that blocks forever must not turn a timeout into a hang — which is an argument
for unwinding with a deadline rather than not unwinding.

What to check alongside it

Per CLAUDE.md § The recurring bug shape, the question is what else reaches the same
abandonment path. Two triggers are confirmed above; these are unverified and worth
checking together:

• a coroutine suspended when run_loop() returns
• a coroutine live when the process-level --time-limit / max_steps deadline fires
• worker_timeout_ms, the sibling option to timeout_ms
• a step abandoned by the agent_timeout_ms path (BUG: agent handlers run unbounded — action agent / agent_call have no timeout, in a runtime whose premise is bounded execution #424 established that a host agent
  handler is not on the instruction stream, so its abandonment is already special)

A regression test should assert on the unwind path itself rather than on one trigger's
output — a behaviour-only test passes on whichever trigger gets fixed first.

Related

• A step failure kills independent fan-out branches, and a join has no failure semantics #475 — a step failure kills independent branches. Trigger 1 is a consequence of it,
  and this is the part that makes it a resource-safety issue rather than a scheduling
  one. Argo's failFast: true default explicitly "waits until all DAG nodes are
  completed before failing the DAG itself", so even its fail-fast path does not abandon
  in-flight work.
• BUG: agent handlers run unbounded — action agent / agent_call have no timeout, in a runtime whose premise is bounded execution #424 — every timeout bounds the instruction stream. Same family: what a deadline
  does to work already in progress.
• Step results are binary: no partial-success envelope, so a retry re-runs work that already succeeded #468 — no partial-success envelope. An abandoned step is a partial outcome with no
  way to report what it had already done.

Affected versions

v5.0.4 (current). Not known to be a regression; I-VM-06 has been stated in its current
unqualified form since the invariants doc was written.

Provenance

Surfaced while cross-checking Argo Workflows
(C:\codev\Orchestration DSL research\argo-workflows) against 5.0.4 for #475.

Argo's answer to the resource problem is a first-class exit hook rather than reliance on
in-task cleanup — OnExit exists at three levels (workflow, template, DAG task),
each documented as "invoked at the end of the …, irrespective of the success, failure, or
error of the primary …" (pkg/apis/workflow/v1alpha1/workflow_types.go:385-388,
:1794-1797, :3657-3660).

Checking what Nodus has in that position is what produced this: it has finally and
nothing else, and finally turns out not to hold under abandonment.