Skip to content

No cancellation anywhere: asynchronous primitives, not structured concurrency #395

Closed

Description

@Masterplanner25

Summary

Nodus has no way to stop work it started. A grep for cancel across
src/nodus/ returns one hit — print("Login cancelled.") in the CLI.

For a runtime whose stated purpose is coordinating long-running external calls,
that is a structural gap rather than a missing convenience.

What is absent

Mechanism Status
Cancellation of a spawned coroutine absent
Parent/child relationship between coroutines absent — Coroutine has no parent field, the scheduler has no tree
Scope-bound cleanup (nursery / task group) absent
Error propagation from child to parent absent — self._coroutine_errors collects errors, it does not propagate them
Cancellation of an in-flight graph run absent

_spawned_without_loop counts coroutines spawned with no run loop, which is a
diagnostic after the fact, not a lifecycle guarantee.

What this rules out

Structured concurrency's defining property is that a concurrent task cannot
outlive its lexical scope, and that failing or cancelling a scope cancels its
children. Nodus has none of the three mechanisms required, so what it provides
is asynchronous primitives, not structured concurrency — and docs should say
so rather than implying the stronger property.

Concretely unavailable today: cancel a workflow run in progress; abandon a
spawned coroutine blocked on a slow agent call; bound a fan-out so that the
first failure stops the siblings.

For comparison, Python asyncio has CancelledError and TaskGroup; Go has
context and select; Claude Code gives subagents a child AbortController
with propagation. A worker pool — a bounded set of independent workers with no
lifecycle relationship — is the fairest description of what Nodus has.

Why it is tractable

Coroutine already carries blocked_on and blocked_reason, which is the state
a cancellation mechanism needs to know how to unblock a waiter. The scheduler
already owns the ready deque and the timer heap. What is missing is a parent
link and a propagation rule, not a rewrite.

Design questions to settle before implementing:

Source

Confirmed while verifying an external architecture audit (§15) against the
implementation. The audit called this the gap with the clearest remediation path.

Affected versions

v4.1.1 and current main. Never present.

Activity

  1. added a commit that references this issue on Aug 30, 2026
  2. Masterplanner25 commented on Aug 30, 2026

    @Masterplanner25
    OwnerAuthor

    Design proposal merged — docs/design/v5/04-cancellation.md (PR #669). Issue stays open; nothing is built.

    Written in the shape of 01-goal-stopping-condition.md / 02-capability-policy.md: the record of what was decided and why, before implementing.

    Two of the four questions in this issue are already answered — by code

    This issue was filed 2026-08-15. #502 landed after it, and settled the two hardest questions in the list:

    # Question Answer Where
    1 Cooperative flag, or injected throw? Injected throw. VM.unwind_cancelled_coroutine (vm.py:2702) sets coroutine.cancelling, loads the coroutine's context, and drives handle_exception through the handler stack. #502
    2 Does a cancelled coroutine run its finally blocks? Yes — and handle_exception (vm.py:547) refuses to enter a catch while cancelling, so a step cannot swallow its own cancellation. Bounded by task_step_budget, so a looping finally cannot turn a cancellation into a hang. #502

    Answering either differently now would mean two unwind paths for one question, which is the defect shape CLAUDE.md documents twenty-three instances of. Whatever gets built must route through unwind_cancelled_coroutine.

    Q2 was arguably never a free choice: I-VM-06 says finally always executes, and defending that invariant is what #502 was for.

    So the gap is smaller and sharper than this issue describes

    The expensive half is built. What is missing is that nothing can pull the trigger — the scheduler's per-coroutine timeout check (scheduler.py:272) is the only caller. No cancel builtin, no host API, no CLI subcommand, no route. Cancellation happens to a coroutine, on exactly one condition, and nothing can request it.

    Also confirmed still absent: no parent field on Coroutine, builtin_spawn returns None so a program cannot name what it started, and _coroutine_errors still collects rather than propagates (there is nothing to propagate to).

    The remaining fork, and the recommendation: a handle, not a scope

    Question 4 answered: builtin only, no language surface.

    The scope's one distinctive win is first failure stops the siblings — and that exists already, at the altitude where Nodus actually expresses fan-out. A step failure stops the run dispatching new work (task_graph.py:1225), #475 decided the opt-out (allow_failure), and #480 made fan-out dynamic (step render each page in discover). A nursery would be a second way to express bounded concurrent fan-out, competing with the workflow DSL and carrying its own failure vocabulary.

    The residue is stated rather than hidden: raw spawn + channel stays a worker pool. Which is what this issue asked the docs to say.

    Question 3 answered: both halves

    "Does cancelling a graph run cancel step coroutines, or only stop dispatching?" — both. Stopping dispatch alone leaves an in-flight step running to completion, and a step blocked on a slow agent call is exactly what someone cancelling a run is trying to stop. #502's argument applies unchanged: dropping a step without unwinding loses its finally.

    Three things the doc adds that the issue does not have

    • The five blocked states a cancel must handle, one row each — ready deque, timer heap, channel waiter, host agent call, other blocking builtin. This issue points at blocked_on/blocked_reason as the state a fix builds on; that is true but partial, since every assignment is in builtins/coroutine.py and covers channels only. A cancel that handles two of five is the sibling-path shape.
    • BUG: agent handlers run unbounded — action agent / agent_call have no timeout, in a runtime whose premise is bounded execution #424's doctrine inherited, not reopened. Un-preemptable host work is abandoned, not stopped. Cancellation is delivered at yield points; a host agent handler keeps running and is recorded. Any design promising to stop a host call promises something the runtime cannot deliver.
    • The naming collision. TASK_STATUSES already contains cancelled, meaning never dispatched because a sibling failed (task_graph.py:1496). Recommendation: an actively cancelled step reports failed with a distinguishable error kind — task vocabulary untouched, test_status_vocabulary.py unchanged. A cancelled run genuinely needs an eighth run state, with _REHYDRATABLE_STATUSES (runner.py:180) and cleanup retention as the named consequences.

    Plus: cross-process cancellation can only mark the store and be observed at step boundaries, so it is cooperative and bounded by the current step's duration — a limit to document rather than paper over.

    Stale claims in this issue, corrected in the doc

    Every claim in §2 was read out of main at 5fab54e rather than taken from here, which is how these surfaced:

    The cheapest next step

    §9 is the docs-only half of this issue and needs no code. It is the part that says "what it provides is asynchronous primitives, not structured concurrency — and docs should say so": three edits, one of them the stale TECH_DEBT line above. Worth doing on its own, independently of whether the handle gets built.

  3. Masterplanner25 commented on Aug 30, 2026

    @Masterplanner25
    OwnerAuthor

    The documentation half is done — PR #670 (merged). Issue stays open for the code.

    This is the part of this issue quoted below, landed on its own:

    "What it provides is asynchronous primitives, not structured concurrency — and docs should say so rather than implying the stronger property."

    One document was doing more than implying it. FAILURE_AND_DEGRADATION_MODEL.md §9.1 said an unhandled exception in a spawned coroutine "is recorded on the coroutine object" and that "the spawning code receives the error when it resumes or waits on the coroutine." Neither is true, and the second describes a parent/child await that does not exist — spawn returns nil, so there is no handle to wait on, and resuming a finished coroutine is itself a runtime error.

    Measured, not read:

    Thrown error at sib.nd:1:30: first fails      <- stderr, immediately
    sibling ran anyway
    errors: ["first fails"]                        <- run_loop()'s return value
    === exit: 0 ===
    

    Six things actually happen: stderr trace; the error goes to the scheduler's list; the on_error hook fires if registered; run_loop() returns a list of error strings; siblings run to completion; and the run exits 0, with result["ok"] == true when embedded. The last is the one that surprises people and the fourth is the only mitigation — and LANGUAGE_SPEC.md had never documented run_loop()'s return value at all.

    What shipped

    Each of them says where the stronger property does live — the workflow DSL, with allow_failure and step … each.

    Two stale claims found while doing it

    Neither was on the design doc's list, and both were in the same neighbourhood:

    The pattern is worth naming: the errors were not in prose describing the missing feature — they were in prose describing features that had already changed. Nobody re-reads a document when the code beneath it moves. #412 phase 2 had just produced four instances of the same thing in BYTECODE_REFERENCE.md.

    What is left on this issue

    All of the code. The design record is docs/design/v5/04-cancellation.md, whose §9 now records this half as done; §§4–7 are the proposal (a cancel handle, cancel_run, an eighth run state) and remain unbuilt.

  4. Masterplanner25 commented on Aug 30, 2026

    @Masterplanner25
    OwnerAuthor

    The handle is decided, jointly with #157 — PR #674. Issue stays open for the code.

    docs/design/v5/06-task-handle.md decides the object cancel acts on. This issue and #157 wanted the same thing, so they were decided together rather than sequenced; §4 of that document is why that was not optional.

    §5.1's record handle is withdrawn, and it would have shipped wrong

    The proposal here was a record — { id: 3i, name: "worker", state: "running" }. Two problems, found by checking it before ratifying it:

    1. A record is a value, so state freezes at spawn time. t.state would read "created" for the life of the task while it ran and finished. A handle whose most-read field is permanently stale is worse than no handle.
    2. It was unnecessary, and it would have created a forgery surface the coroutine type does not have. Measured:
    let c = coroutine(fn() { sleep(5i); return 77i })
    print(coroutine_status(c))   // created
    spawn(c)
    print(coroutine_status(c))   // created
    run_loop()
    print(coroutine_status(c))   // finished
    

    c is already a live handle, and the idiom already forces a program to hold it (spawn takes a coroutine value, not a function literal). Meanwhile ensure_coroutine (vm/vm.py:979) is an identity check, so a coroutine cannot be counterfeited — coroutine_status({id: 1i, name: "worker", state: "running"}) is refused today with a type error, while the proposed record is constructible by any guest. The bare-id form §5.1 also offered is dropped for the same reason: an id is exactly the forgeable handle the coroutine type avoids being.

    Decided: cancel(c) takes the coroutine. spawn(c) returns c for convenience. No new value type. The near-miss is worth recording — the rejected design would have created the #411 lesson (a guarantee that trusts a name rather than a binding) through a convenience choice about handle shape.

    §6.2's table gains a sixth row, and that is why the sequencing mattered

    join (#157) parks a coroutine on another task — blocked_reason = "task_join". §6.2 enumerates five states and exists specifically to stop a cancel handling some and not others. Ship join after cancel and that table is incomplete on the day it lands: the recurring shape arriving by scheduling rather than by oversight.

    So blocked_reason becomes a named set that the cancel path and every blocking builtin drive off, and that ships with whichever verb lands first. It is the one part of the joint decision that is not deferrable.

    Cancelling a joiner does not cancel the joined — joining is a read, not adoption.

    §10.3 is now decided, and the argument changed

    "Does cancelling a parent run cancel child runs?" was flagged as the decision with the least evidence, held open pending a read of the mechanics. Answer unchanged (yes), argument replaced:

    The cleanup analogy was the weak argument; the nesting is the strong one. §7.1 already unwinds in-flight step coroutines, and the step running a child is one of them — so the child stops whether or not anything is decided here. What is left is marking the child's record, and if it is not marked the unwind leaves a non-terminal record whose parent is terminal and whose executor is gone. That is #476's half-state shape from a new direction.

    Two consequences: mark only non-terminal children (#486 re-entry means each resume produces a new child, so most are already completed and marking them all would rewrite finished history), and cross-process children inherit §7.2's cooperative bound.

    Unchanged

    Everything else in this document stands: the injected-throw unwind and the finally/catch rules (#502), run-level cancellation in both halves, the eighth run state, no task-status change, no scope construct, and §9's documentation half already shipped.

  5. Masterplanner25 commented on Aug 30, 2026

    @Masterplanner25
    OwnerAuthor

    Follow-up on D6: the door it opens was already open — PR #677

    Checking D6's justification before ratifying it turned up a counterexample, and the correction changes the argument rather than the outcome.

    D6 read: "this is the first error-propagation path in Nodus." It is not. resume(c) has always raised a coroutine's failure into the resumer, catchably:

    let c = coroutine(fn() { throw "task failed" })
    try { resume(c) } catch e { print("caught: \(e.message)") }
    // caught: task failed
    

    Verified against 5.7.1 dev source, including a failure after a yield, with the full err record (kind = "thrown", origin = "user"). It is documented nowhere — not LANGUAGE_SPEC.md, not FAILURE_AND_DEGRADATION_MODEL.md, not the guide — which is why it was missed.

    So the language already answers "how does a task's failure reach me" three ways, chosen by the verb:

    driving the work failure arrives as
    resume(c) raised into the caller, catchable
    spawn(c) + run_loop() collected — stderr, run_loop()'s return list, exit 0
    run_workflow(w) returned in the result map (failed: ["a"])

    The decision stands, on a better reason

    resume(c) and join(c) ask the same question — drive this task, give me its outcome. One raising while the other collected would be one question answered in two voices, which is the defect shape CLAUDE.md catalogues twenty-three instances of.

    And the framing inverts the risk. join drives the scheduler, so pairing it with run_loop's collect-and-continue is the intuitive move — taking it would have shipped resume raising and join collecting for one question, and someone would file that as a bug later.

    Why the wording matters rather than just the verdict: "this is the first propagation path, tread carefully" argues for keeping the door shut next time. "This matches resume" argues for consistency. A decision recorded with the wrong reason invites the wrong follow-on decision.

    What D6 now commits to, stated narrowly

    Propagation crosses a task boundary only where a program asked for an outcome, only to the code that asked, and exactly once. join is not the first such crossing, so it must not become a precedent for automatic ones. An unjoined failure is byte-identical to today.

    Two things added so they are not discovered during the build

    • D8 has a consequence worth naming. At top level join drives the scheduler, so a single join call can produce both behaviours at once — raised for the joined task, collected for any sibling that fails during that same drive. That is coherent, since nobody asked about the sibling, but it will read as an inconsistency unless the documentation says it outright.
    • Two build criteria in §5: that join and resume deliver a failure identically (D6 rests on that agreement, so it is asserted rather than assumed — a test that only checks join raises would pass on an implementation that had quietly diverged), and that the raise/collect coexistence is covered in one test.

    The undocumented path is written down

    FAILURE_AND_DEGRADATION_MODEL.md §9.1a now puts the three answers in one place, with the reason they differ: resume is a call, so its failure returns to the caller like any other; spawn is a hand-off, and there is nobody to return to. LANGUAGE_SPEC.md's coroutine section carries it too.

    That gap is the same family as the three doc-drift findings from earlier today — prose describing code that had moved, or in this case never describing it at all. This one was load-bearing for a decision being taken right now, which is the argument for writing behaviour down even when it seems obvious to whoever implemented it.

  6. Masterplanner25 commented on Aug 30, 2026

    @Masterplanner25
    OwnerAuthor

    The coroutine half is shipped in PR #681 (merged). Staying open for run-level cancellation.

    cancel(t) exists, and it is what this issue was missing — the unwind mechanism was already there from #502, with the scheduler's timeout check as its only caller.

    let t = spawn(coroutine(fn() { try { sleep(30i) } finally { print("cleanup ran") } }))
    spawn(coroutine(fn() { sleep(1i); cancel(t) }))
    run_loop()
    // cleanup ran
    

    What is done

    What is left, and why this stays open

    04-cancellation.md §7 — run-level cancellation. None of it is built:

    • NodusRuntime.cancel_run(graph_id) — the embedding case this issue names ("cancel a workflow run in progress")
    • nodus workflow cancel <graph_id>
    • an eighth run state, RUN_STATUS_CANCELLED, with _REHYDRATABLE_STATUSES and cleanup retention as the named consequences (§7.3)
    • the decision that cancelling a run does both halves — stop dispatching and unwind in-flight step coroutines (§7.1) — which is now implementable, since the coroutine verb it needs exists

    Also unbuilt and deliberately so: cross-process cancellation is cooperative and bounded by the current step's duration (§7.2), and that limit should be documented rather than papered over when it lands.

    The design record needs no revision for any of it; §7 already specifies the shape. What changed is that the piece it depended on is no longer hypothetical.

  7. Masterplanner25 commented on Aug 30, 2026

    @Masterplanner25
    OwnerAuthor

    Every decision in the design record is built — PR #683 merged. Closing.

    The design record is docs/design/v5/04-cancellation.md, with the handle decided jointly with #157 in 06-task-handle.md.

    What shipped, across four PRs

    #670 the documentation half — the worker-pool model stated plainly, and FAILURE_AND_DEGRADATION_MODEL.md §9.1 corrected where it described a runtime we do not have
    #681 cancel(t) and wait(t), and spawn(c) returning the handle
    #682 cancel_run(run_id), nodus workflow cancel <graph_id>, and RUN_STATUS_CANCELLED
    #683 §7.1 — cancelling a run stops dispatch and unwinds what is in flight
    let t = spawn(coroutine(fn() { try { sleep(30i) } finally { print("cleanup ran") } }))
    cancel(t)          // finally runs; catch is refused
    
    $ nodus workflow cancel g_9a6e48c5
    Cancelled g_9a6e48c5 (was running)
    A run owned by another process stops at its next step boundary.
    

    What the build changed about the design

    The verb is wait, not join. join collides with std:strings.join and std:path.join, and this repo's own example imports one by name. The underlying cause — a builtin silently shadowing an explicitly imported name — is #680, and it makes any new builtin a potential breaking change.

    D6's justification was wrong, and the correction mattered more than the verdict. It claimed wait raising would be "the first error-propagation path in Nodus". resume(c) has always raised a task's failure into the resumer, undocumented until #677. The decision stands on consistency rather than on a fresh commitment — and that framing inverts the risk, because pairing wait with run_loop's collect-and-continue was the intuitive move and would have shipped two answers to one question.

    The run-status vocabulary was named three times — REHYDRATABLE_RUN_STATUSES in store.py, _REHYDRATABLE_STATUSES in runner.py, and the members again in _KNOWN_RUN_STATUSES. Adding an eighth status is exactly when that costs something. nodus_gate --shapes did not flag it: its species-B detector looks for a strict subset, and these were equal.

    D4 earned its place immediately. The named blocked-reason set was not bookkeeping: the test for cancelling a task parked on a channel went red and found a real defect — the waiter was removed but the channel stayed registered, so the loop reported Deadlock: 0 coroutine(s) blocked on recv(), a deadlock of nobody.

    What this deliberately does not do

    Nodus is still a worker pool, not structured concurrency. No parent links, no scope, no automatic propagation — cancelling a waiter does not cancel the task it waited on. Propagation crosses a task boundary only where a program asked, only to the asker, exactly once. 04 §4.1 rejects a nursery construct, and 06 §D7 rejects a language surface for either verb; both stand.

    Cross-process cancellation is cooperative, bounded by the duration of the step currently running. That is documented rather than papered over.

    Left over, filed separately

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions