Repository navigation
No cancellation anywhere: asynchronous primitives, not structured concurrency #395
Description
Summary
Nodus has no way to stop work it started. A grep for cancel across
src/nodus/ returns one hit — print("Login cancelled.") in the CLI.
For a runtime whose stated purpose is coordinating long-running external calls,
that is a structural gap rather than a missing convenience.
What is absent
| Mechanism | Status |
|---|---|
| Cancellation of a spawned coroutine | absent |
| Parent/child relationship between coroutines | absent — Coroutine has no parent field, the scheduler has no tree |
| Scope-bound cleanup (nursery / task group) | absent |
| Error propagation from child to parent | absent — self._coroutine_errors collects errors, it does not propagate them |
| Cancellation of an in-flight graph run | absent |
_spawned_without_loop counts coroutines spawned with no run loop, which is a
diagnostic after the fact, not a lifecycle guarantee.
What this rules out
Structured concurrency's defining property is that a concurrent task cannot
outlive its lexical scope, and that failing or cancelling a scope cancels its
children. Nodus has none of the three mechanisms required, so what it provides
is asynchronous primitives, not structured concurrency — and docs should say
so rather than implying the stronger property.
Concretely unavailable today: cancel a workflow run in progress; abandon a
spawned coroutine blocked on a slow agent call; bound a fan-out so that the
first failure stops the siblings.
For comparison, Python asyncio has CancelledError and TaskGroup; Go has
context and select; Claude Code gives subagents a child AbortController
with propagation. A worker pool — a bounded set of independent workers with no
lifecycle relationship — is the fairest description of what Nodus has.
Why it is tractable
Coroutine already carries blocked_on and blocked_reason, which is the state
a cancellation mechanism needs to know how to unblock a waiter. The scheduler
already owns the ready deque and the timer heap. What is missing is a parent
link and a propagation rule, not a rewrite.
Design questions to settle before implementing:
- Is cancellation cooperative (a flag checked at yield points) or does it unwind
via an injectedthrow? The latter composes withfinally, which the VM
already honours correctly — see finally is skipped when catch rethrows — and its regression test cannot fail #361/A return deferred to finally survives an exception from that finally and is applied by an unrelated finally later #370/Deferred return is VM-wide, not per-coroutine: two coroutines suspended in a finally consume each other's return #371. - Does a cancelled coroutine run its
finallyblocks? (It should.) - Does cancelling a graph run cancel step coroutines, or only stop dispatching
new steps? - Does this get a language surface (a scope construct) or only a builtin?
Source
Confirmed while verifying an external architecture audit (§15) against the
implementation. The audit called this the gap with the clearest remediation path.
Affected versions
v4.1.1 and current main. Never present.
Activity
docs(audit): verify external architecture audit 01; file 5 findings (#…
design: cancellation proposal for #395, and the two questions already…
Masterplanner25 commentedon Aug 30, 2026
Design proposal merged —
docs/design/v5/04-cancellation.md(PR #669). Issue stays open; nothing is built.Written in the shape of
01-goal-stopping-condition.md/02-capability-policy.md: the record of what was decided and why, before implementing.Two of the four questions in this issue are already answered — by code
This issue was filed 2026-08-15. #502 landed after it, and settled the two hardest questions in the list:
throw?VM.unwind_cancelled_coroutine(vm.py:2702) setscoroutine.cancelling, loads the coroutine's context, and driveshandle_exceptionthrough the handler stack.finallyblocks?handle_exception(vm.py:547) refuses to enter acatchwhile cancelling, so a step cannot swallow its own cancellation. Bounded bytask_step_budget, so a loopingfinallycannot turn a cancellation into a hang.Answering either differently now would mean two unwind paths for one question, which is the defect shape
CLAUDE.mddocuments twenty-three instances of. Whatever gets built must route throughunwind_cancelled_coroutine.Q2 was arguably never a free choice: I-VM-06 says
finallyalways executes, and defending that invariant is what #502 was for.So the gap is smaller and sharper than this issue describes
The expensive half is built. What is missing is that nothing can pull the trigger — the scheduler's per-coroutine timeout check (
scheduler.py:272) is the only caller. Nocancelbuiltin, no host API, no CLI subcommand, no route. Cancellation happens to a coroutine, on exactly one condition, and nothing can request it.Also confirmed still absent: no parent field on
Coroutine,builtin_spawnreturnsNoneso a program cannot name what it started, and_coroutine_errorsstill collects rather than propagates (there is nothing to propagate to).The remaining fork, and the recommendation: a handle, not a scope
Question 4 answered: builtin only, no language surface.
The scope's one distinctive win is first failure stops the siblings — and that exists already, at the altitude where Nodus actually expresses fan-out. A step failure stops the run dispatching new work (
task_graph.py:1225), #475 decided the opt-out (allow_failure), and #480 made fan-out dynamic (step render each page in discover). A nursery would be a second way to express bounded concurrent fan-out, competing with the workflow DSL and carrying its own failure vocabulary.The residue is stated rather than hidden: raw
spawn+channelstays a worker pool. Which is what this issue asked the docs to say.Question 3 answered: both halves
"Does cancelling a graph run cancel step coroutines, or only stop dispatching?" — both. Stopping dispatch alone leaves an in-flight step running to completion, and a step blocked on a slow agent call is exactly what someone cancelling a run is trying to stop. #502's argument applies unchanged: dropping a step without unwinding loses its
finally.Three things the doc adds that the issue does not have
blocked_on/blocked_reasonas the state a fix builds on; that is true but partial, since every assignment is inbuiltins/coroutine.pyand covers channels only. A cancel that handles two of five is the sibling-path shape.action agent/agent_callhave no timeout, in a runtime whose premise is bounded execution #424's doctrine inherited, not reopened. Un-preemptable host work is abandoned, not stopped. Cancellation is delivered at yield points; a host agent handler keeps running and is recorded. Any design promising to stop a host call promises something the runtime cannot deliver.TASK_STATUSESalready containscancelled, meaning never dispatched because a sibling failed (task_graph.py:1496). Recommendation: an actively cancelled step reportsfailedwith a distinguishable error kind — task vocabulary untouched,test_status_vocabulary.pyunchanged. A cancelled run genuinely needs an eighth run state, with_REHYDRATABLE_STATUSES(runner.py:180) andcleanupretention as the named consequences.Plus: cross-process cancellation can only mark the store and be observed at step boundaries, so it is cooperative and bounded by the current step's duration — a limit to document rather than paper over.
Stale claims in this issue, corrected in the doc
Every claim in §2 was read out of
mainat5fab54erather than taken from here, which is how these surfaced:cancelacrosssrc/nodus/returns one hit" — now returns 23, across six files.TECH_DEBT.md:588repeats that line verbatim, so the debt register understates what is built.task_graph.py:1497cites "the open half of A step failure kills independent fan-out branches, and a join has no failure semantics #475"; A step failure kills independent fan-out branches, and a join has no failure semantics #475 is closed.The cheapest next step
§9 is the docs-only half of this issue and needs no code. It is the part that says "what it provides is asynchronous primitives, not structured concurrency — and docs should say so": three edits, one of them the stale
TECH_DEBTline above. Worth doing on its own, independently of whether the handle gets built.Masterplanner25 commentedon Aug 30, 2026
The documentation half is done — PR #670 (merged). Issue stays open for the code.
This is the part of this issue quoted below, landed on its own:
One document was doing more than implying it.
FAILURE_AND_DEGRADATION_MODEL.md §9.1said an unhandled exception in a spawned coroutine "is recorded on the coroutine object" and that "the spawning code receives the error when it resumes or waits on the coroutine." Neither is true, and the second describes a parent/child await that does not exist —spawnreturnsnil, so there is no handle to wait on, and resuming afinishedcoroutine is itself a runtime error.Measured, not read:
Six things actually happen: stderr trace; the error goes to the scheduler's list; the
on_errorhook fires if registered;run_loop()returns a list of error strings; siblings run to completion; and the run exits0, withresult["ok"] == truewhen embedded. The last is the one that surprises people and the fourth is the only mitigation — andLANGUAGE_SPEC.mdhad never documentedrun_loop()'s return value at all.What shipped
FAILURE_AND_DEGRADATION_MODEL.md— new §9.0 stating the worker-pool model normatively (a spawned coroutine outlives its scope; a failure does not stop its siblings; there is nothing to wait on and nothing to cancel); §9.1 rewritten to the six measured behaviours; new §9.4 documenting the timeout unwind as the one cancellation path, with BUG: agent handlers run unbounded —action agent/agent_callhave no timeout, in a runtime whose premise is bounded execution #424's un-preemptable-host-handler caveat; a duplicated### 9.2heading removed.LANGUAGE_SPEC.md— the same statement normatively, plusrun_loop()'s return value.standard-library.md— a runnable two-coroutine example showing a sibling surviving a failure, with four takeaways and aTESTEDnote recording the exit-0 behaviour as designed rather than filing it.TECH_DEBT.md— the stale "onecancelhit insrc/nodus/, a CLI print" line, copied from this issue's premise (written before An abandoned step does not run its finally: I-VM-06 does not hold under step timeout or sibling failure #502), replaced with what is built and what is missing.task_graph.py— the "open half of A step failure kills independent fan-out branches, and a join has no failure semantics #475" comment. A step failure kills independent fan-out branches, and a join has no failure semantics #475 closed in 5.4.0 andallow_failureis the answer, verified ona → {b,c},d after cwithbfailing:discancelledby default,completedwhenbis tolerated.Each of them says where the stronger property does live — the workflow DSL, with
allow_failureandstep … each.Two stale claims found while doing it
Neither was on the design doc's list, and both were in the same neighbourhood:
LANGUAGE_SPEC.mdstill calledstd:async'sworker_poolandpipeline"currently non-functional", citing ASYNC-MOD-003: std:async worker_pool/pipeline never run their tasks (parallel/series fixed in 4.1.1) #339 — three releases after ASYNC-MOD-003: std:async worker_pool/pipeline never run their tasks (parallel/series fixed in 4.1.1) #339 was fixed (2026-08-15).docs/guide/standard-library.mdhad it correctly in the past tense, so the two documents contradicted each other and the spec was the wrong one. Both verified working by running them.The pattern is worth naming: the errors were not in prose describing the missing feature — they were in prose describing features that had already changed. Nobody re-reads a document when the code beneath it moves. #412 phase 2 had just produced four instances of the same thing in
BYTECODE_REFERENCE.md.What is left on this issue
All of the code. The design record is
docs/design/v5/04-cancellation.md, whose §9 now records this half as done; §§4–7 are the proposal (a cancel handle,cancel_run, an eighth run state) and remain unbuilt.Masterplanner25 commentedon Aug 30, 2026
The handle is decided, jointly with #157 — PR #674. Issue stays open for the code.
docs/design/v5/06-task-handle.mddecides the objectcancelacts on. This issue and #157 wanted the same thing, so they were decided together rather than sequenced; §4 of that document is why that was not optional.§5.1's record handle is withdrawn, and it would have shipped wrong
The proposal here was a record —
{ id: 3i, name: "worker", state: "running" }. Two problems, found by checking it before ratifying it:statefreezes at spawn time.t.statewould read"created"for the life of the task while it ran and finished. A handle whose most-read field is permanently stale is worse than no handle.cis already a live handle, and the idiom already forces a program to hold it (spawntakes a coroutine value, not a function literal). Meanwhileensure_coroutine(vm/vm.py:979) is an identity check, so a coroutine cannot be counterfeited —coroutine_status({id: 1i, name: "worker", state: "running"})is refused today with a type error, while the proposed record is constructible by any guest. The bare-id form §5.1 also offered is dropped for the same reason: an id is exactly the forgeable handle the coroutine type avoids being.Decided:
cancel(c)takes the coroutine.spawn(c)returnscfor convenience. No new value type. The near-miss is worth recording — the rejected design would have created the #411 lesson (a guarantee that trusts a name rather than a binding) through a convenience choice about handle shape.§6.2's table gains a sixth row, and that is why the sequencing mattered
join(#157) parks a coroutine on another task —blocked_reason = "task_join". §6.2 enumerates five states and exists specifically to stop a cancel handling some and not others. Shipjoinaftercanceland that table is incomplete on the day it lands: the recurring shape arriving by scheduling rather than by oversight.So
blocked_reasonbecomes a named set that the cancel path and every blocking builtin drive off, and that ships with whichever verb lands first. It is the one part of the joint decision that is not deferrable.Cancelling a joiner does not cancel the joined — joining is a read, not adoption.
§10.3 is now decided, and the argument changed
"Does cancelling a parent run cancel child runs?" was flagged as the decision with the least evidence, held open pending a read of the mechanics. Answer unchanged (yes), argument replaced:
run_workflowinside a parent's step body — synchronously nested in the parent's step coroutine, not a peer run scheduled elsewhere;parent_graph_id,child_graph_ids— A nested run inside a workflow step is an orphan record, and every resume leaks another one #501), andnodus workflow cleanupalready walks parent→child in a fixpoint loop (cli/cli.py:831-851).The cleanup analogy was the weak argument; the nesting is the strong one. §7.1 already unwinds in-flight step coroutines, and the step running a child is one of them — so the child stops whether or not anything is decided here. What is left is marking the child's record, and if it is not marked the unwind leaves a non-terminal record whose parent is terminal and whose executor is gone. That is #476's half-state shape from a new direction.
Two consequences: mark only non-terminal children (#486 re-entry means each resume produces a new child, so most are already
completedand marking them all would rewrite finished history), and cross-process children inherit §7.2's cooperative bound.Unchanged
Everything else in this document stands: the injected-throw unwind and the
finally/catchrules (#502), run-level cancellation in both halves, the eighth run state, no task-status change, no scope construct, and §9's documentation half already shipped.design: one task handle, two verbs -- the joint decision for #157 and #…
Masterplanner25 commentedon Aug 30, 2026
Follow-up on D6: the door it opens was already open — PR #677
Checking D6's justification before ratifying it turned up a counterexample, and the correction changes the argument rather than the outcome.
D6 read: "this is the first error-propagation path in Nodus." It is not.
resume(c)has always raised a coroutine's failure into the resumer, catchably:Verified against 5.7.1 dev source, including a failure after a yield, with the full err record (
kind = "thrown",origin = "user"). It is documented nowhere — notLANGUAGE_SPEC.md, notFAILURE_AND_DEGRADATION_MODEL.md, not the guide — which is why it was missed.So the language already answers "how does a task's failure reach me" three ways, chosen by the verb:
resume(c)spawn(c)+run_loop()run_loop()'s return list, exit 0run_workflow(w)failed: ["a"])The decision stands, on a better reason
resume(c)andjoin(c)ask the same question — drive this task, give me its outcome. One raising while the other collected would be one question answered in two voices, which is the defect shapeCLAUDE.mdcatalogues twenty-three instances of.And the framing inverts the risk.
joindrives the scheduler, so pairing it withrun_loop's collect-and-continue is the intuitive move — taking it would have shippedresumeraising andjoincollecting for one question, and someone would file that as a bug later.Why the wording matters rather than just the verdict: "this is the first propagation path, tread carefully" argues for keeping the door shut next time. "This matches
resume" argues for consistency. A decision recorded with the wrong reason invites the wrong follow-on decision.What D6 now commits to, stated narrowly
Propagation crosses a task boundary only where a program asked for an outcome, only to the code that asked, and exactly once.
joinis not the first such crossing, so it must not become a precedent for automatic ones. An unjoined failure is byte-identical to today.Two things added so they are not discovered during the build
joindrives the scheduler, so a singlejoincall can produce both behaviours at once — raised for the joined task, collected for any sibling that fails during that same drive. That is coherent, since nobody asked about the sibling, but it will read as an inconsistency unless the documentation says it outright.joinandresumedeliver a failure identically (D6 rests on that agreement, so it is asserted rather than assumed — a test that only checksjoinraises would pass on an implementation that had quietly diverged), and that the raise/collect coexistence is covered in one test.The undocumented path is written down
FAILURE_AND_DEGRADATION_MODEL.md§9.1a now puts the three answers in one place, with the reason they differ:resumeis a call, so its failure returns to the caller like any other;spawnis a hand-off, and there is nobody to return to.LANGUAGE_SPEC.md's coroutine section carries it too.That gap is the same family as the three doc-drift findings from earlier today — prose describing code that had moved, or in this case never describing it at all. This one was load-bearing for a decision being taken right now, which is the argument for writing behaviour down even when it seems obvious to whoever implemented it.
feat: cancel(t) and wait(t) -- a task can be stopped, and its outcome…
feat: cancel(t) and wait(t) — a task can be stopped, and its outcome …
Masterplanner25 commentedon Aug 30, 2026
The coroutine half is shipped in PR #681 (merged). Staying open for run-level cancellation.
cancel(t)exists, and it is what this issue was missing — the unwind mechanism was already there from #502, with the scheduler's timeout check as its only caller.What is done
cancel(t)reuses An abandoned step does not run its finally: I-VM-06 does not hold under step timeout or sibling failure #502's unwind unchanged rather than adding a second one.finallyruns,catchis refused so a task cannot swallow its own cancellation, bounded by the same step budget.wait(t)after a cancel raisesTask cancelled. A cancelled task did not produce a value and must not appear to.unpark, driven by the namedBLOCKED_REASONSset (D4). That set earned its place immediately: the test for cancelling a task parked on a channel went red and found a real defect — the waiter was removed but the channel stayed registered, so the loop reportedDeadlock: 0 coroutine(s) blocked on recv(), a deadlock of nobody.action agent/agent_callhave no timeout, in a runtime whose premise is bounded execution #424's doctrine is inherited, not reopened. Un-preemptable host work is abandoned, not stopped; cancellation is delivered at yield points.waitreplacedjoin— see An explicitly imported name is silently shadowed by a same-named builtin #680 for why, and06-task-handle.md§2a.What is left, and why this stays open
04-cancellation.md§7 — run-level cancellation. None of it is built:NodusRuntime.cancel_run(graph_id)— the embedding case this issue names ("cancel a workflow run in progress")nodus workflow cancel <graph_id>RUN_STATUS_CANCELLED, with_REHYDRATABLE_STATUSESandcleanupretention as the named consequences (§7.3)Also unbuilt and deliberately so: cross-process cancellation is cooperative and bounded by the current step's duration (§7.2), and that limit should be documented rather than papered over when it lands.
The design record needs no revision for any of it; §7 already specifies the shape. What changed is that the piece it depended on is no longer hypothetical.
feat: a workflow run can be cancelled (#395 section 7) (#682)
feat: cancelling a run stops dispatch and unwinds what is in flight (#…
Masterplanner25 commentedon Aug 30, 2026
Every decision in the design record is built — PR #683 merged. Closing.
The design record is
docs/design/v5/04-cancellation.md, with the handle decided jointly with #157 in06-task-handle.md.What shipped, across four PRs
FAILURE_AND_DEGRADATION_MODEL.md §9.1corrected where it described a runtime we do not havecancel(t)andwait(t), andspawn(c)returning the handlecancel_run(run_id),nodus workflow cancel <graph_id>, andRUN_STATUS_CANCELLEDWhat the build changed about the design
The verb is
wait, notjoin.joincollides withstd:strings.joinandstd:path.join, and this repo's own example imports one by name. The underlying cause — a builtin silently shadowing an explicitly imported name — is #680, and it makes any new builtin a potential breaking change.D6's justification was wrong, and the correction mattered more than the verdict. It claimed
waitraising would be "the first error-propagation path in Nodus".resume(c)has always raised a task's failure into the resumer, undocumented until #677. The decision stands on consistency rather than on a fresh commitment — and that framing inverts the risk, because pairingwaitwithrun_loop's collect-and-continue was the intuitive move and would have shipped two answers to one question.The run-status vocabulary was named three times —
REHYDRATABLE_RUN_STATUSESinstore.py,_REHYDRATABLE_STATUSESinrunner.py, and the members again in_KNOWN_RUN_STATUSES. Adding an eighth status is exactly when that costs something.nodus_gate --shapesdid not flag it: its species-B detector looks for a strict subset, and these were equal.D4 earned its place immediately. The named blocked-reason set was not bookkeeping: the test for cancelling a task parked on a channel went red and found a real defect — the waiter was removed but the channel stayed registered, so the loop reported
Deadlock: 0 coroutine(s) blocked on recv(), a deadlock of nobody.What this deliberately does not do
Nodus is still a worker pool, not structured concurrency. No parent links, no scope, no automatic propagation — cancelling a waiter does not cancel the task it waited on. Propagation crosses a task boundary only where a program asked, only to the asker, exactly once.
04 §4.1rejects a nursery construct, and06 §D7rejects a language surface for either verb; both stand.Cross-process cancellation is cooperative, bounded by the duration of the step currently running. That is documented rather than papered over.
Left over, filed separately
--shapesdetector's blind spot for equal duplicate enumerations, recorded in04 §7.fix: a named import of a builtin name is refused, not silently ignored (
release: 5.8.0 -- work already in motion can be stopped, and a bad re…
feat(runtime): spawn() accepts a function; fix(parser): match is usab…