A run is split across two stores with no shared lifecycle: cleanup orphans state in both directions

Summary

A durable workflow run is spread across two independently-written stores with no
shared lifecycle:

	Store	Written by	Cleaned by
Graph state + checkpoint	`.nodus/graphs/{id}.json`, `{id}.checkpoint.json`	`orchestration/task_graph.py`	`nodus workflow cleanup` (`cli.py:1211`)
Run record	`.nodus/workflow_framework/runs/{id}.json`	`nodus_lang_workflow/store.py`	`max_terminal_runs`, opt-in, off by default

Resume needs both — runner.resume_workflow reads store.get_run(graph_id) and
load_graph_state(graph_id). Nothing keeps them in step, so either can be removed
while the other survives, in both directions.

Reproduction

Verified against src/ at 5.0.4. Delete only the run records — the cleanup
CLAUDE.md currently describes as safe ("rm -rf .nodus/workflow_framework/runs is
safe (test artifacts only)") — and a live waiting run becomes permanently
unresumable while its state stays on disk:

$ nodus run wait.nd          # a workflow that waits
graph: g_5ec714b4

$ ls .nodus/graphs/
g_5ec714b4.checkpoint.json
g_5ec714b4.json

$ rm -rf .nodus/workflow_framework/runs

$ nodus run resume.nd
{"ok": false, "error": "Workflow run 'g_5ec714b4' not found"}

The run is not "not found". Its graph state and its checkpoint are both sitting in
.nodus/graphs/, and they will sit there forever — nodus workflow cleanup only
removes snapshots in a terminal status, and this one is waiting.

The reverse orphan is the routine one: nodus workflow cleanup deletes graph state
and checkpoints (cli.py:1235-1236) and never touches the run records, so every
cleanup leaves the store's directory growing.

Why this is worth fixing rather than documenting

LocalWorkflowStore.list_runs() cost is linear in accumulated files — the measured
table in CLAUDE.md puts 10,000 files at 3.8 s per scan after the #380 fix. Both
directories accumulate, and only one of them has a retention story that a host can
turn on. #380 (bounding the store's cost) is open and is the natural home for
half of this; the other half is that the two halves of a run should share a
lifecycle at all.

Fix direction

Options, roughly in increasing order of ambition:

Make cleanup symmetric. nodus workflow cleanup removes the run record
alongside the graph state; store pruning removes the graph state alongside the
record. Cheapest, and removes the routine orphan.
Make the diagnosis honest. When get_run returns None but
load_graph_state finds state, say so — "run record missing, graph state present"
— instead of "not found". Same class of fix as Cross-process resume fails with 'Unknown graph' whenever the script uses the run_workflow result — and re-runs top-level side effects on every failed attempt #399, which replaced a misleading
"Unknown graph" with the real reason.
One owner. The run record and the graph state describe one run; a single store
interface writing both is the version of this with no failure mode. Largest change,
and probably the right long-term shape.

(1) and (2) are independently worth doing and do not block (3).

Affected versions

v5.0.4 (current when filed).

Provenance

C:\codev\Crewai research\CREWAI_ON_NODUS_IMPLEMENTATION_STUDY.md §6, "Two
Durability Stories — The New Finding", and §9 Q1, its top open question:

"Two well-built durability layers that have never been tested against each other is
a more interesting risk than one missing layer, because it is invisible until a
crash lands in the overlap."

The study frames the overlap as aindy-runtime's PostgreSQL FlowRun versus Nodus's
nodus_lang_workflow, and asks which is authoritative when a .nd workflow runs
under sys.v1.nodus.execute. That question belongs to the host and is not ours to
answer.

But the same shape exists entirely inside Nodus, between task_graph's graph
state and nodus_lang_workflow's run records, and that one is fully within our
control. The study's §3.1 and §3.2 enumerate both stores in detail and treat them as
one durability story; they are two, and this is what that costs.

Filed from a cross-check of C:\codev\Crewai research (4 documents) against 5.0.4. The Nodus-facing one is CREWAI_ON_NODUS_IMPLEMENTATION_STUDY.md, which verifies Nodus at 1a04d1d (v4.2.0-4) — a major behind — and ends with six open questions. This batch answers five of them.

Answered by these issues:

Q2 (typed join, firing and failure semantics) → A step failure kills independent fan-out branches, and a join has no failure semantics #475. The join exists and fires correctly; the failure semantics are the gap.
Q1 (two durability layers, which is authoritative) → A run is split across two stores with no shared lifecycle: cleanup orphans state in both directions #476, scoped to the half that is inside Nodus. The aindy-vs-Nodus half belongs to the host.
Q3 (a2a package-name collision) → nodus-a2a-wire declares the taken PyPI name nodus-a2a, so the A2A wire protocol is uninstallable #477.

Already answered by earlier batches in this series:

Q5 ("does the task-graph checkpoint survive a changed graph? MAF hashes topology and validates on restore") → Resume does not validate graph topology: a source edit yields a spurious 'Dependency cycle detected' #470. It does not — a source edit between wait and resume produces a spurious Dependency cycle detected. The study guessed right about where to look and marked it [Unknown]; it is reproduced there.
Q4 ("does delegate/agent_call/mcp_call survive being routed through capability gating") → CapabilityPolicy is never consulted for tool_call, syscall, agent_call or memory: a deny-everything policy denies nothing #473. Not as things stand: agent_call and tool_call are not in BUILTIN_CAPABILITIES, so a CapabilityPolicy is never consulted for them. The study's §7 sixth-test failure ("no capability gate on the delegated call") is a property of Nodus, not of the demo.

Q6 (cross-machine A2A unproven) needs a second host and is not a Nodus issue.

Two things the study got right that are worth keeping. Its nodus_lang_workflow/store.py line citations are all still exact a major later, and its §4.1 self-correction — withdrawing "workflow-as-data is a surviving gap" at the Nodus layer — holds up: the graph is atomically written, fsync'd and byte-stable. What it is missing is not durability but identity (#470).

#476: the two halves of a run share a lifecycle (#567)

Fixed in PR #567 (merged as f6e44e7). Options (1) and (2), both directions of each; option (3) recorded here as the long-term shape rather than taken.

Symmetric cleanup, direction one: nodus workflow cleanup deletes the run record with the graph state (reported as run_records_removed). One guard: a record that says the run is still live — the administrative mark_waiting shape, a waiting record over a completed snapshot — is not cleanup's to take without --force.
Symmetric cleanup, direction two: the store's opt-in max_terminal_runs cap prunes the graph state and checkpoint with the record it deletes. A host that caps records wants the run gone, not half of it.
delete_run(run_id) -> bool on the stores — a concrete no-op default on the WorkflowStore ABC (an abstract method would have broken every host store implementation), overridden by Local and SQLite. The SQLite one goes through a new _exec_rowcount helper because the SQLiteWorkflowStore leaks cursors: the store depends on CPython refcounting to commit #516 cursor-discipline source assertion caught the first cut calling conn.execute directly — that test doing exactly its job.
Honest diagnosis: this issue's repro (rm -rf .nodus/workflow_framework/runs, re-verified on main before fixing) now yields "has graph state under .nodus/graphs but no run record in the workflow store — the two halves of the run were cleaned independently" with category: "run_record_missing" and a recovery hint, instead of not found. An id with neither half still says not found, pinned by a control test.
CLAUDE.md corrected: its "rm -rf .nodus/workflow_framework/runs is safe (test artifacts only)" note was this issue's repro; it now says what the deletion actually does and points at workflow cleanup.

Option (3) — one owner — deliberately not taken here. It is the version with no failure mode and probably the right long-term shape, but it is a store-architecture change, and (1)+(2) remove the routine orphan and the misleading answer without it. If it is ever picked up, #380 (bounding the local store's scan cost) is the natural companion, and delete_run plus the symmetric-prune points added here are where the unified owner would slot in.

Not covered, for the record: a run record removed and graph state removed independently by something outside these code paths still ends at not found, which is then true. And the cross-store pairing relies on both stores having been created from the same working directory — true for every in-process path that writes both, noted in the prune comment.

Regression tests: tests/test_run_store_lifecycle.py (# closes: #476).

release: 5.4.0 -- a resume that tells the truth, an inspection that c…

Shipped in v5.4.0 — live on PyPI: https://pypi.org/project/nodus-lang/5.4.0/ · release notes

Validated pre-publish (Gate 10: 6/6 dependent suites, 35/35 release-claim probes, 9/9 adversarial boundaries against the wheel in a clean venv) and post-publish (Stage 5, including a run started on 5.3.0 and resumed on 5.4.0). Eval documents: docs/evals/v5.4.0/.

housekeeping: correct the prose the 5.4.0 cycle left behind, and file…

workflow: cleanup retires a run that never started (#734) (#750)

Run enumeration is the host's job, permanently (#761) (#773)

Navigation Menu

Repository navigation

A run is split across two stores with no shared lifecycle: cleanup orphans state in both directions #476

Summary

Reproduction

Why this is worth fixing rather than documenting

Fix direction

Affected versions

Provenance

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

A run is split across two stores with no shared lifecycle: cleanup orphans state in both directions #476

Description

Summary

Reproduction

Why this is worth fixing rather than documenting

Fix direction

Affected versions

Provenance

Activity

Masterplanner25 commented on Aug 18, 2026

Masterplanner25 commented on Aug 25, 2026

Masterplanner25 commented on Aug 25, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions