Resume does not validate graph topology: a source edit yields a spurious 'Dependency cycle detected'

Summary

Nothing records what a run's graph looked like. _persist_graph_state
(src/nodus/orchestration/task_graph.py:166-236) writes per-task status keyed by
task_id plus step_to_task â€” no edges, no topology, no hash. On resume the
graph is rebuilt from source and the persisted state is re-applied by step name,
with nothing checking that the rebuilt shape is the shape the run was planned for.

If the source changed in between, the resume fails with a diagnosis that is
false: it reports a dependency cycle in code that has none.

Reproduction

Verified against src/ at 5.0.4.

Start a run that waits, from wf.nd:

workflow waiter {
    step a { checkpoint "started"; return workflow_wait("go") }
    step b after a { print("resumed"); return 2i }
}

graph: g_fbd6a406
status: waiting

Edit wf.nd â€” insert a step between a and b:

workflow waiter {
    step a { checkpoint "started"; return workflow_wait("go") }
    step z after a { print("z RAN"); return 9i }
    step b after z { print("resumed"); return 2i }
}

Resume the run from step 1:

Dependency cycle detected: z -> z

There is no cycle. The edited file runs clean when executed fresh â€” a -> z -> b
is a chain. The self-edge is manufactured by the resume path mapping the persisted
step_to_task onto a graph whose shape has moved.

Expected behavior

Refuse the resume with an accurate message naming the real cause â€” that the run was
planned against a different version of the workflow â€” instead of reporting a
structural defect in correct source.

Fix direction

Hash the topology at run start, store it on the run, compare on rebuild:

Compute over the step names and the dependency edges (step_to_task plus each
node's dependencies), not over the source text â€” a comment or a body edit must
not quarantine a run whose shape is unchanged.
Store it in the graph metadata next to workflow_source_path
(orchestration/workflow_lowering.py:385-393).
Compare in _rebuild_workflow_graph (src/nodus/vm/vm.py:1512+) and raise
WorkflowRebuildError with the real reason on mismatch â€” the machinery for a
good message is already there from Cross-process resume fails with 'Unknown graph' whenever the script uses the run_workflow result — and re-runs top-level side effects on every failed attempt #399.

Reference design, worth copying rather than rediscovering: Microsoft Agent
Framework stores graph_signature_hash on every checkpoint (_checkpoint.py:72),
computes it from the discovered step names (_functional.py:698), and refuses on
mismatch (_functional.py:921-926) with:

"Checkpoint '...' was created by a different version of workflow '...' and is not
compatible with the current version. The workflow's step structure may have changed
since this checkpoint was saved."

That is the message this should produce.

Scope note

The exposure is narrower than it looks and should be stated honestly: a CLI run
stores its source in the run metadata, so its rebuild uses the stored source and the
topology cannot drift. The drift window is runs rebuilt from a path on disk â€”
which today means embedded run_file runs. It widens the moment the sibling issue
about recording source for embedded runs is decided, and a stored-source design that
falls back to the path for any reason re-opens it. The hash is the check that does
not depend on getting that decision right.

Affected versions

v5.0.4 (current when filed).

Provenance

C:\codev\Autogen research\MAF_ON_AINDY_RUNTIME_REFERENCE_STUDY.md §3.3
("Graph-signature-validated resume"), cross-checked against 5.0.4. It raises exactly
this failure mode against aindy-runtime, notes the runtime "has no analogue", and
calls the fix "cheap, and worth having regardless… it converts a
silent-wrong-execution failure mode into a loud one."

The document did not check Nodus — it reads Nodus at v4.2.0 and says so. The gap is
here too, and the failure is worse than silent: it reports a dependency cycle in
correct source.

Filed together from one cross-check of the four C:\codev\Autogen research documents against 5.0.4 (MICROSOFT_AGENT_FRAMEWORK_AINDY_LENS_AUDIT.md, MAF_ON_AINDY_RUNTIME_REFERENCE_STUDY.md, ACCURACY_CHECK_vs_aindy-runtime_2.4.0.md, Microsoft Agent Framework Audit.md), with MAF's own source read at C:\codev\Autogen research\agent-framework.

The set:

Embedded run_source workflows can never be resumed: NodusRuntime records no source, so the rebuild has nothing to rebuild from #469 — embedded run_source runs are unresumable (reproduced; the only outright defect)
Resume does not validate graph topology: a source edit yields a spurious 'Dependency cycle detected' #470 — no graph-signature validation on resume (reproduced; spurious cycle error)
Workflow edges are unconditional: no predicate-selected branch in the graph, only inside step bodies #471 — no conditional edges in the workflow graph (design question)
workflow_wait declares no payload schema, so resume_workflow delivers an unvalidated payload #472 — untyped resume payload

Context these came from. The corpus names one question as unresolved across five documents — whether Nodus's task-graph model is ahead of, behind, or orthogonal to MAF's _edge.py — and says Nodus was never re-audited (it reads Nodus at v4.2.0, and the pin it checks is nodus-lang==5.0.1; current is 5.0.4). Answering it against 5.0.4 is what produced these four.

The answer: orthogonal. Three of MAF's four edge shapes are already present and working in Nodus — one-to-one, fan-out (concurrent, on the in-VM scheduler) and fan-in (step d after b, c), verified by running a diamond: levels: [["a"], ["b", "c"], ["d"]]. So the corpus's "fan-out/join is the most-derived missing primitive" is an aindy-runtime finding that does not transfer here. What Nodus lacks is the fourth shape (#471) and, more importantly, graph identity: MAF's topology is serializable data it can hash and validate (#470), Nodus's is derived from code it re-executes. Nodus buys expressive topology from syntax and pays by having no artifact to validate; MAF buys a validatable artifact and pays with _missing_callable on every predicate. Neither is ahead.

Full detail in each issue.

Fourth system, and the clearest statement of the target — Temporal.

C:\codev\Temporal research\TEMPORAL_AINDY_LENS_AUDIT.md, first "Absorb into Nodus" item:

Workflow-as-data discipline — Temporal's "history is the program record" reinforces Nodus's task-graph-as-file (.nodus/graphs/<id>.json); harden checkpoint/resume so a graph reload is authoritative, and surface the graph as the canonical program artifact.

"A graph reload is authoritative" is a better one-line statement of this issue's goal than the title. Today a reload is the opposite of authoritative: the persisted state carries no topology, the graph is rebuilt by re-executing source that may have changed, and the mismatch surfaces as Dependency cycle detected: z -> z in code that has no cycle.

Temporal is worth citing here specifically because it is the system that took this furthest: the history is the record, and replay is validated against it rather than re-derived from source. Nodus is at the other end — the source is the record and the persisted state is only status — with MAF's graph_signature_hash sitting in between as the cheap middle option this issue proposes.

Four independent corpora have now converged on it: MAF (§3.3, where it was reproduced), CrewAI (§9 Q5, flagged [Unknown] and pointed at the right place), Temporal (here), and the OpenClaw audit indirectly, since a workflow you cannot reload authoritatively is not the "canonical program artifact" the positioning claims.

Also filed alongside: #494 — resume re-executes with no determinism boundary, from this audit's second absorb item. The two are the same seam from different sides: this issue is "the graph may have changed", #494 is "the world may have changed." Neither is detected today.

How Nix actually solves this — and why Nodus cannot copy it, which is the useful part.

Studied from C:\codev\Orchestration DSL research\nix (shallow clone). The whole mechanism is fifteen lines, src/libstore/derivations.cc:91-115:

static auto infoForDerivation(const StoreDirConfig & store, const Derivation & drv)
{
    StorePathSet references;
    for (const auto & input : drv.inputs)
        references.insert(input.getBaseStorePath());
    auto suffix   = std::string(drv.name) + drvExtension;
    auto contents = unparse(drv, store);                    // canonical serialization
    auto hash     = hashString(HashAlgorithm::SHA256, contents);
    auto ca       = TextInfo{.hash = hash, .references = references};
    return std::tuple{suffix, contents, references, store.makeFixedOutputPathFromCA(suffix, ca)};
}

The identity is the hash of the node's own serialized form, and that hash becomes its name. Inputs are recorded as references, so identity is transitively closed over the whole upstream graph.

That is a stronger property than what this issue proposes. Nodus wants to detect a changed graph on reload. Nix makes reloading a changed graph unreachable: a changed graph has a different name, so there is nothing to mismatch. Temporal's "a graph reload is authoritative" is the goal; this is the construction that achieves it.

What goes into the hash — and the tension it exposes

src/libstore/derivation/aterm.cc serializes everything: outputs, the input tree, platform, builder, args, and env/structuredAttrs. Change the build command by one character and it is a different derivation.

That is the opposite of what this issue proposes, and the disagreement is worth resolving explicitly rather than by default. This issue currently says:

Compute over the step names and the dependency edges, not over the source text — a comment or a body edit must not quarantine a run whose shape is unchanged.

Nix would say the opposite: a body edit must invalidate, because the result is no longer the result you asked for.

Nix can afford that because its units are pure and re-runnable. If a derivation changed, you rebuild it; nothing is lost. Nodus steps have effects — a step that already sent an email, charged a card or pushed a commit cannot be re-derived from scratch. So the cost of over-invalidating is asymmetric between the two systems, and copying Nix's commitment wholesale would make long-running workflows unresumable after any edit.

The design this suggests, which is neither extreme

Nix ships two addressing modes — input-addressed (hash of the inputs) and content-addressed (hash of the outputs) — which is a precedent for not forcing one answer. The analogue here:

Hash	Covers	On mismatch
topology	step names + `after` edges	refuse the resume — the run was planned against a different shape (this issue's proposal)
body	each step's lowered body	warn and proceed, recording it on the run — the shape holds, the logic moved

That separates the two failure modes this issue conflates. A renamed or re-wired step is unrecoverable and should be refused. An edited step body is recoverable and usually intentional, but the operator should be able to see it happened — which today they cannot, because nothing records anything.

It also gives #494 (determinism) somewhere to attach: a step whose body hash changed between checkpoint and resume is exactly the case where re-execution may observe different behaviour, and it is now detectable rather than silent.

One transferable detail

Nix hashes a canonical serialization, not a live structure — unparse produces a deterministic text form and that is what gets hashed. Nodus's _persist_graph_state already writes with sort_keys=True, separators=(",", ":") (orchestration/task_graph.py), so the byte-stable serialization discipline this needs is already in the tree and already used for the state file. The topology hash can reuse it rather than inventing an ordering.

A likely mechanism for the spurious cycle, and the reference for a topology check

The mechanism

_rebuild_workflow_graph (src/nodus/vm/vm.py:1523-1533) has two sources for the code it
rebuilds from, and they behave oppositely:

source_code = metadata.get("workflow_source_code")
source_path = metadata.get("workflow_source_path")
if not isinstance(source_code, str):
    if not isinstance(source_path, str) or not source_path or not os.path.exists(source_path):
        raise WorkflowRebuildError(...)
    with open(source_path, "r", encoding="utf-8") as f:
        source_code = f.read()

Stored source present → the run replays the program as it was at first run. A source
edit is ignored entirely, silently. Verified in Resume silently replays the source stored at first run, so edits are ignored — unless nothing was stored, when it reads the current file #497: adding a step and resuming runs the
old graph and reports success; a file defining no workflow at all resumes and executes
the original steps.
Stored source absent, path present → it reads the current file from disk at resume
time, and recompiles it against a task set planned for the old shape.

This report's symptom — a source edit producing Dependency cycle detected in code that
has none — is what the second branch would produce. Worth confirming which branch the
original repro took when this is worked, because if it is that one, then #497 and this
issue share a fix: the fork itself is the defect, and the false cycle is a downstream
consequence of recompiling an edited file against stale task bookkeeping.

vm.source_code is set only by src/nodus/tooling/runner.py and
src/nodus/dap/server.py, so which branch a run takes is decided by its entry point.
(When neither is set — every embedded NodusRuntime run — resume fails outright: #496.)

The reference for what a topology check would compare

This issue's core point stands: nothing records what the graph looked like. From
libs/checkpoint/langgraph/checkpoint/base/__init__.py, LangGraph's Checkpoint records
per-node progress rather than program text:

class Checkpoint(TypedDict):
    channel_values: dict[str, Any]
    channel_versions: ChannelVersions
    versions_seen: dict[str, ChannelVersions]
    """Map from node ID to map from channel name to version seen.
    Used to determine which nodes to execute next."""
    updated_channels: list[str] | None

Two things are transferable here:

versions_seen is the resume decision, stored. "Which nodes still need to run" is
derived from recorded per-node channel versions, not re-derived from a rebuilt graph.
That is the direction this issue asks for — a resume that is authoritative from the
record rather than from a recompile.
Channel equality compares policy, which is what makes a spec check possible.
BaseChannel subclasses define __eq__ on their configuration, not their value:
LastValue.__eq__ is true for any LastValue; NamedBarrierValue compares its
names set; BinaryOperatorAggregate compares operators; Topic compares
accumulate. So a rebuilt graph's channel spec can be compared against the
checkpointed one and a mismatch named precisely — rather than discovered as a cycle.

Note the caveat LangGraph itself documents, because it applies to any hash-based
version Nodus might write: lambdas all share the name <lambda>, so
_operators_equal treats any pairing involving a lambda as equal rather than risk a
false mismatch. A Nodus graph_signature_hash over step bodies would face the same
problem in reverse — a formatting change that alters the hash without altering the
graph. Hashing the topology (step names, after edges, checkpoint labels) rather
than the code avoids it, and is the smaller artifact anyway.

Note on this issue's premise

The summary says "Nothing records what a run's graph looked like ... no edges, no
topology, no hash" — accurate about _persist_graph_state, and it remains the right
statement of the gap. What the run does store is the whole program source
(workflow_lowering.py:391, see #499), from which the graph is re-derived. So the
artifact is there; it is code rather than topology, which is precisely why a comparison
is not possible.

Provenance

C:\codev\Langgraph research\langgraph,
libs/checkpoint/langgraph/checkpoint/base/__init__.py and
libs/langgraph/langgraph/channels/ — read from the source tree rather than via the study
document.

#470/#497: a rebuilt graph must be the planned shape, and both source…

Fixed in PR #563 (merged as e3c33ae, will ship in the next release).

What shipped, against this issue's fix direction:

Every run now records workflow_topology — step names + after edges, sorted — in its graph metadata (graph_topology() in workflow_lowering.py). Data rather than a hash, so a refusal can name what moved.
_rebuild_workflow_graph compares on rebuild and refuses a mismatch with the real cause, the MAF-style message this issue asked for: run '<id>' was planned against a different version of workflow 'w': its step structure has changed since the run started (steps added: z). A resume replays the planned structure; start a new run to use the edited workflow.
The original repro was re-confirmed on main before fixing (a legacy pre-Embedded run_source workflows can never be resumed: NodusRuntime records no source, so the rebuild has nothing to rebuild from #469 run, file edited to insert a step → Dependency cycle detected: z -> z, completely silent) and is now the regression test: the refusal replaces the false diagnosis. The mechanism was as suspected in Resume silently replays the source stored at first run, so edits are ignored — unless nothing was stored, when it reads the current file #497's cross-note — the inserted step collides with a stored task id during rebuild-from-disk.
Structure only, deliberately. The Nix-study trade recorded above was resolved as the two-tier comment proposed for the refuse tier: names and edges refuse; a body or when edit does not, because over-invalidating would make effectful long-running workflows unresumable after any edit. The warn-tier body hash was not built — with Embedded run_source workflows can never be resumed: NodusRuntime records no source, so the rebuild has nothing to rebuild from #469 having pinned every entry point, a pinned run's body cannot drift, and whole-source drift already warns (workflow_source_drift, and now source_drift: true on the result, Resume silently replays the source stored at first run, so edits are ignored — unless nothing was stored, when it reads the current file #497).
Legacy runs without the recorded topology are checked on step names alone (from step_to_task). An edge-only rewire on such a run remains undetectable — a stated limit, documented in the test file and the guide, and it decays away as pre-Embedded run_source workflows can never be resumed: NodusRuntime records no source, so the rebuild has nothing to rebuild from #469 runs age out.

Scope note held up: with #469's pinning, the live drift window really was the legacy disk-read branch — and the hash is now also the check that does not depend on that decision staying right (hand-edited state files, lowering changes across versions).

Guide §8 (workflows-and-tasks.md) documents both rules. Regression tests: tests/test_resume_topology_validation.py (# closes: #470).

release: 5.4.0 -- a resume that tells the truth, an inspection that c…

Shipped in v5.4.0 — live on PyPI: https://pypi.org/project/nodus-lang/5.4.0/ · release notes

Validated pre-publish (Gate 10: 6/6 dependent suites, 35/35 release-claim probes, 9/9 adversarial boundaries against the wheel in a clean venv) and post-publish (Stage 5, including a run started on 5.3.0 and resumed on 5.4.0). Eval documents: docs/evals/v5.4.0/.

release: 5.4.0 eval documents and the Stage 6 consumer republish (#583)

#704: key the bytecode cache on content, not only on the clock (#707)

Navigation Menu

Repository navigation

Resume does not validate graph topology: a source edit yields a spurious 'Dependency cycle detected' #470

Summary

Reproduction

Expected behavior

Fix direction

Scope note

Affected versions

Provenance

What goes into the hash — and the tension it exposes

The design this suggests, which is neither extreme

One transferable detail

A likely mechanism for the spurious cycle, and the reference for a topology check

The mechanism

The reference for what a topology check would compare

Note on this issue's premise

Provenance

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Resume does not validate graph topology: a source edit yields a spurious 'Dependency cycle detected' #470

Description

Summary

Reproduction

Expected behavior

Fix direction

Scope note

Affected versions

Provenance

Activity

Masterplanner25 commented on Aug 18, 2026

Masterplanner25 commented on Aug 19, 2026

Masterplanner25 commented on Aug 19, 2026

What goes into the hash — and the tension it exposes

The design this suggests, which is neither extreme

One transferable detail

Masterplanner25 commented on Aug 19, 2026

A likely mechanism for the spurious cycle, and the reference for a topology check

The mechanism

The reference for what a topology check would compare

Note on this issue's premise

Provenance

Masterplanner25 commented on Aug 25, 2026

Masterplanner25 commented on Aug 25, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions