Resume silently replays the source stored at first run, so edits are ignored — unless nothing was stored, when it reads the current file #497

Summary

resume_workflow does not run the workflow in the file you are running. It recompiles
the whole program source captured at the original run, stored in the run's metadata.
So a source edit made between the run and the resume has no effect, and there is no
warning that the code being executed is not the code on disk.

Add a step and resume: the step never runs, and the resume reports success. Delete a step
and resume: it runs anyway. Resume from a file that defines no workflow at all: the
original steps run.

And when the run stored no source — which is every run from an embedded NodusRuntime
(#496) — the same function falls back to reading the current file from
workflow_source_path, which does pick up edits. Two different rebuild sources,
selected by how the run was started, with opposite behaviour and nothing distinguishing
them to the author.

Reproduction

Verified against src/ at 5.0.4.

The original run (pE_run.nd):

workflow w {
    state log = ""
    step a { print("  a ran"); log = log + "a"; checkpoint "cp"; return 1i }
    step b after a { print("  b ran"); log = log + "b"; return 2i }
}

fn main() {
    let r = run_workflow(w)
    let g = r["graph_id"]
    let s = r["state"]
    print("gid=\(g) state=\(s)")
}

  a ran
  b ran
gid=g_e3d501ff state={"log": "ab"}

Case 1 — a step added. A new file with the same workflow plus step c after a:

    step c after a { print("  *** c ran -- c did not exist in the original run ***"); log = log + "c"; return 3i }

  a ran
  b ran
resumed state={"log": "aab"}
resumed steps={"a": 1, "b": 2}

c never ran. No error, no warning; the resume reports success and steps simply does
not mention it. (The "aab" is #486 — pre-checkpoint work re-executing — and is not this
issue.)

Case 2 — a step removed. A file whose workflow w contains only step a:

  a ran
  b ran
resumed state={"log": "aaab"}
resumed steps={"a": 1, "b": 2}

b ran, from code that is not in the file.

Case 3 — no workflow at all. The entire program:

fn main() {
    let r = resume_workflow("g_e3d501ff", "cp")
    print("result: \(r)")
}

  a ran
  b ran
result: {"tasks": {"task_1": 1, "task_2": 2}, "steps": {"a": 1, "b": 2}, ... "workflow": "w"}

Where the code comes from

src/nodus/orchestration/workflow_lowering.py:391 stores it on every run:

"workflow_source_code": getattr(vm, "source_code", None),

vm.source_code is the whole module source, not the workflow — the stored blob for
the run above contains fn main() and everything else in the file.

src/nodus/vm/vm.py:1523-1533 is the fork:

source_code = metadata.get("workflow_source_code")
source_path = metadata.get("workflow_source_path")
if not isinstance(source_code, str):
    if not isinstance(source_path, str) or not source_path or not os.path.exists(source_path):
        raise WorkflowRebuildError(...)
    with open(source_path, "r", encoding="utf-8") as f:
        source_code = f.read()

• Stored source present → pinned: edits ignored, silently.
• Stored source absent, path present → current file: edits picked up, silently.
• Neither → WorkflowRebuildError. That is Workflow runs started from an embedded NodusRuntime cannot be resumed after the process exits #496.

Which branch a run takes is fixed at run time by its entry point, and vm.source_code is
set only by tooling/runner.py and dap/server.py.

Why this is worth fixing rather than documenting

Pinning is defensible on its own — arguably it is the right default, since replaying
against the source the run was planned for is what makes deterministic re-execution
meaningful, and it is what #494 would want. Two things make it a defect as it stands:

1. It is silent in the direction that matters. The normal debugging loop is: a
   workflow fails, you edit a step, you resume. Nodus runs the old step and reports
   success. There is no signal at all — not a warning, not a field in the result map.
2. The behaviour is not a property of resume, it is a property of the entry point.
   The same call means "replay the pinned program" from the CLI and "recompile whatever is
   on disk now" from anything that did not set source_code. Neither is documented, and
   nothing in the API surface distinguishes them.

This is very likely the mechanism behind #470: that report edits the source and gets a
spurious Dependency cycle detected, which is what recompiling an edited file against a
task set planned for the old shape would produce — i.e. the source_path branch. Worth
confirming when that one is worked, because if so the two issues share a fix.

Fix direction

Pick one meaning and make the other impossible:

1. Pin, and say so. Keep the stored source authoritative, and record a hash of it. On
   resume, if source_path exists and its current content hashes differently, report it —
   in the result map at minimum, and probably as a refusal unless the caller opts in with
   something like resume_workflow(id, label, {allow_source_drift: true}). The author then
   learns immediately that their edit is not in the run.
2. Remove the fallback. The source_path branch is what makes the behaviour
   entry-point dependent. If pinning is the intent, a run with no stored source should
   fail rather than silently recompile a file that may have moved on — and Workflow runs started from an embedded NodusRuntime cannot be resumed after the process exits #496 should be
   fixed by storing the source on the embedded path, not by leaning on the path branch.

Either way the decision belongs in one place. As written, it lives in the two-branch
if above, which is the shape CLAUDE.md § The recurring bug shape warns about.

Related

• Workflow runs started from an embedded NodusRuntime cannot be resumed after the process exits #496 — an embedded run stores neither, so it cannot resume at all. Same three lines.
• Resume does not validate graph topology: a source edit yields a spurious 'Dependency cycle detected' #470 — resume does not validate topology. Probably the same fork; see above.
• Resume re-executes with no determinism boundary: a replayed step observes a different clock, and nothing says so #494 — resume re-executes with no determinism boundary. Source pinning is one half
  of a determinism story that has no other half.
• A mid-step checkpoint is a whole-step re-entry label, so work before it re-executes on resume #486 — a mid-step checkpoint re-enters the whole step. Visible in the outputs above
  as the repeated a.

Affected versions

v5.0.4 (current when filed). Not a regression.

Provenance

C:\codev\Langgraph research\langgraph, libs/checkpoint/langgraph/checkpoint/base/__init__.py
— read from the source tree rather than via the study document.

LangGraph's Checkpoint never refers to program text. It carries channel_values,
channel_versions, versions_seen ("map from node ID to map from channel name to
version seen — used to determine which nodes to execute next") and updated_channels.
The graph is supplied by the caller at resume; what the checkpoint pins is how far each
node got, not what the nodes were.

That is the opposite trade from Nodus's, and neither is obviously right — LangGraph's
means a resumed graph can silently differ from the checkpointed one, which is exactly
#470. What it does have is a mechanism for noticing: BaseChannel.__eq__ compares
policy rather than value (LastValue.__eq__ is true for any LastValue;
NamedBarrierValue compares its names set; BinaryOperatorAggregate compares
operators), so a rebuilt graph's channel spec can be checked against the checkpointed one.

Nodus stores neither a topology nor a spec — it stores the program and re-derives. Asking
what happens when the program changes underneath that is what produced this.