Repository navigation
Resume re-executes with no determinism boundary: a replayed step observes a different clock, and nothing says so #494
Description
Summary
Resume works by re-executing a step from its checkpoint. Nothing records what the
original execution observed, so any non-deterministic read — the clock, randomness, the
environment, a file, an HTTP response — returns a different value on the replay. There
is no replay-safe API, no recorded observation, and no divergence detection.
Checkpointed state is restored faithfully. It is the fresh reads that silently move.
Reproduction
Verified against src/ at 5.0.4.
import "std:time" as time
workflow w {
state first_seen = ""
step observe {
let now = "\(time.now())"
if (first_seen == "") { first_seen = now }
print("step ran, clock = \(now)")
print(" state first_seen = \(first_seen)")
checkpoint "observed"
return now
}
}
First run:
step ran, clock = record {"epoch_ms": 1787116463489, ..., "second": 23, "ms": 489, ...}
state first_seen = record {"epoch_ms": 1787116463489, ...}
resume_workflow(id, "observed"):
step ran, clock = record {"epoch_ms": 1787116471945, ..., "second": 31, "ms": 945, ...}
state first_seen = record {"epoch_ms": 1787116463489, ...}
The replayed step saw a clock 8 seconds later. first_seen is correct, because it
was checkpointed; now is not, because it was read fresh.
Here the if (first_seen == "") guard absorbs it. A step that branches on
time.now(), secrets.random_*, an env read or an HTTP response takes a different path
on replay, and nothing anywhere says so.
Why this needs a position rather than a fix
Nodus has never claimed determinism, and re-execution is a legitimate resume strategy —
it is what makes checkpoint-restore cheap. The problem is that the property is
undocumented, undetectable and unavoidable:
docs/runtime/EXECUTION_INVARIANTS.mdmentions "deterministic" exactly once, about
exception unwinding. Nothing states what a resumed step may or may not observe.- There is no replay-safe alternative to reach for. A workflow author who wants a stable
clock across replays has to checkpoint it intostateby hand, and has to know to. - A mid-step checkpoint is a whole-step re-entry label, so work before it re-executes on resume #486 makes this materially more reachable than it looks: a mid-step
checkpointis
a whole-step re-entry label, so everything before it re-executes on resume — including
reads the author placed before the checkpoint precisely because they thought they were
done.
Options
- State the boundary. The cheapest and probably the most valuable: document that a
resumed step re-executes and that any fresh read may differ, with the checkpoint-it-into-
stateidiom as the supported answer. This is a guide section, not a code change. - Detect divergence. Record a hash of a step's observed inputs at checkpoint time and
compare on replay, warning or failing on mismatch. Temporal's non-determinism error is
the reference. Real work, and it needs (3) to be worth much. - Replay-safe APIs.
workflow_now()and friends, returning the value recorded at
first execution rather than a fresh read. This is Temporal's answer
(workflow.now()/workflow.random()), and the audit's point is that it belongs in
the language VM, not in a host runtime.
(1) is worth doing regardless of whether (2) or (3) is ever taken.
Related
- A mid-step checkpoint is a whole-step re-entry label, so work before it re-executes on resume #486 — a mid-step checkpoint re-runs the whole step. That is the mechanism that
makes this reachable; this issue is what the re-execution can silently change. - Resume does not validate graph topology: a source edit yields a spurious 'Dependency cycle detected' #470 — resume does not validate topology. Both are "resume does something other than
what the author assumed"; that one is about the graph, this one about observations. - A step failure kills independent fan-out branches, and a join has no failure semantics #475, Concurrent fan-out branches silently lose state writes: no write-merge discipline behind a join #485 — the other workflow-semantics gaps found in this sweep.
Affected versions
v5.0.4 (current when filed). Not a regression — no determinism boundary has ever existed.
Provenance
C:\codev\Temporal research\TEMPORAL_AINDY_LENS_AUDIT.md, second "Absorb into Nodus"
item:
Determinism boundary clarity — adopt Temporal's runtime/SDK split as a design
rule for Nodus: keep replay-divergence/determinism a language-VM concern, never a
kernel concern.
Temporal is the one system in this corpus built entirely around this constraint: its
workflow history is the program record, workflow code must be deterministic so it can be
replayed against that history, and the SDK supplies replay-safe substitutes for every
non-deterministic primitive. The audit's recommendation is not to copy the mechanism but
to accept the layering — that this is the language's problem.
Checking whether Nodus has any position on it at all is what produced this. It has none:
no replay-safe API, no recorded observations, no divergence detection, and no statement in
the runtime docs.
The audit's first absorb item — "harden checkpoint/resume so a graph reload is
authoritative, and surface the graph as the canonical program artifact" — is #470,
independently found from the MAF corpus and reproduced there.
Activity
Masterplanner25 commentedon Aug 19, 2026
A fourth option, cheaper than divergence detection: declare what is not durable
This issue lists three options — state the boundary, detect divergence, provide
replay-safe APIs. The LangGraph source tree has a fourth that is neither detection nor
substitution, and it is the cheapest of the four to reason about:
Make the durability of each state cell a declared property, so a value that must not be
replayed comes back empty rather than stale.
libs/langgraph/langgraph/channels/untracked_value.py:and
ephemeral_value.py, "Stores the value received in the step immediately preceding,clears after."
The property that matters here is what happens on the read. An untracked cell restored
from a checkpoint is empty, and reading it raises
EmptyChannelError. So the programcannot silently observe a value from a previous life — it is forced to re-derive, at a
point the author chose. That is divergence prevented by construction rather than
detected after the fact, and it needs no observation hashing, no history, and no
replay-safe API surface.
It does not replace option (3).
time.now()still needs an answer, because thenon-determinism is in a builtin, not in a cell. But it covers the case where the
non-deterministic value was read once and carried forward — which is the idiom this issue
already recommends by hand:
With a durability declaration, both halves of that become sayable rather than conventional:
a cell that is checkpointed is the stable-clock idiom made explicit, and a cell marked
untracked is the "never trust this after a resume" case that currently cannot be expressed
at all.
Related, and it constrains the design
#498 (filed today) is the same declaration slot from the other side: workflow
statetoday has no durability declaration and an unstated, uniform requirement that every cell
be JSON-serializable. A closure or a channel in
stateaborts the run. So theuntrackedcase is not merely a nicety for this issue — it is the only way a workflowcould hold a live handle at all.
Which means the durability axis is going to be designed regardless, for #498. Settling it
with this issue in view costs nothing extra and avoids two unrelated annotations on the
same declaration. #485's merge policy and #479's type are the other two.
#497 (filed today) also sharpens option (1). Resume replays the source stored at the
original run, so an author investigating replay behaviour by editing their workflow and
resuming sees no change and no warning. Any guidance written for this issue should say
that too, or the reader's first experiment will mislead them.
Provenance
C:\codev\Langgraph research\langgraph,libs/langgraph/langgraph/channels/— read fromthe source tree.
untracked_value.pyandephemeral_value.pyappear in no audit documentto date;
LANGGRAPH_ON_NODUS_IMPLEMENTATION_STUDY.mdnames four of the eight channels andneither of these.
fix(embedding): pin resume to the source the run was planned against (#…
Masterplanner25 commentedon Aug 25, 2026
Cross-note: #470 (the 'the graph may have changed' side of this seam) is fixed as of PR #563 — every run records its topology (step names + edges) and a rebuild refuses a mismatch with the real cause. The 'the world may have changed' side this issue tracks is still open and unchanged. One hook from that work is relevant here: with #469 pinning source on every entry point and #563 validating structure, the remaining nondeterminism on resume is exactly what this issue names — clock, environment, and effects observed by re-executed step prefixes (#486). The topology record gives a determinism boundary somewhere to attach if one is built.
Masterplanner25 commentedon Aug 29, 2026
Fixed in PR #651 (merged) — option 1, the boundary is stated
Reproduced at 5.6.0 first:
New
I-WFLOW-07inEXECUTION_INVARIANTS.mdstates it; the guide's Checkpoints section gives the supported answer — write the observation intostatebefore the checkpoint — with output pasted from a real run.Options 2 and 3 are deliberately not taken. There is no divergence detection and no replay-safe clock, and the invariant says so explicitly: it records the position, not a mechanism. This issue's own framing is the reason — Nodus has never claimed determinism and re-execution is what makes checkpoint-restore cheap. Reopen for a mechanism if a real case needs one.
The more serious thing, found while checking
I-WFLOW-06was wrong. It claimed a resume never re-executes completed steps, naming bothresume_workflow(id)andresume_workflow(id, "label"). Measured on three completed steps with the checkpoint in the middle:abcSo it holds for the plain form only. This matters more than a wording slip: the labelled form is what a debugging loop reaches for, and a reader trusting the invariant would conclude their completed steps were protected. Corrected, with the measurement in place.
That also sharpens this issue's own point about #486: it is not only that everything before a mid-step checkpoint re-runs — everything downstream of that step re-runs too.
One detail beyond the issue
The step's return value is the replay's observation.
steps["gather"]after a resume is the second reading, not the first — what a host reporting "when did this run?" from a resumed result would get wrong.Tests
tests/test_resume_determinism_boundary.py, five of them, asserting both halves. A test that only checked "the fresh read moved" would pass on a runtime that had frozen the entire step; one that only checked "the checkpointed value held" would pass on a runtime that replayed nothing. Neither alone describes a boundary.The probe is
print, not workflowstate— state is re-derived from the checkpoint on resume, so a counter written into it cannot measure whether a step re-ran. That is the exact instrument error that produced a wrong claim in the compensation spec earlier in this cycle.