Guard the Nexus operation result future against false nondeterminism on replay#1353
Merged
Sushisource merged 2 commits intoJun 24, 2026
Merged
Conversation
…on replay
`StartedNexusOperation::result()` awaits `result_future.clone()`, where
`result_future` is a `Shared<WFCommandFut<NexusOperationResult, ()>>`. A `Shared`
future has internal waker machinery (like `FuturesOrdered` inside `join_all`): when
the inner future is unblocked during replay it invokes its waker outside the active
`SdkWakeGuard`, so the wake tracker records a non-SDK wake and fails the workflow task
with TMPRL1100 ("a waker was invoked by a non-SDK source").
This breaks any workflow that awaits a Nexus operation result and then keeps running.
The first activation that resolves the operation succeeds, but any later replay of that
history — a legacy query, a sticky-cache eviction, or a worker restart — fails the task,
so the execution can no longer be queried or durably recovered. Every other result future
(child workflow, activity) consumes `self` and uses the `WFCommandFut` directly, avoiding
`Shared`; the Nexus result is the only one that wraps it without guarding the poll.
Wrap the poll in `SdkGuardedFuture`, exactly as `join_all`'s `FuturesOrdered` already is
in `workflows.rs`, so the shared future's wakes are recognized as SDK-internal. `Shared`
is `Unpin`, which is all `SdkGuardedFuture` requires; the `&self` API is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
|
Sushisource
approved these changes
Jun 24, 2026
Co-authored-by: Spencer Judge <sjudge@hey.com>
Sushisource
enabled auto-merge (squash)
June 24, 2026 19:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
StartedNexusOperation::result()can fail a workflow task on replay with[TMPRL1100] Nondeterministic future detected: a waker was invoked by a non-SDK source,even though the workflow only awaits SDK-provided futures. Any workflow that awaits a Nexus
operation result and then keeps running becomes unqueryable and unrecoverable after the
operation completes.
Root cause
result()awaits a clone ofresult_future, aShared<WFCommandFut<NexusOperationResult, ()>>:A
Sharedfuture has internal waker machinery — the same class the SDK already special-cases.When the inner
WFCommandFutis unblocked during replay, theSharedinvokes its registeredwaker outside the active
SdkWakeGuard, soWakeTracker/PerPollWakeTrackerrecords a non-SDKwake and
workflow_future.rsfails the task with TMPRL1100.The SDK already documents and handles exactly this for
join_all:Every other operation result future avoids
Shared— e.g. child-workflowresult(self)consumesselfand uses theWFCommandFutdirectly. The Nexus result is the only path that wraps it inShared(soresult(&self)can be called repeatedly / cancelled after) without guarding the poll.Fix
Wrap the shared future's poll in
SdkGuardedFuture, mirroringjoin_all:SharedisUnpin(allSdkGuardedFuturerequires); the&selfAPI is unchanged.How it manifests
The first activation that resolves the operation succeeds, but any later replay of that
history fails the workflow task:
So the execution completes its turn once, then gets stuck (its workflow tasks time out) and can't
be queried. It only affects callers that keep running after awaiting the result — a caller that
returns immediately (as the existing
nexus_async/nexus_basictests do) never replays whilerunning, which is why CI hasn't caught it.
Validation
Reproduced and fixed end-to-end against a Temporal-compatible server in a downstream consumer: a
control workflow schedules an async Nexus operation (
WorkflowRunOperation→ backing workflow),awaits the result, records it, then parks on a
wait_conditionfor the next turn.get_run_statequeries fail and the workflow's taskstime out in a loop (
WorkflowTaskTimedOut), never recovering — worker logs show TMPRL1100.destroy → WorkflowExecutionCompletedwith zero workflow-task timeouts.On a regression test
The existing nexus tests call
fetch_history_and_replayon a completed workflow and pass. Thisbug needs a replay while the caller is still running (query / eviction / restart), which that
helper doesn't exercise — so a "continue-then-
fetch_history_and_replay" test wouldn't reliablyreproduce it. Happy to add a regression test in whatever form you prefer (e.g. a query- or
eviction-driven replay of a caller that parks after the result) — guidance on the harness you'd
like is welcome.
🤖 Generated with Claude Code