Skip to content

Guard the Nexus operation result future against false nondeterminism on replay#1353

Merged
Sushisource merged 2 commits into
temporalio:mainfrom
tokeira:fix/guard-nexus-result-future
Jun 24, 2026
Merged

Guard the Nexus operation result future against false nondeterminism on replay#1353
Sushisource merged 2 commits into
temporalio:mainfrom
tokeira:fix/guard-nexus-result-future

Conversation

@iw

@iw iw commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Summary

StartedNexusOperation::result() can fail a workflow task on replay with
[TMPRL1100] Nondeterministic future detected: a waker was invoked by a non-SDK source,
even though the workflow only awaits SDK-provided futures. Any workflow that awaits a Nexus
operation result and then keeps running becomes unqueryable and unrecoverable after the
operation completes.

Root cause

result() awaits a clone of result_future, a Shared<WFCommandFut<NexusOperationResult, ()>>:

// crates/workflow/src/workflow_context.rs
pub async fn result(&self) -> NexusOperationResult {
    self.unblock_dat.result_future.clone().await
}

A Shared future has internal waker machinery — the same class the SDK already special-cases.
When the inner WFCommandFut is unblocked during replay, the Shared invokes its registered
waker outside the active SdkWakeGuard, so WakeTracker/PerPollWakeTracker records a non-SDK
wake and workflow_future.rs fails the task with TMPRL1100.

The SDK already documents and handles exactly this for join_all:

// crates/workflow/src/workflows.rs
JoinAll(SdkGuardedFuture(futures_util::future::join_all(iter)))

A future wrapper that activates SdkWakeGuard during poll. Use this around futures whose
internal waker machinery (e.g., FuturesOrdered inside join_all) would otherwise trigger
false positives in nondeterminism detection.

Every other operation result future avoids Shared — e.g. child-workflow result(self) consumes
self and uses the WFCommandFut directly. The Nexus result is the only path that wraps it in
Shared (so result(&self) can be called repeatedly / cancelled after) without guarding the poll.

Fix

Wrap the shared future's poll in SdkGuardedFuture, mirroring join_all:

pub async fn result(&self) -> NexusOperationResult {
    SdkGuardedFuture(self.unblock_dat.result_future.clone()).await
}

Shared is Unpin (all SdkGuardedFuture requires); the &self API is unchanged.

How it manifests

The first activation that resolves the operation succeeds, but any later replay of that
history fails the workflow task:

  • a (legacy) query of the still-running workflow,
  • a sticky-cache eviction, or
  • a worker restart.

So the execution completes its turn once, then gets stuck (its workflow tasks time out) and can't
be queried. It only affects callers that keep running after awaiting the result — a caller that
returns immediately (as the existing nexus_async/nexus_basic tests do) never replays while
running, which is why CI hasn't caught it.

Validation

Reproduced and fixed end-to-end against a Temporal-compatible server in a downstream consumer: a
control workflow schedules an async Nexus operation (WorkflowRunOperation → backing workflow),
awaits the result, records it, then parks on a wait_condition for the next turn.

  • Before: the turn completes once, then get_run_state queries fail and the workflow's tasks
    time out in a loop (WorkflowTaskTimedOut), never recovering — worker logs show TMPRL1100.
  • After: the turn completes, the post-completion query returns normally, and
    destroy → WorkflowExecutionCompleted with zero workflow-task timeouts.

On a regression test

The existing nexus tests call fetch_history_and_replay on a completed workflow and pass. This
bug needs a replay while the caller is still running (query / eviction / restart), which that
helper doesn't exercise — so a "continue-then-fetch_history_and_replay" test wouldn't reliably
reproduce it. Happy to add a regression test in whatever form you prefer (e.g. a query- or
eviction-driven replay of a caller that parks after the result) — guidance on the harness you'd
like is welcome.

🤖 Generated with Claude Code

…on replay

`StartedNexusOperation::result()` awaits `result_future.clone()`, where
`result_future` is a `Shared<WFCommandFut<NexusOperationResult, ()>>`. A `Shared`
future has internal waker machinery (like `FuturesOrdered` inside `join_all`): when
the inner future is unblocked during replay it invokes its waker outside the active
`SdkWakeGuard`, so the wake tracker records a non-SDK wake and fails the workflow task
with TMPRL1100 ("a waker was invoked by a non-SDK source").

This breaks any workflow that awaits a Nexus operation result and then keeps running.
The first activation that resolves the operation succeeds, but any later replay of that
history — a legacy query, a sticky-cache eviction, or a worker restart — fails the task,
so the execution can no longer be queried or durably recovered. Every other result future
(child workflow, activity) consumes `self` and uses the `WFCommandFut` directly, avoiding
`Shared`; the Nexus result is the only one that wraps it without guarding the poll.

Wrap the poll in `SdkGuardedFuture`, exactly as `join_all`'s `FuturesOrdered` already is
in `workflows.rs`, so the shared future's wakes are recognized as SDK-internal. `Shared`
is `Unpin`, which is all `SdkGuardedFuture` requires; the `&self` API is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@iw
iw requested a review from a team as a code owner June 24, 2026 18:13
@CLAassistant

CLAassistant commented Jun 24, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@Sushisource Sushisource left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

Comment thread crates/workflow/src/workflow_context.rs Outdated
Co-authored-by: Spencer Judge <sjudge@hey.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants