Replay fails: UpdateCompleted events between version markers breaks replay

[gauravthadani]

Expected Behavior

When an Update handler contains a SideEffect or LocalActivity and completes during the same workflow task as getVersion markers, replay should succeed deterministically without errors.

Actual Behavior

A workflow task failure occurs during replay with the following error:

[TMPRL1100] getVersion call before the existing version marker event. The most probable cause is retroactive addition of a getVersion call with an existing 'changeId'

Version: failure executing RESULT_NOTIFIED_REPLAYING->NON_MATCHING_EVENT, transition history is [CREATED->CHECK_EXECUTION_STATE, REPLAYING->SCHEDULE, MARKER_COMMAND_CREATED_REPLAYING->RECORD_MARKER]

Failure handling event 26 of type 'EVENT_TYPE_WORKFLOW_EXECUTION_UPDATE_COMPLETED' during replay. {WorkflowTaskStartedEventId=41, CurrentStartedEventId=3}
The UpdateCompleted events are interleaved between two version markers in history, causing a mismatch during replay.

Steps to Reproduce the Problem

• Start a workflow that uses getVersion markers (patching).
• Send Updates simultaneously while the workflow task is in flight.
• Update handler blocks on a SideEffect or LocalActivity.
• Observe that UpdateCompleted events are written between the two version marker events in history.
• Trigger replay — workflow task fails with the error above.

Additional Context

Workaround: using Workflow.await(Workflow::isEveryHandlerFinished) before getVersion calls prevents the race, but this feels like a workaround rather than a fix.
The SDK should replay in the same order it initially executed — this failure suggests a bug in the SDK state machine when Update completions interleave with version markers.

some sample code here - https://github.com/gauravthadani/samples-kotlin/tree/main/app/src/main/kotlin/io/temporal/samples/update_nde

Specifications

• Version:
• Platform: