Skip to content

Resend child workflow async when missing on passive - #11705

Open
jiechenz wants to merge 2 commits into
jiechenz/xdc-parent-childfrom
jiechenz/resend-child
Open

Resend child workflow async when missing on passive#11705
jiechenz wants to merge 2 commits into
jiechenz/xdc-parent-childfrom
jiechenz/resend-child

Conversation

@jiechenz

Copy link
Copy Markdown
Contributor

What changed?

  • Added an opt-in recovery path for a child workflow missing from a standby cluster.
  • After the existing resend delay, VerifyFirstWorkflowTaskScheduled asynchronously fetches the child state from the active cluster, applies it locally, and verifies it again.
  • Added deduplication, per-shard concurrency limits, metrics, namespace checks, and transition-history gating.
    Corrected the discard-time source check to verify the child workflow rather than the parent.
  • Moved the reusable in-flight resend tracker into the shared workflowresend package.
  • Updated the existing XDC parent-child test to assert that the missing child and its first workflow task are restored.

Why?

Cross-shard replication may deliver the parent’s ChildWorkflowExecutionStarted event before the child workflow reaches the standby cluster. Previously, verification repeatedly returned NotFound and eventually discarded the standby task, leaving the child missing.
This adds the child-side symmetric recovery behavior to the parent resend implemented in #11424 .

How did you test it?

  • built
  • run locally and tested manually
  • covered by existing tests
  • added new unit test(s)
  • added new functional test(s)

Rollout

history.enableChildWorkflowResend default to false.

Known

  • Enabling this feature introduces additional cross-cluster state-sync traffic. It is disabled by default and protected by deduplication and a per-shard concurrency limit.
  • Regular replication may race with state sync; duplicate application is treated as success.
  • The resend delay and replication timeout should remain below the standby task discard delay so recovery has time to complete.

@jiechenz
jiechenz marked this pull request as ready for review August 21, 2026 16:54
@jiechenz
jiechenz requested a review from a team August 21, 2026 16:54
@jiechenz
jiechenz requested review from a team as code owners August 21, 2026 16:54
return nil
}
switch errVerify.(type) {
case *serviceerror.NotFound, *serviceerror.WorkflowNotReady:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should resend only when not found?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel we should also retry on workflow started, but not scheduled scenario, where the error is WorkflowNotReady

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants