Resend parent workflow asynchronously during standby child completion verification

[michaely520]

Problem

A standby child workflow's CloseExecutionTask verifies its parent recorded the completion, and past MaxLocalParentWorkflowVerificationDuration also resends the parent from the active cluster. That resend is a cross-cluster state sync plus a possibly paginated history backfill — minutes of work — but the whole call was bounded by the standby task's hard-coded 3s taskTimeout. Measured at the active cluster's history shard, the deadline arriving there was 2.999s. So the resend never completed.

Change

• Run the resend in the background, bounded by ReplicationTaskApplyTimeout — the setting that already bounds this same work on the replication stream. The verify RPC returns immediately and the standby task retries until the parent lands, so it never holds a transfer-queue worker for the sync.
• The background context is detached from the request (gRPC cancels that when the handler returns) and rooted at the shard lifecycle, so the work stops with the shard.
• One in-flight resend per parent, tracked in a shard-level map, and at most history.parentWorkflowResendMaxInFlight (8) concurrent resends per shard. Callers retry while an earlier resend runs; without this the test measured 5 full state fetches where 1 suffices. The cap bounds the goroutines this path can create.
• Lifted two client ceilings so the deadline can actually propagate — admin.SyncWorkflowState (was 10s) and history.SyncWorkflowState (was 30s) now share a DefaultStateSyncTimeout backstop. This also fixes the same 10s cap on the replication stream's ExecutableTaskImpl.SyncState, where production's 5m setting was never reachable either.
• Metrics: parent_workflow_resend_{attempts,skipped,limited,failures,latency}. Async failures reach no caller, so _failures is the alert signal; _limited means the shard is shedding resends. The background goroutine recovers panics, which would otherwise take down the process.

Also fixes the history-client codegen template, which hardcoded createContext and silently ignored the timeout-tier field.

Rollout

history.enableAsyncParentWorkflowResend, default false. Disabled = the previous inline behavior, bounded by the caller's task deadline. Opt in per cell.

Testing

Unit tests cover the inline, async, and per-parent-dedup paths.

An xdc test (added in abae9cec, removed in b6eb6631) withholds the parent's replication tasks so the child must pull it, asserts the parent is absent from the standby, then stalls the active cluster's SyncWorkflowState for 4 minutes:

--- PASS: TestChildPullsParentWhenParentReplicationIsWithheld (286.03s)
incoming-ctx-remaining: 4m59.999859334s    (2.999s before this change)
sync-state-calls:       1                  (5 without the per-parent guard)
dropped-parent-tasks:   9

During the stall, 4 verify RPCs reached the standby parent shard (t+0, +50s, +101s, +169s) and exactly 1 SyncWorkflowState reached the active cluster: the task retried and the guard turned the retries away.

4 minutes exceeds every deadline that previously bounded this path (3s / 10s / 30s) with ~1m headroom against the 5m setting, so the setting is demonstrably what governs.

To reproduce: git revert b6eb6631, then
go test -tags test_dep ./tests/xdc/ -run TestVerifyChildCompletionParentResendSuite -timeout 30m

Known gaps

• Concurrency across distinct parents is unbounded (ordinary fan-out, not amplification).
• When the parent is deleted on the source, the async path can't report that back, so the child retries to the 15m discard instead of finishing immediately. The workflowNotFoundCache TODO already in this file would address it.

[xwduan]

Maybe consider adding a max size for this map? So we can use it to control the maximum go routine count that are created for resending parent for each shard. Otherwise I am afraid some namespace pattern would cause sudden increase of new go routine created from this code path.

[michaely520]

my hope is this will reduce the outstanding number of go-routines, but as a sanity mechanism added a cap

[yux0]

Is this initialized from fx?

[michaely520]

its initialized lazily on the first claim, we just use the zeroed value that is instantiated when the history engine is first instantiated per shard