Skip to content

Retry safe ingress attach/result reads after drain and transport failures #769

Description

@pcholakov

Problem

The TypeScript ingress client retries invocation submission/call when ConnectionOpts.retry is enabled, but it does not retry safe GET-based result/attach reads. This leaves a customer-visible gap during Cloud ingress drain: a result(send), workflowAttach, or workflowOutput request can race a draining ingress pod and fail even though the durable invocation completed exactly once.

Current behavior

  • HttpIngress.result() performs one GET and throws on a non-OK response; it has no retry loop.
  • workflowAttach and workflowOutput use the same one-shot GET behavior.
  • The existing retry policy is used for invocation submission/call and covers fetch rejection, 429, and 5xx.
  • Existing retry integration coverage exercises connect().call, not attach/result failure and retry behavior.

Requested change

When ConnectionOpts.retry is enabled, retry the safe GET operations:

  • result(send) / attachable send result
  • workflowAttach
  • workflowOutput (if it has the same GET semantics)

These are reads, so they do not need an idempotency-key gate. Reuse the established retry policy for transport failures, 429, and 5xx, including ingress drain 503 and Retry-After. Respect explicit caller cancellation; retries must not outlive a caller-provided abort signal.

Regression coverage

Add persistent-connection tests for:

  1. 503 drain response followed by success (including Connection: close on HTTP/1.1).
  2. Rejected fetch / transport failure followed by success.
  3. Explicit abort does not retry.

Why this matters

Cloud-ingress rollout testing observed exactly-once durable completion while an attach/read path timed out. Server-side Connection: close reduces exposure, but it cannot make a one-shot GET transparently recover from a drain race.

Evidence

Verified against current upstream origin/main at 9bb2570. No matching existing issue was found.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions