Skip to content

Dapr.Testcontainers: probe gRPC port with HTTP/2 preface to eliminate transient Unavailable on first RPC - #1821

Merged
WhitWaldo merged 5 commits into
masterfrom
copilot/fix-integration-test-retry-logic
May 18, 2026
Merged

Dapr.Testcontainers: probe gRPC port with HTTP/2 preface to eliminate transient Unavailable on first RPC#1821
WhitWaldo merged 5 commits into
masterfrom
copilot/fix-integration-test-retry-logic

Conversation

Copilot AI commented May 18, 2026

Copy link
Copy Markdown
Contributor

Description

Integration tests intermittently fail on the first RPC with:

Grpc.Core.RpcException: Status(StatusCode="Unavailable",
  Detail="Error connecting to subchannel. ... Connection refused")

Root cause: DaprdContainer.StartAsync only validates the gRPC port at the TCP layer. A successful TCP connect proves Docker's port forwarding accepts SYNs — it does not prove the upstream HTTP/2 (gRPC) server is actively serving. On slow CI hosts there is a brief window between Docker publishing the port and daprd's gRPC Serve() loop being wired up; the test's first RPC lands in that window and gets Unavailable. The HTTP reachability probe already in place only covers the HTTP server on a different port.

For reference, daprd's startup log order (dapr/dapr/pkg/runtime/runtime.go L790/L802/L813) is API gRPCHTTPInternal gRPC, so the existing "Internal gRPC server is running" wait is in fact the last of the three and is not the source of the flake. An earlier commit on this branch that changed that wait message is reverted here.

Changes:

  • ContainerReadinessProbe.WaitForGrpcServerReadyAsync(host, port, timeout, ct) — new probe that performs the HTTP/2 connection handshake itself: opens TCP, sends the 24-byte client preface (RFC 7540 §3.5: PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n), and waits for any bytes back from the server's SETTINGS frame. Per-attempt bound of 2 s, 200 ms inter-attempt backoff, retries SocketException/IOException/attempt-timeout until the overall timeout. Dependency-free; works against any HTTP/2 (gRPC) server.
  • DaprdContainer.StartAsync — invokes the new probe against 127.0.0.1:GrpcPort as the final readiness gate, alongside the existing HTTP reachability probe for the HTTP port.
  • ContainerReadinessProbeTests — 4 tests: HTTP/2 server preface → returns; accepts-but-never-responds (simulates the CI race) → TimeoutException; port refused → retries then times out; pre-cancelled token → OperationCanceledException.
  • Reverts the earlier wait-message change in DaprdContainer.cs; the log line is back to "Internal gRPC server is running".

Out of scope:

  • test/Dapr.E2E.Test/DaprTestApp.cs (uses the dapr run CLI, different startup mechanism).
  • PortUtilities.GetAvailablePort releases the socket before returning the number — TOCTOU race used in app-first mode, already wrapped in DaprTestApplicationBuilder's retry loop.

Issue reference

Checklist

  • Code compiles correctly
  • Created/updated tests
  • Extended the documentation

@codecov

codecov Bot commented May 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.80488% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.01%. Comparing base (62d3a6e) to head (cc369fa).

Files with missing lines Patch % Lines
...r.Testcontainers/Common/ContainerReadinessProbe.cs 86.11% 3 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1821      +/-   ##
==========================================
+ Coverage   64.90%   65.01%   +0.10%     
==========================================
  Files         300      300              
  Lines        8833     8874      +41     
  Branches     1040     1043       +3     
==========================================
+ Hits         5733     5769      +36     
- Misses       2846     2849       +3     
- Partials      254      256       +2     
Flag Coverage Δ
net10.0 64.99% <87.80%> (+0.10%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@WhitWaldo
WhitWaldo marked this pull request as ready for review May 18, 2026 22:34
@WhitWaldo
WhitWaldo requested review from a team as code owners May 18, 2026 22:34
@WhitWaldo
WhitWaldo merged commit d708be4 into master May 18, 2026
693 of 696 checks passed
@WhitWaldo
WhitWaldo deleted the copilot/fix-integration-test-retry-logic branch May 18, 2026 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants