Description
Description
Two gRPC users have reported problems with streaming calls make to services hosted by GCP/Google:
- gRPC stream "Response ended prematurely (ResponseEnded)" grpc/grpc-dotnet#2361
- gRPC Streaming From C# Resulting in GoAway error from server grpc/grpc-dotnet#2358
The streaming calls end after a small amount of activity with this error:
System.Net.Http.HttpIOException: The response ended prematurely while waiting for the next frame from the server. (ResponseEnded)
at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer, Boolean partOfSyncRead)
at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)
They're both verified that only the .NET client has this problem. Other gRPC implementations (Rust, go, Python) are successful.
Debugging logs shows that the problem is caused by the server aborting the connection because the client is sending too many pings. Logs:
poolId=44799463 workerId=57352375 requestId=0 memberName=ProcessOutgoingFramesAsync message=writeBytes=17
poolId=44799463 workerId=57352375 requestId=0 memberName=SendPingAsync message=Started writing. pingContent=-5
poolId=44799463 workerId=57352375 requestId=0 memberName=FlushOutgoingBytesAsync message=ActiveLength=17
poolId=44799463 workerId=57352375 requestId=0 memberName=ReadFrameAsync message=initialFrame=False
poolId=44799463 workerId=57352375 requestId=0 memberName=ProcessIncomingFramesAsync message=Frame 13: StreamId=0; Type=GoAway; Flags=None; PayloadLength=22.
poolId=44799463 workerId=57352375 requestId=0 memberName=ReadGoAwayFrame message=lastStreamId=1, errorCode=EnhanceYourCalm
The client sends a ping to the server, and the server responds with a GOAWAY with an error code of ENHANCE_YOUR_CALM.
The problem can be worked around by disabling RTT pings:
AppContext.SetSwitch("System.Net.SocketsHttpHandler.Http2FlowControl.DisableDynamicWindowSizing", true);Reproduction Steps
A customer has created a reproduction - grpc/grpc-dotnet#2361
Expected behavior
HttpClient should be able to successfully call GCP/Google hosted endpoints without the server aborting the connection.
Resolving this problem requires low level knowledge of HTTP/2 and client/server interactions. There is no way developers will understand how to solve this problem.
Actual behavior
GOAWAY + ENHANCE_YOUR_CALM
Regression?
No response
Known Workarounds
No response
Configuration
No response
Other information
No response