gRPC stream "Response ended prematurely (ResponseEnded)"

[jeffijoe]

What version of gRPC and what language are you using?

C#, Grpc.Net.Client version 2.60.0

What operating system (Linux, Windows,...) and version?

macOS 14.2.1, Linux (Ubuntu on Google Cloud Kubernetes Engine)

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

.NET SDK:
 Version:           8.0.100

What did you do?

This may potentially be related to #2358

We have a long-lived gRPC streaming call that gets canceled after receiving a few messages with the following:

Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error reading next message. HttpIOException: The response ended prematurely while waiting for the next frame from the server. (ResponseEnded)", DebugException="System.Net.Http.HttpIOException: The response ended prematurely while waiting for the next frame from the server. (ResponseEnded)")
 ---> System.Net.Http.HttpIOException: The response ended prematurely while waiting for the next frame from the server. (ResponseEnded)
   at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
   at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer, Boolean partOfSyncRead)
   at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.StreamExtensions.ReadMessageAsync[TResponse](Stream responseStream, GrpcCall call, Func`2 deserializer, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.GrpcCall`2.ReadMessageAsync(Stream responseStream, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.HttpContentClientStreamReader`2.MoveNextCore(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Grpc.Net.Client.Internal.HttpContentClientStreamReader`2.MoveNextCore(CancellationToken cancellationToken)
   at Grpc.Core.AsyncStreamReaderExtensions.ReadAllAsyncCore[T](IAsyncStreamReader`1 streamReader, CancellationToken cancellationToken)+MoveNext()
   at Grpc.Core.AsyncStreamReaderExtensions.ReadAllAsyncCore[T](IAsyncStreamReader`1 streamReader, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()

So far, we can reproduce this when using a L7 external load balancer in Google Cloud (Classic Application Load Balancer). When using an L4 external load balancer, it works as expected.

We have built an extensive reproduction, including multiple servers and clients in different languages to isolate the root cause as being the gRPC dotnet client library. The following clients have been tested and work without issue:

• grpcurl
• custom Golang client (see repo)
• custom Rust client (see repo)

The linked repository also includes information about the infrastructure setup.

The following proto is used:

syntax = "proto3";

option csharp_namespace = "LbIssue";
option go_package = "github.com/taxfyle/lb-issue-repro/src/go/pb";

package lb_issue;

service Demo {
  rpc StreamMessages (StreamMessagesRequest) returns (stream StreamMessagesResponse);
}

message StreamMessagesRequest {
  string name = 1;
}

message StreamMessagesResponse {
  string message = 1;
}

At first, we thought the root cause was the L7 load balancer, however, given other gRPC client libraries work without issue, we figured it might be something specific in the dotnet one.

The issue occurs after having received between 5-8 messages on the stream when sending a message roughly every second. 6 messages is the most common one we've observed.

If we send messages faster (using a while loop that calls a HTTP endpoint on the server to trigger messages), we get between 20-25 messages before the connection is broken.

We also observed that the connection breaks after receiving a message, never while waiting for one.

[JamesNK]

I looked at your .NET client and server sample apps and they both look fine. I also tested them on my local computer (no load balancer involved), and they worked without issue. That must mean there is an interaction between .NET gRPC client + the L7 load balancer that causes the failure.

Is setting up the solution in GCP required? I don't have experience with using it so that would be a bit of a barrier to me quickly testing it failing. I think the issue is in the gRPC client (since the server works with all the other clients) so that's the part I'd want to test.

If you have a GCP endpoint I can point a client on my local machine at, that will speed things up. I can set up gathering of low-level diagnostics and then run the client until there is a failure. If you don't want to share the address publicly then contact me over email at username@microsoft.com (jamesnk)

If that isn't possible then can I get you to get some diagnostics. I can create a PR to your repository with changes to enable detailed logging in the client. You can send attached the results here.

[jeffijoe]

Thank you, James. I've sent you an email with the endpoint.

[JamesNK]

I reproduced your error and looked at detailed internal logs. The problem is the .NET client is sending the server pings to calculate the RTT, but the server closes the connection because it believes the client is malicious.

Logs: ping-goaway.txt

Important parts:

ProcessIncomingFramesAsync,Frame 15: StreamId=0; Type=GoAway; Flags=None; PayloadLength=22.
ReadGoAwayFrame,lastStreamId=1, errorCode=EnhanceYourCalm

What is happening is the server is sending a GOAWAY frame with an EnhanceYourCalm error code to the client right after the client sends a ping. The server is telling the client that it thinks the client is malicious.

Adding this code to the start of the client app disables RTT pings fixes the problem:

AppContext.SetSwitch("System.Net.SocketsHttpHandler.Http2FlowControl.DisableDynamicWindowSizing", true);

A bug fix needs to happen inside .NET to improve how pings are sent, but you can use that setting to work around the problem until a fix is publicly available.

[jeffijoe]

It sounds like this is related to #2358 then.

So the server sends a GOAWAY, but the client does not surface this information in the exception that is thrown; is that intentional, or is that one of the things that need to be fixed? And regarding the ping, is it because the .NET client sends excessive pings?

EDIT: Also found this, seems related: dotnet/runtime#62216

[JamesNK]

but the client does not surface this information in the exception that is thrown; is that intentional, or is that one of the things that need to be fixed?

Providing the real reason why a HTTP request failed is a bug and should be fixed. I passed feedback to the team. The ENHANCE_YOUR_CALM status code should appear in the exception message and shouldn't require low-level debugging.

And regarding the ping, is it because the .NET client sends excessive pings?

Yes. It fails after the 5th ping. Some servers are very conservative and see five pings as excessive. The algorithm used to send pings needs to be updated.

[jeffijoe]

The AppContext switch works perfectly, thank you!

Will this issue remain open until the underlying issues have been fixed? Or how will we be able to track when we can remove the switch?

[JamesNK]

It sounds like this is related to #2358 then.

Yes it is. Exactly the same.

Will this issue remain open until the underlying issues have been fixed? Or how will we be able to track when we can remove the switch?

The fix will happen in dotnet/runtime repository. I'll put a comment here to a new issue there when I close it.

[antonfirsov]

Regardless of the fact if this will be fixed in .NET or the GCP load balancer, we need to involve the owners of the load balancer in order to make the right decision.

I warmed up an old email thread about a similar issue, but the primary contact in the discussion seems to have left google, so I don't have strong hopes I can reach the people I need over there.

@jeffijoe could you please also raise a ticket against Google? IMO that would be the most efficient way to have this routed to the right people so we can chat.

[jeffijoe]

@antonfirsov we don't have a direct way to contact Google but we've opened a ticket with DoIT who's our support provider and linked them to this issue, they may be able to escalate this to Google.

[jeffijoe]

@antonfirsov our support rep forwarded the ticket to Google, this is what we got back:

I have reached out regarding this request and it is currently not possible to involve the product team as the issue lies on the .NET client side. The .NET team provided a workaround that the customer can use in their code in order to disable the pings while they (the .NET team) work on fixing the issue with the .NET client. Please note that changing the LB to accept this behavior will potentially introduce a vulnerability that can be used by any gRPC client software to generate attacks.
In conclusion, kindly proceed with the workaround proposed by the .NET team in order for traffic to not be considered as malicious by the LB.