TCP Connection reset after a single lost keepalive packet

[Lucaber]

What version of gRPC are you using?

1.54.0

What version of Go are you using (go version)?

go1.20.2 linux/amd64

What operating system (Linux, Windows, …) and version?

Linux 6.2

Bug Report

The configuration of TCP_USER_TIMEOUT to 20 seconds in #2307 and #5219 together with go default tcp keepalive interval of just 15 seconds results in a tcp connection being reset after a single lost keepalive packet.

Lost packets of a tcp connection are normally being re-transmitted after a short amount of time, well within the 20 seconds timeout. But tcp keepalive packets are not being re-transmitted (ACK segments that contain no data are not reliably transmitted by TCP). Therefore the timeout is reached after just a single lost packet.

Normally not re-transmitting tcp keepalive packets is fine as the connection is only reseted after TCP_KEEPCNT(default=9) lost keepalive packets.

Test

TCPDump of a test grpc connection (disabled keepalives on the client to reduce packet count, the same issue can be reproduced with default keepalives on both the server and client):

• After packet 199 I dropped all traffic TO port 8090

sudo iptables -I INPUT -p tcp --dport 8090 -j DROP

• Packet 200 gets received by the client and answered in packet 201
• Packet 201 is only visible in the tcpdump but does not get received by the server
• Packet 202 resets the connection by the server when the next keepalive would have been send

Increasing the TCP_USER_TIMEOUT to 50 seconds results in the connection only being reset after 3 lost keepalive packets.

		grpc.KeepaliveParams(
			keepalive.ServerParameters{
				Timeout: 50 * time.Second,
			},
		),

Note: Here "keepalive" refers to the grpc/http2 keepalive mechanism and timeout configuration, not the tcp keepalives.

[easwars]

Our keepalive implementation is based on proposals A8 and A9. Any changes here would need to made across grpc implementations. If you want to propose any changes to this spec, you can do on the grpc-proposal repository.

Lost TCP keepalive packets might not be re-transmitted. But gRPC keepalives use a HTTP/2 PING frame. Why are these getting lost and not retransmitted?

Another thing to note is that gRPC sets the TCP keepalive timeout only when gRPC keepalive Time is not configured. See the code here: https://github.com/grpc/grpc-go/blob/master/internal/transport/http2_client.go#L264. This is the keepalive configuration struct on the client side: https://pkg.go.dev/google.golang.org/grpc@v1.55.0/keepalive#ClientParameters.

Can you share you gRPC keepalive configuration on the client and server side? Thanks.

[Lucaber]

Other grpc implementations are most likely not affected by the same issue.

The Design Background of A8 states that tcp keepalives are disabled by default, but when enabled after an implementation-specific duration of inactivity (on the order of 1-2 hours, but sysadmin configurable).
This is probably based on RFC1122 This interval MUST be configurable and MUST default to no less than two hours.

With this high interval, tcp keepalives are likely never been send out because other keepalive mechanisms in grpc/http2 send data more regularly. These keealives are normal tcp packets and are been re-transmitted when a single packet drops.

But TCP connections on GO default to a very short keepalive interval of just 15 seconds. Therefore this issue is only present in the go grpc implementation.

I am using the default keepalive settings on both the server and client side. Thats why i think that this issue should at least be documented, or better "fixed" by setting more forgiving default settings.

[easwars]

Thanks for the explanation @Lucaber

Do you see the problem even in the case where you set both keepalive.ClientParameters.Time and keepalive.ClientParameters.Timeout on the client and keepalive.ServerParameters.Time on the server to non-default values?

And since you are seeing the TCP connection getting closed after a single lost keepalive frame, are you saying that TCP_KEEPCNT=9 is not taking effect?

[Lucaber]

Correct, TCP_KEEPCNT=9 is not taking effect.

This depends on the values configured. The client and server configuration can mostly be viewed independently as the same issue is happening in both directions.

Setting the Timeout value controls how many keepalive packets can be dropped. Setting this value to 50 seconds, allows 2 lost packets. The connection gets reset after the third dropped packet.

Setting the Time value

• to time.Duration(math.MaxInt64) disables the configuration of TCP_USER_TIMEOUT and solves the problem. Without a TCP_USER_TIMEOUT, the connecting now times-out only after 9(TCP_KEEPCNT)*15 seconds
• to a value higher than 15 seconds still enables the TCP keepalive mechanism and the issue persists.
• to a value lower than 15 seconds effectively disables tcp keepalives, because the connection never goes idle. (At least when a grpc call is running and the connection is being used). But the active connection results in an ENHANCE_YOUR_CALM error after a few grpc-keepalives. Im not sure where i can configure this behavior.

Another way to fix this problem is disabling (or setting to 2 hours) the tcp keepalive interval. This can only be done be the user because grpc-go does not initialize the tcp socket:

var lc net.ListenConfig
lc.KeepAlive = -1
listener, err := lc.Listen(context.Background(), "tcp", "0.0.0.0:8090")

grpc.WithContextDialer(func(ctx context.Context, s string) (net.Conn, error) {
	var d net.Dialer
	d.KeepAlive = -1
	return d.Dial("tcp", s)
}),

[easwars]

But the active connection results in an ENHANCE_YOUR_CALM error after a few grpc-keepalives. Im not sure where i can configure this behavior.

This can be controlled by setting the MinTime field of server enforcement policy defined here: https://pkg.go.dev/google.golang.org/grpc@v1.55.0/keepalive#EnforcementPolicy. But you shouldn't have to set the value to such a low one to get around this problem.

If the client sets keepalive.ClientParameters.Time to whatever value (except infinity) and set keepalive.ClientParameters.Timeout to a value greater than 9*15s, that would ensure correct keepalive behavior, wouldn't it?

Apart from documenting this, do you have any other suggestions for handling this issue? Thanks.