grpc-dotnet Interop tests highly flaky on grpc/grpc repository

[jtattermusch]

This started flaking a few days ago.
Once of the first occurrences is
https://source.cloud.google.com/results/invocations/29f7f9fc-47d6-4846-8ec4-368cbb688436/targets/github%2Fgrpc%2Finterop_test/tests
and it's been ~50% flaky since.

Likely caused by a recent change in grpc-dotnet?

[JamesNK]

Error is coming from underlying HttpClient logic. Most likely culprit is updating interop tests to .NET 6 - b9cb880

[JamesNK]

I couldn't reproduce this when calling the sandbox server from my local machine. I put server streaming and large unary in a loop and ran them both 100 times without error.

[JamesNK]

Changed the interop tests to use net5.0. If tests consistently pass then that confirms the issue is a regression from 5.0 to 6.0.

[jtattermusch]

Looks like after merging #1514, the failure on the CI went away, so this really about the net5.0 vs net6.0 difference.

I also tried to reproduce with net6.0 locally and also wasn't able to do so.

[jtattermusch]

Since I wasn't able to reproduce locally, I tried to find the CI log where the test is failing with the extra diagnostic info (#1513) but before we reverted back to net5.0 #1514.

Looks like this is the log:
https://source.cloud.google.com/results/invocations/e229d041-9371-4e16-99bb-592991ebf10b/targets/github%2Fgrpc%2Finterop_test/tests;group=tests;test=cloud_to_prod:aspnetcore:default:large_unary:tls;row=3

Is that helpful?

[JamesNK]

I believe the issue is SocketsHttpHandler is sending too many pings, and the server is closing the connection. This is for a feature to measure latency between the client and server, and dynamically adjust window sizes. It is a new feature in .NET 6 which is why it only showed up when interop tests moved from 5 to 6.

See dotnet/runtime#62216

This should get fixed in a .NET 6 servicing release sometime early next year. Until then the interop tests can continue to use .NET 5.