[HttpStress] `H3_REQUEST_REJECTED` in Get / Get Slow

[antonfirsov]

This issue was not fixed by the aspnetcore update. Seems to occur on Windows. Since the inner exception is QuicException: Stream aborted by peer (267). it might be related to #100534.

System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.Net.Http.HttpProtocolException: The HTTP/3 server reset the stream. HTTP/3 error code 'H3_REQUEST_REJECTED' (0x10b). (HttpProtocolError)
 ---> System.Net.Quic.QuicException: Stream aborted by peer (267).
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 177
   at System.Net.Quic.QuicStream.HandleEventPeerSendAborted(_PEER_SEND_ABORTED_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 607
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 662
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* stream, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 694
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 251
   at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 323
   at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1255
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.HandleReadResponseContentException(Exception ex, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1294
   at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1275
   at System.IO.Stream.<CopyToAsync>g__Core|30_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Stream.cs:line 106
   at System.Net.Http.HttpConnectionResponseContent.<SerializeToStreamAsync>g__Impl|6_0(Stream stream, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionResponseContent.cs:line 61
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 538
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 544
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 542
   at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in C:\app\ClientOperations.cs:line 95
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_0>d.MoveNext() in C:\app\ClientOperations.cs:line 193

Examples:

• https://dev.azure.com/dnceng-public/public/_build/results?buildId=847388&view=logs&j=0bc77094-9fcd-5c38-f6e4-27d2ae131589&t=4fd93899-d50b-5dc1-f9eb-66ab1b883c22
• https://dev.azure.com/dnceng-public/public/_build/results?buildId=848435&view=logs&j=0bc77094-9fcd-5c38-f6e4-27d2ae131589&t=4fd93899-d50b-5dc1-f9eb-66ab1b883c22

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[rzikm]

The issue is likely in ASP.NET Core H3 implementation, I tried to debug the issue via #109152, and scarcely got any hits, but the few that we managed to catch was that the receive headers timeout was getting miscalculated and considered elapsed very early (absolutely earlier than the default 30s timeout),

We should monitor the issue, but is of low priority given that so far we do not have any customer reports of its occurence

[karelz]

Triage: Happens rarely, pushing to 11.