Description
I've seen it (rarely) happening on GetAsync_LargeHeader_Success, but it is most probably a generic HTTP/3 test issue (I suspect LoopbackServer)
System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_HttpClientHandlerTest_Headers_Http3_MsQuic.GetAsync_LargeHeader_Success(headerName: "Content-Security-Policy", headerValueLength: 4618) [FAIL]
System.Net.Http.HttpRequestException : An error occurred while sending the request.
---- System.Net.Quic.QuicOperationAbortedException : Operation aborted.
Stack Trace:
C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(290,0): at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(298,0): at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3Connection.cs(214,0): at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(795,0): at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(871,0): at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(901,0): at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(920,0): at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\RedirectHandler.cs(30,0): at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\HttpClient.cs(527,0): at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.Headers.cs(223,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Headers.<>c__DisplayClass8_0.<<GetAsync_LargeHeader_Success>b__1>d.MoveNext()
--- End of stack trace from previous location ---
C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Threading\Tasks\TaskTimeoutExtensions.cs(63,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Threading\Tasks\TaskTimeoutExtensions.cs(82,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Net\Http\GenericLoopbackServer.cs(38,0): at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Net\Http\Http3LoopbackServer.cs(100,0): at System.Net.Test.Common.Http3LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout, GenericLoopbackOptions options)
C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.Headers.cs(220,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Headers.GetAsync_LargeHeader_Success(String headerName, Int32 headerValueLength)
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(1386,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.HandleEventConnectionClose(State state)
C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(965,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.HandleEventShutdownComplete(State state, StreamEvent& evt)
C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(813,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.HandleEvent(State state, StreamEvent& evt)
C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(772,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.NativeCallbackHandler(IntPtr stream, IntPtr context, StreamEvent& streamEvent)
--- End of stack trace from previous location ---
C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\Internal\ResettableCompletionSource.cs(57,0): at System.Net.Quic.Implementations.MsQuic.Internal.ResettableCompletionSource`1.GetResult(Int16 token)
C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(796,0): at System.Net.Http.Http3RequestStream.ReadHeadersAsync(Int64 headersLength, CancellationToken cancellationToken)
C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(322,0): at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken)
C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(201,0): at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
Callstack shows MsQuicStream.HandleEventConnectionClose which means connection was closed while stream was in flight reading the response.
This brings me to thing we've discussed today:
Important thing I've remembered: it is not enough to wait until SHUTDOWN_COMPLETE event on stream to be sure it is safe to close the connection: it means shutdown is acked on transport level, but on app level peer might still be in the middle of reading. Long time ago we had a huge email thread on this topic with Nick and aligned that such synchronization should be done on app level. That's why in RunClientServer test method there was additional synchronization done via semaphores.
This exact issue manifests itself in our HTTP/3 tests, so we need to add some kind of app-level synchro there (or think of another solution).