Multiple calls of QuicConnection.CloseAsync lead to Debug.Assert

[wfurt]

I bump to this while looking at #56115
On failure, Kestrel calls

ShutdownAsync: called on [conn][0x7FEA58000B60] with NONE and 256
   at System.Net.Quic.Implementations.MsQuic.MsQuicConnection.ShutdownAsync(QUIC_CONNECTION_SHUTDOWN_FLAGS Flags, Int64 ErrorCode) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Quic/src/System/Net/Quic/Implementations/MsQuic/MsQuicConnection.cs:line 659
   at System.Net.Quic.Implementations.MsQuic.MsQuicConnection.CloseAsync(Int64 errorCode, CancellationToken cancellationToken) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Quic/src/System/Net/Quic/Implementations/MsQuic/MsQuicConnection.cs:line 850
   at System.Net.Quic.QuicConnection.CloseAsync(Int64 errorCode, CancellationToken cancellationToken) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicConnection.cs:line 106
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Internal.QuicConnectionContext.Abort(ConnectionAbortedException abortReason) in /home/furt/github/aspnetcore/src/Servers/Kestrel/Transport.Quic/src/Internal/QuicConnectionContext.cs:line 73
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Internal.QuicConnectionContext.Abort() in /home/furt/github/aspnetcore/src/Servers/Kestrel/Transport.Quic/src/Internal/QuicConnectionContext.cs:line 67
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Connection.StopProcessingNextRequest(Boolean serverInitiated) in /home/furt/github/aspnetcore/src/Servers/Kestrel/Core/src/Internal/Http3/Http3Connection.cs:line 112
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Connection.StopProcessingNextRequest() in /home/furt/github/aspnetcore/src/Servers/Kestrel/Core/src/Internal/Http3/Http3Connection.cs:line 87
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.StopProcessingNextRequest() in /home/furt/github/aspnetcore/src/Servers/Kestrel/Core/src/Internal/HttpConnection.cs:line 145
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.<>c__12`1.<ProcessRequestsAsync>b__12_1(Object state) in /home/furt/github/aspnetcore/src/Servers/Kestrel/Core/src/Internal/HttpConnection.cs:line 104
   at System.Threading.CancellationTokenSource.Invoke(Delegate d, Object state, CancellationTokenSource source) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 885
   at System.Threading.CancellationTokenSource.CallbackNode.<>c.<ExecuteCallback>b__9_0(Object s) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 1127
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 183
   at System.Threading.CancellationTokenSource.CallbackNode.ExecuteCallback() in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 1123
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 711
   at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean throwOnFirstException) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 625
   at System.Threading.CancellationTokenSource.Cancel(Boolean throwOnFirstException) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 250
   at System.Threading.CancellationTokenSource.Cancel() in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 221
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection.RequestClose() in /home/furt/github/aspnetcore/src/Servers/Kestrel/Core/src/Internal/Infrastructure/KestrelConnection.cs:line 158
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportConnectionManager.CloseAllConnectionsAsync(CancellationToken token) in /home/furt/github/aspnetcore/src/Servers/Kestrel/Core/src/Internal/Infrastructure/TransportConnectionManager.cs:line 67

[conn][0x7FEA58000B60] Connection received event SHUTDOWN_COMPLETE 8/4/2021 10:54:09 AM

that will close the connection. And than again

ShutdownAsync: called on [conn][0x7FEA58000B60] with NONE and 256
   at System.Net.Quic.Implementations.MsQuic.MsQuicConnection.ShutdownAsync(QUIC_CONNECTION_SHUTDOWN_FLAGS Flags, Int64 ErrorCode) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Quic/src/System/Net/Quic/Implementations/MsQuic/MsQuicConnection.cs:line 659
   at System.Net.Quic.Implementations.MsQuic.MsQuicConnection.CloseAsync(Int64 errorCode, CancellationToken cancellationToken) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Quic/src/System/Net/Quic/Implementations/MsQuic/MsQuicConnection.cs:line 850
   at System.Net.Quic.QuicConnection.CloseAsync(Int64 errorCode, CancellationToken cancellationToken) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicConnection.cs:line 106
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Internal.QuicConnectionContext.Abort(ConnectionAbortedException abortReason) in /home/furt/github/aspnetcore/src/Servers/Kestrel/Transport.Quic/src/Internal/QuicConnectionContext.cs:line 73
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportConnectionManager.AbortAllConnectionsAsync() in /home/furt/github/aspnetcore/src/Servers/Kestrel/Core/src/Internal/Infrastructure/TransportConnectionManager.cs:line 84
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 38
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportConnectionManager.AbortAllConnectionsAsync() in Microsoft.AspNetCore.Server.Kestrel.Core.dll:token 0x60006e1+0x1f
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.<>c__DisplayClass14_0.<<StopTransportsAsync>g__StopTransportConnection|0>d.MoveNext() in /home/furt/github/aspnetcore/src/Servers/Kestrel/Core/src/Internal/Infrastructure/TransportManager.cs:line 114
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 38
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.<>c__DisplayClass14_0.<StopTransportsAsync>g__StopTransportConnection|0(ActiveTransport transport) in Microsoft.AspNetCore.Server.Kestrel.Core.dll:token 0x60010b8+0x26
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.StopTransportsAsync(List`1 transportsToStop, CancellationToken cancellationToken) in /home/furt/github/aspnetcore/src/Servers/Kestrel/Core/src/Internal/Infrastructure/TransportManager.cs:line 123

This may be wrong pattern but I feel we should eater act in some more predictable way: either ignore the second call and throw InvalidOperationException. It either case it would be nice to add some tests for this.

cc: @JamesNK

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I bump to this while looking at #56115
On failure, Kestrel calls

ShutdownAsync: called on [conn][0x7FEA58000B60] with NONE and 256
   at System.Net.Quic.Implementations.MsQuic.MsQuicConnection.ShutdownAsync(QUIC_CONNECTION_SHUTDOWN_FLAGS Flags, Int64 ErrorCode) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Quic/src/System/Net/Quic/Implementations/MsQuic/MsQuicConnection.cs:line 659
   at System.Net.Quic.Implementations.MsQuic.MsQuicConnection.CloseAsync(Int64 errorCode, CancellationToken cancellationToken) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Quic/src/System/Net/Quic/Implementations/MsQuic/MsQuicConnection.cs:line 850
   at System.Net.Quic.QuicConnection.CloseAsync(Int64 errorCode, CancellationToken cancellationToken) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicConnection.cs:line 106
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Internal.QuicConnectionContext.Abort(ConnectionAbortedException abortReason) in /home/furt/github/aspnetcore/src/Servers/Kestrel/Transport.Quic/src/Internal/QuicConnectionContext.cs:line 73
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Internal.QuicConnectionContext.Abort() in /home/furt/github/aspnetcore/src/Servers/Kestrel/Transport.Quic/src/Internal/QuicConnectionContext.cs:line 67
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Connection.StopProcessingNextRequest(Boolean serverInitiated) in /home/furt/github/aspnetcore/src/Servers/Kestrel/Core/src/Internal/Http3/Http3Connection.cs:line 112
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Connection.StopProcessingNextRequest() in /home/furt/github/aspnetcore/src/Servers/Kestrel/Core/src/Internal/Http3/Http3Connection.cs:line 87
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.StopProcessingNextRequest() in /home/furt/github/aspnetcore/src/Servers/Kestrel/Core/src/Internal/HttpConnection.cs:line 145
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.<>c__12`1.<ProcessRequestsAsync>b__12_1(Object state) in /home/furt/github/aspnetcore/src/Servers/Kestrel/Core/src/Internal/HttpConnection.cs:line 104
   at System.Threading.CancellationTokenSource.Invoke(Delegate d, Object state, CancellationTokenSource source) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 885
   at System.Threading.CancellationTokenSource.CallbackNode.<>c.<ExecuteCallback>b__9_0(Object s) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 1127
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 183
   at System.Threading.CancellationTokenSource.CallbackNode.ExecuteCallback() in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 1123
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 711
   at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean throwOnFirstException) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 625
   at System.Threading.CancellationTokenSource.Cancel(Boolean throwOnFirstException) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 250
   at System.Threading.CancellationTokenSource.Cancel() in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 221
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection.RequestClose() in /home/furt/github/aspnetcore/src/Servers/Kestrel/Core/src/Internal/Infrastructure/KestrelConnection.cs:line 158
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportConnectionManager.CloseAllConnectionsAsync(CancellationToken token) in /home/furt/github/aspnetcore/src/Servers/Kestrel/Core/src/Internal/Infrastructure/TransportConnectionManager.cs:line 67

[conn][0x7FEA58000B60] Connection received event SHUTDOWN_COMPLETE 8/4/2021 10:54:09 AM

that will close the connection. And than again

ShutdownAsync: called on [conn][0x7FEA58000B60] with NONE and 256
   at System.Net.Quic.Implementations.MsQuic.MsQuicConnection.ShutdownAsync(QUIC_CONNECTION_SHUTDOWN_FLAGS Flags, Int64 ErrorCode) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Quic/src/System/Net/Quic/Implementations/MsQuic/MsQuicConnection.cs:line 659
   at System.Net.Quic.Implementations.MsQuic.MsQuicConnection.CloseAsync(Int64 errorCode, CancellationToken cancellationToken) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Quic/src/System/Net/Quic/Implementations/MsQuic/MsQuicConnection.cs:line 850
   at System.Net.Quic.QuicConnection.CloseAsync(Int64 errorCode, CancellationToken cancellationToken) in /home/furt/github/wfurt-runtime/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicConnection.cs:line 106
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Internal.QuicConnectionContext.Abort(ConnectionAbortedException abortReason) in /home/furt/github/aspnetcore/src/Servers/Kestrel/Transport.Quic/src/Internal/QuicConnectionContext.cs:line 73
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportConnectionManager.AbortAllConnectionsAsync() in /home/furt/github/aspnetcore/src/Servers/Kestrel/Core/src/Internal/Infrastructure/TransportConnectionManager.cs:line 84
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 38
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportConnectionManager.AbortAllConnectionsAsync() in Microsoft.AspNetCore.Server.Kestrel.Core.dll:token 0x60006e1+0x1f
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.<>c__DisplayClass14_0.<<StopTransportsAsync>g__StopTransportConnection|0>d.MoveNext() in /home/furt/github/aspnetcore/src/Servers/Kestrel/Core/src/Internal/Infrastructure/TransportManager.cs:line 114
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /home/furt/github/wfurt-runtime/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 38
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.<>c__DisplayClass14_0.<StopTransportsAsync>g__StopTransportConnection|0(ActiveTransport transport) in Microsoft.AspNetCore.Server.Kestrel.Core.dll:token 0x60010b8+0x26
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.StopTransportsAsync(List`1 transportsToStop, CancellationToken cancellationToken) in /home/furt/github/aspnetcore/src/Servers/Kestrel/Core/src/Internal/Infrastructure/TransportManager.cs:line 123

This may be wrong pattern but I feel we should eater act in some more predictable way: either ignore the second call and throw InvalidOperationException. It either case it would be nice to add some tests for this.

cc: @JamesNK

Author:	wfurt
Assignees:	-
Labels:	area-System.Net.Quic
Milestone:	-

[geoffkizer]

Yeah, seems like we should ignore a call to CloseAsync if we are already closed.

These are not concurrent calls, are they? We should probably handle that too. Would Kestrel ever call CloseAsync from multiple threads at the same time? Seems like it very well might.

[wfurt]

It does not seems to call it in parallel - at least not in case I'm looking at. I was just surprised to see assert when I run asp tests with debug runtime. So far no other sign of specific badness.

[geoffkizer]

Running ASP.NET tests with a debug runtime seems like a very good idea.

[wfurt]

I would like to to that also with out stress and debug build of msquic. @nibanks mentioned they find interesting stuff that way. I have basic packages changes in but the whole loop still did not close.

[ManickaP]

Triage: we should investigate if there's no serious underlying issue and clean up Kestrel runs. If it's easy we should fix this.