TaskScheduler.UnobservedTaskException catches quic exceptions with kestrel combined with grpc + http3

[xeropresence]

Description

I've got a grpc service that has been running for sometime using just http2. I recently enabled http3 to see if there would be any performance benefit or improvement in connectivity some users were facing. I noticed almost immediately some exceptions were being caught where previously there weren't any.

Three different exceptions, all slightly different seem to be thrown

2025-04-01T15:54:54.8403559-07:00	CRIT	[LogicBase]	[0]	System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Connection aborted by peer (256).)
 ---> System.Net.Quic.QuicException: Connection aborted by peer (256).
   at System.Net.Quic.QuicConnection.HandleEventShutdownInitiatedByPeer(_SHUTDOWN_INITIATED_BY_PEER_e__Struct& data)
   at System.Net.Quic.QuicConnection.HandleConnectionEvent(QUIC_CONNECTION_EVENT& connectionEvent)
   at System.Net.Quic.QuicConnection.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_CONNECTION_EVENT* connectionEvent)
--- End of stack trace from previous location ---
   at System.Net.Quic.QuicConnection.AcceptInboundStreamAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Internal.QuicConnectionContext.AcceptAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   
  
   2025-04-01T15:54:54.8406298-07:00	CRIT	[LogicBase]	[0] System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Connection aborted by peer (256).)
 ---> System.Net.Quic.QuicException: Connection aborted by peer (256).
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final)
   at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data)
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent)
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent)
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Internal.QuicStreamContext.DoReceiveAsync()
   --- End of inner exception stack trace ---
   
   
   2025-04-01T15:54:54.8410699-07:00	CRIT	[LogicBase]	[0]	System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Connection aborted by peer (256).)
 ---> Microsoft.AspNetCore.Connections.ConnectionResetException: Connection aborted by peer (256).
 ---> System.Net.Quic.QuicException: Connection aborted by peer (256).
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final)
   at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data)
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent)
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent)
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Internal.QuicStreamContext.DoReceiveAsync()
   --- End of inner exception stack trace ---
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.GetReadAsyncResult()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3ControlStream.HandleControlStream()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3ControlStream.ProcessRequestAsync[TContext](IHttpApplication`1 application)
   --- End of inner exception stack trace ---
   
   

There maybe more exceptions, but when it happens so many are generated its hard for me to tell.

Reproduction Steps

I'm not exactly sure what's happening on the clientside to produce the issue, but the serverside setup is almost exactly a basic copy of

https://protobuf-net.github.io/protobuf-net.Grpc/gettingstarted

with http3 enabled.

Expected behavior

Exceptions are handled in the layer they are being generated

Actual behavior

Exceptions are going uncaught

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

I'm currently using .net8, but I also tried upgrading to .net9 but the issue persisted.
Possibly related to #80111 ?

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[ManickaP]

This likely overlaps with the experiments in #105336.

In 9.0, I fixed the issues for the client side in #104894. This might need work in Kestrel and QUIC transport code, maybe even gRPC. But the first step will be cleaning unobserved exceptions from Kestrel.

Since this is not affecting running code, it has a lower priority (i.e. its uncomfortable, but doesn't crash your service).

[BrennanConroy]

The 3 stacks listed above all have Kestrel as the calling code that isn't catching the exception. I think we can move this issue to dotnet/aspnetcore.

[ManickaP]

Sure, I'll be glad to hand it over 😄
Just a few hints since the S.N.Quic Task are mostly backed by ManualResetValueTaskSourceCore and handled manually:

• the stream ReadAsync call stack is likely https://github.com/dotnet/aspnetcore/blob/eb05c23e1a2c63890abf4e0947ad32672a74a3f2/src/Servers/Kestrel/Transport.Quic/src/Internal/QuicStreamContext.cs#L226 ==> _stream.ReadsClosed needs to be observed
• the connection AcceptInboundStreamAsync got follow up fix in [QUIC] Observe exceptions from _connectionCloseTcs #112190, which is only in 10.0 preview.2 @xeropresence would you be able to test it out with this version?

[antonfirsov]

Is it expected that whenever a user touches WritesClosed and ReadsClosed they should also observe the exceptions?
Note that this is a somewhat arbitrary behavior since the Tasks exposed by those properties are lazily initialized. (If you never touch them you can forget they exist from the POV of unobserved exceptions.) Even SocketsHttpHandler HTTP/3 code suffers from the problem of not observing WritesClosed exceptions (this is why we see them in #105336).

Shouldn't we rather implement a counterpart of #112190 for those tasks? Gonna be tricky because of the lazy init, but I would be happy to give it a try. @ManickaP thoughts?

[ManickaP]

Is it expected that whenever a user touches WritesClosed and ReadsClosed they should also observe the exceptions?|

Yes.

Shouldn't we rather implement a counterpart of #112190 for those tasks?

That would be actually really nice, if it's possible and has a reasonable solution. But I wouldn't sweat it if it proves to be too complicated. It's not super critical.

[xeropresence]

Sure, I'll be glad to hand it over 😄 Just a few hints since the S.N.Quic Task are mostly backed by ManualResetValueTaskSourceCore and handled manually:

• the stream ReadAsync call stack is likely https://github.com/dotnet/aspnetcore/blob/eb05c23e1a2c63890abf4e0947ad32672a74a3f2/src/Servers/Kestrel/Transport.Quic/src/Internal/QuicStreamContext.cs#L226 ==> _stream.ReadsClosed needs to be observed
• the connection AcceptInboundStreamAsync got follow up fix in [QUIC] Observe exceptions from _connectionCloseTcs #112190, which is only in 10.0 preview.2 @xeropresence would you be able to test it out with this version?

I can try to get around to it this weekend.

I noticed another exception, but this one wasn't being caught in the task.unobservedtaskexception, is this related to the exceptions present in this existing issue, should I create a new issue, or is this intended behavior.

crit: Microsoft.AspNetCore.Server.Kestrel[0]
      Unexpected exception in HttpConnection.ProcessRequestsAsync.
      System.Net.Quic.QuicException: An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER
         at System.Net.Quic.ThrowHelper.ThrowMsQuicException(Int32 status, String message)
         at System.Net.Quic.QuicStream.Abort(QuicAbortDirection abortDirection, Int64 errorCode)
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Internal.QuicStreamContext.Abort(ConnectionAbortedException abortReason)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3FrameWriter.Abort(ConnectionAbortedException error)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3ControlStream.Abort(ConnectionAbortedException abortReason, Http3ErrorCode errorCode)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.ProcessRequestsAsync[TContext](IHttpApplication`1 httpApplication)

[ManickaP]

The new exception actually looks like dotnet/aspnetcore#45105, I root caused it in dotnet/aspnetcore#45105 (comment). I'd suggest to comment on that aspnet issue that you're experiencing it as well.