[HTTP/3] map `QuicException` to `OCE` in case of a cancellation

[alexandrehtrb]

Description

Hello,

I am writing a HTTP stress tool using C# and HttpClient, and in some rare cases, a QuicException arises when using HTTP/3.

Reproduction Steps

Server: dotnet new webapi with HTTP/3 enabled, using default GET /weatherforecast endpoint.

Console application to run stress test below. I had to dotnet run 10 times until the Exception showed up, because it doesn't always happen. (long code warning)

using System.Net;
using System.Threading.Channels;

namespace ExampleParallelHttp3;

public static class Program
{
    public static async Task Main(string[] args)
    {
        Console.WriteLine("Hello, World!");
        var hc = MakeHttpClient();
        var channelReader = StartRepetition(hc, 25000, 20, MakeConsoleCancellationToken());
        int i = 1;
        await foreach (var result in channelReader.ReadAllAsync())
        {
            if (result.Item3 is null)
            {
                Console.WriteLine($"i = {i++}, status code = {result.Item1}");
            }
            else
            {
                Console.WriteLine($"i = {i++}, exception = \n{result.Item3}\n");
                break;
            }
        }
        Console.WriteLine("Finished");
    }

    private static CancellationToken MakeConsoleCancellationToken()
    {
        // Add this to your C# console app's Main method to give yourself
        // a CancellationToken that is canceled when the user hits Ctrl+C.
        var cts = new CancellationTokenSource();
        Console.CancelKeyPress += (s, e) =>
        {
            Console.WriteLine("Canceling...");
            cts.Cancel();
            e.Cancel = true;
        };
        return cts.Token;
    }

    private static HttpClient MakeHttpClient()
    {
        SocketsHttpHandler httpHandler = new()
        {
            PooledConnectionLifetime = TimeSpan.FromMinutes(20),
            AutomaticDecompression = DecompressionMethods.All
        };
        
        httpHandler.SslOptions.RemoteCertificateValidationCallback =
            (sender, certificate, chain, sslPolicyErrors) => true;

        HttpClient httpClient = new(httpHandler, disposeHandler: false)
        {
            Timeout = TimeSpan.FromMinutes(5)
        };
        return httpClient;
    }

    public static ChannelReader<(HttpStatusCode?, string?, Exception?)> StartRepetition(HttpClient hc, int numReps, int maxDop, CancellationToken cancellationToken)
    {
        var channel = CreateChannel(numReps, maxDop);

        Task.Factory.StartNew(async () =>
        {
            try
            {
                await ExecuteParallelRequestsAsync(hc, channel.Writer, numReps, maxDop, cancellationToken);
            }
            catch (TaskCanceledException) { }
            catch (OperationCanceledException) { }
            catch (Exception ex)
            {
                (HttpStatusCode?, string?, Exception?) result = (null, null, ex);
                await channel.Writer.WriteAsync(result);
            }
            finally { channel.Writer.Complete(); }
        }, TaskCreationOptions.LongRunning);

        return channel.Reader;
    }

    private static async Task ExecuteParallelRequestsAsync(HttpClient hc, ChannelWriter<(HttpStatusCode?, string?, Exception?)> channelWriter, int numReps, int maxDop, CancellationToken cancellationToken)
    {
        ParallelOptions options = new();
        options.MaxDegreeOfParallelism = maxDop;
        options.CancellationToken = cancellationToken;

        await Parallel.ForAsync(0, numReps, options, async (i, ct) =>
        {
            try
            {
                HttpRequestMessage req = new(HttpMethod.Get, "https://localhost:5001/weatherforecast")
                {
                    Version = new(3, 0),
                    VersionPolicy = HttpVersionPolicy.RequestVersionExact
                };
                var res = await hc.SendAsync(req, ct);

                await channelWriter.WriteAsync((res.StatusCode, await res.Content.ReadAsStringAsync(ct), null), ct);
            }
            catch (Exception ex)
            {
                await channelWriter.WriteAsync((null, null, ex), ct);                
            }
        });
    }

    private static Channel<(HttpStatusCode?, string?, Exception?)> CreateChannel(int numReps, int maxDop)
    {
        BoundedChannelOptions channelOpts = new(numReps)
        {
            SingleReader = true,
            SingleWriter = maxDop == 1
        };
        var channel = Channel.CreateBounded<(HttpStatusCode?, string?, Exception?)>(channelOpts);
        return channel;
    }
}

Expected behavior

Exception shouldn't happen, I guess

Actual behavior

Error log in server:

crit: Microsoft.AspNetCore.Server.Kestrel[0]
      Unexpected exception in HttpConnection.ProcessRequestsAsync.
      System.Net.Quic.QuicException: An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER
         at System.Net.Quic.ThrowHelper.ThrowMsQuicException(Int32 status, String message)
         at System.Net.Quic.QuicStream.Abort(QuicAbortDirection abortDirection, Int64 errorCode)
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Internal.QuicStreamContext.Abort(ConnectionAbortedException abortReason)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3FrameWriter.Abort(ConnectionAbortedException error)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Stream.AbortCore(Exception exception, Http3ErrorCode errorCode)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.ProcessRequestsAsync[TContext](IHttpApplication`1 httpApplication)

Error log in console application:

i = 4, exception = 
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Quic.QuicException: Operation aborted.
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final)
   at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data)
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent)
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent)
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Http.Http3RequestStream.FlushSendBufferAsync(Boolean endStream, CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken)
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at ExampleParallelHttp3.Program.<>c__DisplayClass4_0.<<ExecuteParallelRequestsAsync>b__0>d.MoveNext() in /home/alexandre/Projetos/ExampleParallelHttp3/Program.cs:line 99

Regression?

No response

Known Workarounds

No response

Configuration

~> dotnet --info
.NET SDK:
 Version:           8.0.101
 Commit:            6eceda187b
 Workload version:  8.0.100-manifests.69afb982

Ambiente de runtime:
 OS Name:     debian
 OS Version:  12
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /usr/share/dotnet/sdk/8.0.101/

Host:
  Version:      8.0.1
  Architecture: x64
  Commit:       bf5e279d92

libmsquic version: 2.3.1

Other information

I did not test this on Windows

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Hello,

I am writing a HTTP stress tool using C# and HttpClient, and in some rare cases, a QuicException arises when using HTTP/3.

Reproduction Steps

Server: dotnet new webapi with HTTP/3 enabled, using default GET /weatherforecast endpoint.

Console application to run stress test below. I had to dotnet run 10 times until the Exception showed up, because it doesn't always happen. (long code warning)

using System.Net;
using System.Threading.Channels;

namespace ExampleParallelHttp3;

public static class Program
{
    public static async Task Main(string[] args)
    {
        Console.WriteLine("Hello, World!");
        var hc = MakeHttpClient();
        var channelReader = StartRepetition(hc, 25000, 20, MakeConsoleCancellationToken());
        int i = 1;
        await foreach (var result in channelReader.ReadAllAsync())
        {
            if (result.Item3 is null)
            {
                Console.WriteLine($"i = {i++}, status code = {result.Item1}");
            }
            else
            {
                Console.WriteLine($"i = {i++}, exception = \n{result.Item3}\n");
                break;
            }
        }
        Console.WriteLine("Finished");
    }

    private static CancellationToken MakeConsoleCancellationToken()
    {
        // Add this to your C# console app's Main method to give yourself
        // a CancellationToken that is canceled when the user hits Ctrl+C.
        var cts = new CancellationTokenSource();
        Console.CancelKeyPress += (s, e) =>
        {
            Console.WriteLine("Canceling...");
            cts.Cancel();
            e.Cancel = true;
        };
        return cts.Token;
    }

    private static HttpClient MakeHttpClient()
    {
        SocketsHttpHandler httpHandler = new()
        {
            PooledConnectionLifetime = TimeSpan.FromMinutes(20),
            AutomaticDecompression = DecompressionMethods.All
        };
        
        httpHandler.SslOptions.RemoteCertificateValidationCallback =
            (sender, certificate, chain, sslPolicyErrors) => true;

        HttpClient httpClient = new(httpHandler, disposeHandler: false)
        {
            Timeout = TimeSpan.FromMinutes(5)
        };
        return httpClient;
    }

    public static ChannelReader<(HttpStatusCode?, string?, Exception?)> StartRepetition(HttpClient hc, int numReps, int maxDop, CancellationToken cancellationToken)
    {
        var channel = CreateChannel(numReps, maxDop);

        Task.Factory.StartNew(async () =>
        {
            try
            {
                await ExecuteParallelRequestsAsync(hc, channel.Writer, numReps, maxDop, cancellationToken);
            }
            catch (TaskCanceledException) { }
            catch (OperationCanceledException) { }
            catch (Exception ex)
            {
                (HttpStatusCode?, string?, Exception?) result = (null, null, ex);
                await channel.Writer.WriteAsync(result);
            }
            finally { channel.Writer.Complete(); }
        }, TaskCreationOptions.LongRunning);

        return channel.Reader;
    }

    private static async Task ExecuteParallelRequestsAsync(HttpClient hc, ChannelWriter<(HttpStatusCode?, string?, Exception?)> channelWriter, int numReps, int maxDop, CancellationToken cancellationToken)
    {
        ParallelOptions options = new();
        options.MaxDegreeOfParallelism = maxDop;
        options.CancellationToken = cancellationToken;

        await Parallel.ForAsync(0, numReps, options, async (i, ct) =>
        {
            try
            {
                HttpRequestMessage req = new(HttpMethod.Get, "https://localhost:5001/weatherforecast")
                {
                    Version = new(3, 0),
                    VersionPolicy = HttpVersionPolicy.RequestVersionExact
                };
                var res = await hc.SendAsync(req, ct);

                await channelWriter.WriteAsync((res.StatusCode, await res.Content.ReadAsStringAsync(ct), null), ct);
            }
            catch (Exception ex)
            {
                await channelWriter.WriteAsync((null, null, ex), ct);                
            }
        });
    }

    private static Channel<(HttpStatusCode?, string?, Exception?)> CreateChannel(int numReps, int maxDop)
    {
        BoundedChannelOptions channelOpts = new(numReps)
        {
            SingleReader = true,
            SingleWriter = maxDop == 1
        };
        var channel = Channel.CreateBounded<(HttpStatusCode?, string?, Exception?)>(channelOpts);
        return channel;
    }
}

Expected behavior

Exception shouldn't happen, I guess

Actual behavior

Error log in server:

crit: Microsoft.AspNetCore.Server.Kestrel[0]
      Unexpected exception in HttpConnection.ProcessRequestsAsync.
      System.Net.Quic.QuicException: An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER
         at System.Net.Quic.ThrowHelper.ThrowMsQuicException(Int32 status, String message)
         at System.Net.Quic.QuicStream.Abort(QuicAbortDirection abortDirection, Int64 errorCode)
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Internal.QuicStreamContext.Abort(ConnectionAbortedException abortReason)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3FrameWriter.Abort(ConnectionAbortedException error)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Stream.AbortCore(Exception exception, Http3ErrorCode errorCode)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.ProcessRequestsAsync[TContext](IHttpApplication`1 httpApplication)

Error log in console application:

i = 4, exception = 
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Quic.QuicException: Operation aborted.
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final)
   at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data)
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent)
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent)
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Http.Http3RequestStream.FlushSendBufferAsync(Boolean endStream, CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken)
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at ExampleParallelHttp3.Program.<>c__DisplayClass4_0.<<ExecuteParallelRequestsAsync>b__0>d.MoveNext() in /home/alexandre/Projetos/ExampleParallelHttp3/Program.cs:line 99

Regression?

No response

Known Workarounds

No response

Configuration

~> dotnet --info
.NET SDK:
 Version:           8.0.101
 Commit:            6eceda187b
 Workload version:  8.0.100-manifests.69afb982

Ambiente de runtime:
 OS Name:     debian
 OS Version:  12
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /usr/share/dotnet/sdk/8.0.101/

Host:
  Version:      8.0.1
  Architecture: x64
  Commit:       bf5e279d92

libmsquic version: 2.3.1

Other information

I did not test this on Windows

Author:	alexandrehtrb
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Hello,

I am writing a HTTP stress tool using C# and HttpClient, and in some rare cases, a QuicException arises when using HTTP/3.

Reproduction Steps

Server: dotnet new webapi with HTTP/3 enabled, using default GET /weatherforecast endpoint.

Console application to run stress test below. I had to dotnet run 10 times until the Exception showed up, because it doesn't always happen. (long code warning)

using System.Net;
using System.Threading.Channels;

namespace ExampleParallelHttp3;

public static class Program
{
    public static async Task Main(string[] args)
    {
        Console.WriteLine("Hello, World!");
        var hc = MakeHttpClient();
        var channelReader = StartRepetition(hc, 25000, 20, MakeConsoleCancellationToken());
        int i = 1;
        await foreach (var result in channelReader.ReadAllAsync())
        {
            if (result.Item3 is null)
            {
                Console.WriteLine($"i = {i++}, status code = {result.Item1}");
            }
            else
            {
                Console.WriteLine($"i = {i++}, exception = \n{result.Item3}\n");
                break;
            }
        }
        Console.WriteLine("Finished");
    }

    private static CancellationToken MakeConsoleCancellationToken()
    {
        // Add this to your C# console app's Main method to give yourself
        // a CancellationToken that is canceled when the user hits Ctrl+C.
        var cts = new CancellationTokenSource();
        Console.CancelKeyPress += (s, e) =>
        {
            Console.WriteLine("Canceling...");
            cts.Cancel();
            e.Cancel = true;
        };
        return cts.Token;
    }

    private static HttpClient MakeHttpClient()
    {
        SocketsHttpHandler httpHandler = new()
        {
            PooledConnectionLifetime = TimeSpan.FromMinutes(20),
            AutomaticDecompression = DecompressionMethods.All
        };
        
        httpHandler.SslOptions.RemoteCertificateValidationCallback =
            (sender, certificate, chain, sslPolicyErrors) => true;

        HttpClient httpClient = new(httpHandler, disposeHandler: false)
        {
            Timeout = TimeSpan.FromMinutes(5)
        };
        return httpClient;
    }

    public static ChannelReader<(HttpStatusCode?, string?, Exception?)> StartRepetition(HttpClient hc, int numReps, int maxDop, CancellationToken cancellationToken)
    {
        var channel = CreateChannel(numReps, maxDop);

        Task.Factory.StartNew(async () =>
        {
            try
            {
                await ExecuteParallelRequestsAsync(hc, channel.Writer, numReps, maxDop, cancellationToken);
            }
            catch (TaskCanceledException) { }
            catch (OperationCanceledException) { }
            catch (Exception ex)
            {
                (HttpStatusCode?, string?, Exception?) result = (null, null, ex);
                await channel.Writer.WriteAsync(result);
            }
            finally { channel.Writer.Complete(); }
        }, TaskCreationOptions.LongRunning);

        return channel.Reader;
    }

    private static async Task ExecuteParallelRequestsAsync(HttpClient hc, ChannelWriter<(HttpStatusCode?, string?, Exception?)> channelWriter, int numReps, int maxDop, CancellationToken cancellationToken)
    {
        ParallelOptions options = new();
        options.MaxDegreeOfParallelism = maxDop;
        options.CancellationToken = cancellationToken;

        await Parallel.ForAsync(0, numReps, options, async (i, ct) =>
        {
            try
            {
                HttpRequestMessage req = new(HttpMethod.Get, "https://localhost:5001/weatherforecast")
                {
                    Version = new(3, 0),
                    VersionPolicy = HttpVersionPolicy.RequestVersionExact
                };
                var res = await hc.SendAsync(req, ct);

                await channelWriter.WriteAsync((res.StatusCode, await res.Content.ReadAsStringAsync(ct), null), ct);
            }
            catch (Exception ex)
            {
                await channelWriter.WriteAsync((null, null, ex), ct);                
            }
        });
    }

    private static Channel<(HttpStatusCode?, string?, Exception?)> CreateChannel(int numReps, int maxDop)
    {
        BoundedChannelOptions channelOpts = new(numReps)
        {
            SingleReader = true,
            SingleWriter = maxDop == 1
        };
        var channel = Channel.CreateBounded<(HttpStatusCode?, string?, Exception?)>(channelOpts);
        return channel;
    }
}

Expected behavior

Exception shouldn't happen, I guess

Actual behavior

Error log in server:

crit: Microsoft.AspNetCore.Server.Kestrel[0]
      Unexpected exception in HttpConnection.ProcessRequestsAsync.
      System.Net.Quic.QuicException: An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER
         at System.Net.Quic.ThrowHelper.ThrowMsQuicException(Int32 status, String message)
         at System.Net.Quic.QuicStream.Abort(QuicAbortDirection abortDirection, Int64 errorCode)
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Internal.QuicStreamContext.Abort(ConnectionAbortedException abortReason)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3FrameWriter.Abort(ConnectionAbortedException error)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Stream.AbortCore(Exception exception, Http3ErrorCode errorCode)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.ProcessRequestsAsync[TContext](IHttpApplication`1 httpApplication)

Error log in console application:

i = 4, exception = 
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Quic.QuicException: Operation aborted.
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final)
   at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data)
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent)
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent)
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Http.Http3RequestStream.FlushSendBufferAsync(Boolean endStream, CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken)
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at ExampleParallelHttp3.Program.<>c__DisplayClass4_0.<<ExecuteParallelRequestsAsync>b__0>d.MoveNext() in /home/alexandre/Projetos/ExampleParallelHttp3/Program.cs:line 99

Regression?

No response

Known Workarounds

No response

Configuration

~> dotnet --info
.NET SDK:
 Version:           8.0.101
 Commit:            6eceda187b
 Workload version:  8.0.100-manifests.69afb982

Ambiente de runtime:
 OS Name:     debian
 OS Version:  12
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /usr/share/dotnet/sdk/8.0.101/

Host:
  Version:      8.0.1
  Architecture: x64
  Commit:       bf5e279d92

libmsquic version: 2.3.1

Other information

I did not test this on Windows

Author:	alexandrehtrb
Assignees:	-
Labels:	untriaged, area-System.Net.Quic
Milestone:	-

[MihaZupan]

Moving to Quic area given the QUIC_STATUS_INVALID_PARAMETER failure

[ManickaP]

The server exception is dotnet/aspnetcore#45105, please upvote it there so that we know this troubles our users.

And I assume that the client exception happens when you cancel the token. So the only thing here is the way we bubble up the exception when CT gets cancelled. We should wrap it in OCE instead of HRE.

Moving back to HTTP as this is in S.N.Http.

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Hello,

I am writing a HTTP stress tool using C# and HttpClient, and in some rare cases, a QuicException arises when using HTTP/3.

Reproduction Steps

Server: dotnet new webapi with HTTP/3 enabled, using default GET /weatherforecast endpoint.

Console application to run stress test below. I had to dotnet run 10 times until the Exception showed up, because it doesn't always happen. (long code warning)

using System.Net;
using System.Threading.Channels;

namespace ExampleParallelHttp3;

public static class Program
{
    public static async Task Main(string[] args)
    {
        Console.WriteLine("Hello, World!");
        var hc = MakeHttpClient();
        var channelReader = StartRepetition(hc, 25000, 20, MakeConsoleCancellationToken());
        int i = 1;
        await foreach (var result in channelReader.ReadAllAsync())
        {
            if (result.Item3 is null)
            {
                Console.WriteLine($"i = {i++}, status code = {result.Item1}");
            }
            else
            {
                Console.WriteLine($"i = {i++}, exception = \n{result.Item3}\n");
                break;
            }
        }
        Console.WriteLine("Finished");
    }

    private static CancellationToken MakeConsoleCancellationToken()
    {
        // Add this to your C# console app's Main method to give yourself
        // a CancellationToken that is canceled when the user hits Ctrl+C.
        var cts = new CancellationTokenSource();
        Console.CancelKeyPress += (s, e) =>
        {
            Console.WriteLine("Canceling...");
            cts.Cancel();
            e.Cancel = true;
        };
        return cts.Token;
    }

    private static HttpClient MakeHttpClient()
    {
        SocketsHttpHandler httpHandler = new()
        {
            PooledConnectionLifetime = TimeSpan.FromMinutes(20),
            AutomaticDecompression = DecompressionMethods.All
        };
        
        httpHandler.SslOptions.RemoteCertificateValidationCallback =
            (sender, certificate, chain, sslPolicyErrors) => true;

        HttpClient httpClient = new(httpHandler, disposeHandler: false)
        {
            Timeout = TimeSpan.FromMinutes(5)
        };
        return httpClient;
    }

    public static ChannelReader<(HttpStatusCode?, string?, Exception?)> StartRepetition(HttpClient hc, int numReps, int maxDop, CancellationToken cancellationToken)
    {
        var channel = CreateChannel(numReps, maxDop);

        Task.Factory.StartNew(async () =>
        {
            try
            {
                await ExecuteParallelRequestsAsync(hc, channel.Writer, numReps, maxDop, cancellationToken);
            }
            catch (TaskCanceledException) { }
            catch (OperationCanceledException) { }
            catch (Exception ex)
            {
                (HttpStatusCode?, string?, Exception?) result = (null, null, ex);
                await channel.Writer.WriteAsync(result);
            }
            finally { channel.Writer.Complete(); }
        }, TaskCreationOptions.LongRunning);

        return channel.Reader;
    }

    private static async Task ExecuteParallelRequestsAsync(HttpClient hc, ChannelWriter<(HttpStatusCode?, string?, Exception?)> channelWriter, int numReps, int maxDop, CancellationToken cancellationToken)
    {
        ParallelOptions options = new();
        options.MaxDegreeOfParallelism = maxDop;
        options.CancellationToken = cancellationToken;

        await Parallel.ForAsync(0, numReps, options, async (i, ct) =>
        {
            try
            {
                HttpRequestMessage req = new(HttpMethod.Get, "https://localhost:5001/weatherforecast")
                {
                    Version = new(3, 0),
                    VersionPolicy = HttpVersionPolicy.RequestVersionExact
                };
                var res = await hc.SendAsync(req, ct);

                await channelWriter.WriteAsync((res.StatusCode, await res.Content.ReadAsStringAsync(ct), null), ct);
            }
            catch (Exception ex)
            {
                await channelWriter.WriteAsync((null, null, ex), ct);                
            }
        });
    }

    private static Channel<(HttpStatusCode?, string?, Exception?)> CreateChannel(int numReps, int maxDop)
    {
        BoundedChannelOptions channelOpts = new(numReps)
        {
            SingleReader = true,
            SingleWriter = maxDop == 1
        };
        var channel = Channel.CreateBounded<(HttpStatusCode?, string?, Exception?)>(channelOpts);
        return channel;
    }
}

Expected behavior

Exception shouldn't happen, I guess

Actual behavior

Error log in server:

crit: Microsoft.AspNetCore.Server.Kestrel[0]
      Unexpected exception in HttpConnection.ProcessRequestsAsync.
      System.Net.Quic.QuicException: An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER
         at System.Net.Quic.ThrowHelper.ThrowMsQuicException(Int32 status, String message)
         at System.Net.Quic.QuicStream.Abort(QuicAbortDirection abortDirection, Int64 errorCode)
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Internal.QuicStreamContext.Abort(ConnectionAbortedException abortReason)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3FrameWriter.Abort(ConnectionAbortedException error)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Stream.AbortCore(Exception exception, Http3ErrorCode errorCode)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.ProcessRequestsAsync[TContext](IHttpApplication`1 httpApplication)

Error log in console application:

i = 4, exception = 
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Quic.QuicException: Operation aborted.
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final)
   at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data)
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent)
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent)
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Http.Http3RequestStream.FlushSendBufferAsync(Boolean endStream, CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken)
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at ExampleParallelHttp3.Program.<>c__DisplayClass4_0.<<ExecuteParallelRequestsAsync>b__0>d.MoveNext() in /home/alexandre/Projetos/ExampleParallelHttp3/Program.cs:line 99

Regression?

No response

Known Workarounds

No response

Configuration

~> dotnet --info
.NET SDK:
 Version:           8.0.101
 Commit:            6eceda187b
 Workload version:  8.0.100-manifests.69afb982

Ambiente de runtime:
 OS Name:     debian
 OS Version:  12
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /usr/share/dotnet/sdk/8.0.101/

Host:
  Version:      8.0.1
  Architecture: x64
  Commit:       bf5e279d92

libmsquic version: 2.3.1

Other information

I did not test this on Windows

Author:	alexandrehtrb
Assignees:	-
Labels:	area-System.Net.Http, untriaged, area-System.Net.Quic
Milestone:	-