Fix HTTP3 stress #69182

rzikm · 2022-05-11T12:01:54Z

This PR builds latest MsQuic inside the http stress docker image.

It also adds special case for listening on IPv6Any to accept connections from all IP addresses (i.e. including IPv4), which does not happen by default (see microsoft/msquic#2704).

ghost · 2022-05-11T12:02:01Z

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Contributes to #67442.

This PR builds latest MsQuic inside the http stress docker image.

It also adds a (temporary) workaround for microsoft/msquic#2704 to unblock running H3 stress.

Author:	rzikm
Assignees:	-
Labels:	`area-System.Net.Http`
Milestone:	-

rzikm · 2022-05-11T12:02:10Z

cc: @ManickaP

src/libraries/System.Net.Http/tests/StressTests/HttpStress/Dockerfile

ManickaP · 2022-05-11T12:18:16Z

/azp run runtime-libraries stress-http

azure-pipelines · 2022-05-11T12:18:24Z

Azure Pipelines successfully started running 1 pipeline(s).

src/libraries/System.Net.Http/tests/StressTests/HttpStress/Dockerfile

rzikm · 2022-05-12T13:30:09Z

/azp run runtime-libraries stress-http

azure-pipelines · 2022-05-12T13:30:19Z

Azure Pipelines successfully started running 1 pipeline(s).

rzikm · 2022-05-12T18:33:33Z

The stress CI leg is failing because we run out of space when building the docker image, I filed dotnet/arcade#9349

rzikm · 2022-05-13T07:25:53Z

/azp run runtime-libraries stress-http

azure-pipelines · 2022-05-13T07:26:02Z

Azure Pipelines successfully started running 1 pipeline(s).

rzikm · 2022-05-13T08:25:40Z

/azp run runtime-libraries stress-http

azure-pipelines · 2022-05-13T08:25:49Z

Azure Pipelines successfully started running 1 pipeline(s).

ManickaP

LGTM in general.
Some more comments that are not necessarily prereqs for this PR:

what if you use IPv4.Any and client tries IPv6 address? How do sockets behave?
I ran this test:
https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicListenerTests.cs#L31
But I changed IPv6Loopback -> IPv6Any and it got stuck somewhere in the middle of a handshake. I didn't investigate further.

ManickaP · 2022-05-18T13:49:52Z

src/libraries/System.Net.Quic/src/System/Net/Quic/Implementations/MsQuic/MsQuicListener.cs

-            return MsQuicParameterHelpers.GetIPEndPointParam(MsQuicApi.Api, _state.Handle, QUIC_PARAM_LISTENER_LOCAL_ADDRESS);
+            QuicAddr listenAddr = MsQuicParameterHelpers.GetQuicAddrParam(MsQuicApi.Api, _state.Handle, QUIC_PARAM_LISTENER_LOCAL_ADDRESS);
+            int port = BinaryPrimitives.ReadUInt16BigEndian(MemoryMarshal.AsBytes(MemoryMarshal.CreateSpan(ref listenAddr.Ipv4.sin_port, 1)));
+            return new IPEndPoint(listenEndPoint.Address, port);


Are you sure that we always gonna get back the exact same listening address as we passed in?

I have checked MsQuic source code, and they only change the port (if it was 0)

ManickaP · 2022-05-18T13:56:04Z

src/libraries/System.Net.Quic/src/System/Net/Quic/Implementations/MsQuic/MsQuicListener.cs

            }

+            // return the actual bound address, including a port. Since the address family may be Unspecified,
+            // we cannot use GetIPEndPointParam. We have to manually read the port from the raw QuicAddr structure.


We could pass in optional param ipEndPoint.AddressFamily and in case msquic returns QUIC_ADDRESS_FAMILY_UNSPEC we would used it, wdyt?

you mean add optional parameter to GetIPEndPointParam? that seems to work. Thanks.

ManickaP · 2022-05-18T13:56:44Z

eng/pipelines/libraries/stress/http.yml

  pool:
    name: NetCore1ESPool-Public
-    demands: ImageOverride -equals Build.Ubuntu.1804.Amd64.Open
+    demands: ImageOverride -equals 1es-ubuntu-1804-open


Is this solving the issue with not enough space? Or why this change?

yes, it was suggested to me in dotnet/arcade#9349

rzikm · 2022-05-18T14:33:07Z

I ran this test:
https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicListenerTests.cs#L31
But I changed IPv6Loopback -> IPv6Any and it got stuck somewhere in the middle of a handshake. I didn't investigate further.

The test tries to connect to listener.ListenEndPoint, so if you didn't change that, then you tried to connect to IPv6Any, which dos not work.

what if you use IPv4.Any and client tries IPv6 address? How do sockets behave?

Even if you use DualMode sockets, then you can't connect via IPv6 if server listens on IPv4Any

ManickaP · 2022-05-18T15:00:32Z

I ran this test:
https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Quic/tests/FunctionalTests/QuicListenerTests.cs#L31
But I changed IPv6Loopback -> IPv6Any and it got stuck somewhere in the middle of a handshake. I didn't investigate further.

The test tries to connect to listener.ListenEndPoint, so if you didn't change that, then you tried to connect to IPv6Any, which dos not work.

Works with sockets though, I just tested it. It'll connect to IPv6 loopback if you pass IPv6Any to client connect, the same for IPv4Any.

rzikm · 2022-05-18T15:23:07Z

It does not seem to work for me

using System.Net.Sockets;
using System.Net;

int port = 5004;

Socket listener = new Socket(SocketType.Stream, ProtocolType.Tcp);
listener.DualMode = true;
listener.Bind(new IPEndPoint(IPAddress.IPv6Any, port));
listener.Listen();
Console.WriteLine($"Listening on {listener.LocalEndPoint}");

Console.WriteLine($"IsDualMode: {listener.DualMode}");
Console.WriteLine($"IPV6_V6Only: {listener.GetSocketOption(SocketOptionLevel.IPv6, SocketOptionName.IPv6Only)}");

using Socket clientIpv6 = new Socket(AddressFamily.InterNetworkV6, SocketType.Stream, ProtocolType.Tcp);
Console.WriteLine($"clientIpv6.DualMode == {clientIpv6.DualMode}");
await clientIpv6.ConnectAsync(new IPEndPoint(IPAddress.IPv6Any, port));

Console.WriteLine($"IsDualMode: {listener.DualMode}");
Console.WriteLine($"IPV6_V6Only: {listener.GetSocketOption(SocketOptionLevel.IPv6, SocketOptionName.IPv6Only)}");

Listening on [::]:5004
IsDualMode: True
IPV6_V6Only: 0
clientIpv6.DualMode == False
Unhandled exception. System.Net.Sockets.SocketException (10049): The requested address is not valid in its context.
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
   at Program.<Main>$(String[] args) in C:\source\DotnetSandbox\Program.cs:line 22
   at Program.<Main>(String[] args)

ManickaP · 2022-05-18T15:31:12Z

Works for me:

class Program
{
    static async Task Main(string[] args)
    {
        await Task.WhenAll(RunServer(), RunClient());
    }

    static readonly TaskCompletionSource<IPEndPoint> serverEndpoint = new TaskCompletionSource<IPEndPoint>();

    static async Task RunClient()
    {
        var socket = new Socket(SocketType.Stream, ProtocolType.Tcp);
        socket.NoDelay = true;
        var endpoint = await serverEndpoint.Task;
        Console.WriteLine("Client connecting to: " + endpoint);
        socket.Connect(endpoint);
        Console.WriteLine("Client connected to: " + socket.RemoteEndPoint);
        var stream = new NetworkStream(socket, ownsSocket: true);
        await stream.WriteAsync(UTF8Encoding.UTF8.GetBytes("Ahoj"));
        var buffer = new byte[100];
        int readBytes;
        while ((readBytes = await stream.ReadAsync(buffer)) > 0)
        {
            Console.WriteLine("Client:" + UTF8Encoding.UTF8.GetString(buffer, 0, readBytes));
        }
        Console.WriteLine("Client:" + readBytes);
    }
    static async Task RunServer()
    {
        var listenSocket = new Socket(SocketType.Stream, ProtocolType.Tcp);
        listenSocket.Bind(new IPEndPoint(IPAddress.IPv6Any, 0));
        listenSocket.Listen();
        serverEndpoint.SetResult(listenSocket.LocalEndPoint as IPEndPoint);
        Console.WriteLine("Server listening on: " + listenSocket.LocalEndPoint);
        var socket = await listenSocket.AcceptAsync().ConfigureAwait(false);
        var stream = new NetworkStream(socket, ownsSocket: true);
        var buffer = new byte[100];
        int readBytes = await stream.ReadAsync(buffer);
        Console.WriteLine("Server:" + UTF8Encoding.UTF8.GetString(buffer, 0, readBytes));
        stream.Dispose();
    }
}

Server listening on: [::]:38987
Client connecting to: [::]:38987
Client connected to: [::1]:38987
Server:Ahoj
Client:0

ManickaP · 2022-05-18T15:32:07Z

Your code runs for me as well:

Listening on [::]:5004
IsDualMode: True
IPV6_V6Only: 0
clientIpv6.DualMode == False
IsDualMode: True
IPV6_V6Only: 0

This smells like platform inconsistency with sockets. And the question is what behavior is correct.
cc @antonfirsov

rzikm · 2022-05-18T15:33:20Z

That would be probably a difference between Windows and Linux networking stack. It runs for me in WSL

ManickaP

LGTM, we can finish the discussion about sockets offline 😄 Thanks!

ghost added the area-System.Net.Http label May 11, 2022

ghost assigned rzikm May 11, 2022

rzikm requested a review from ManickaP May 11, 2022 12:02

ManickaP reviewed May 11, 2022

View reviewed changes

CarnaViire reviewed May 12, 2022

View reviewed changes

src/libraries/System.Net.Http/tests/StressTests/HttpStress/Dockerfile Outdated Show resolved Hide resolved

rzikm added 5 commits May 12, 2022 13:56

Build msquic locally

1c7d22f

Use IPv4 addresses only when testing HTTP3

e27a380

Code review changes

227af90

fixup! Use IPv4 addresses only when testing HTTP3

bf029fe

Make listening on IPv6Any accept IPv4 connections as well

44ca4ad

rzikm force-pushed the 67442-H3-stress-not-running branch from a34a91e to 44ca4ad Compare May 12, 2022 13:08

Minor changes

ec42d21

rzikm mentioned this pull request May 12, 2022

HTTP/3: RemoteCertificateValidationCallback to allow invalid TLS certification isn't working #68958

Closed

Don't continue if runtime image build fails

d2a5f27

rzikm mentioned this pull request May 12, 2022

Docker runtime build runs out of disk space dotnet/arcade#9349

Closed

2 tasks

Switch to 1es-ubuntu-1804-open image

0bc8a4c

rzikm requested review from CarnaViire and ManickaP May 13, 2022 11:17

rzikm mentioned this pull request May 13, 2022

QPACK error during HTTP/3 stress #69310

Closed

ManickaP reviewed May 18, 2022

View reviewed changes

Code review feedback

e6ebcea

ManickaP approved these changes May 18, 2022

View reviewed changes

rzikm merged commit 8d4a724 into dotnet:main May 20, 2022

ghost locked as resolved and limited conversation to collaborators Jun 19, 2022

karelz added this to the 7.0.0 milestone Jul 19, 2022

Fix HTTP3 stress #69182

Fix HTTP3 stress #69182

Uh oh!

Conversation

rzikm commented May 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented May 11, 2022

Uh oh!

rzikm commented May 11, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ManickaP commented May 11, 2022

Uh oh!

azure-pipelines bot commented May 11, 2022

Uh oh!

Uh oh!

rzikm commented May 12, 2022

Uh oh!

azure-pipelines bot commented May 12, 2022

Uh oh!

rzikm commented May 12, 2022

Uh oh!

rzikm commented May 13, 2022

Uh oh!

azure-pipelines bot commented May 13, 2022

Uh oh!

rzikm commented May 13, 2022

Uh oh!

azure-pipelines bot commented May 13, 2022

Uh oh!

ManickaP left a comment

Choose a reason for hiding this comment

Uh oh!

ManickaP May 18, 2022

Choose a reason for hiding this comment

Uh oh!

rzikm May 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ManickaP May 18, 2022

Choose a reason for hiding this comment

Uh oh!

rzikm May 18, 2022

Choose a reason for hiding this comment

Uh oh!

ManickaP May 18, 2022

Choose a reason for hiding this comment

Uh oh!

rzikm May 18, 2022

Choose a reason for hiding this comment

Uh oh!

rzikm commented May 18, 2022

Uh oh!

ManickaP commented May 18, 2022

Uh oh!

rzikm commented May 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ManickaP commented May 18, 2022

Uh oh!

ManickaP commented May 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rzikm commented May 18, 2022

Uh oh!

ManickaP left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

rzikm commented May 11, 2022 •

edited

Loading

rzikm May 18, 2022 •

edited

Loading

rzikm commented May 18, 2022 •

edited

Loading

ManickaP commented May 18, 2022 •

edited

Loading