HTTP/3: RemoteCertificateValidationCallback to allow invalid TLS certification isn't working

[JamesNK]

Description

An ASP.NET Core HTTP/3 test is broken. It configures the server to allow a client certificate and to accept invalid certs. However, invalid certs continue to fail.

Unfortunately, we don't know the exact date it started breaking as all our test failures were ignored. We noticed it failing when the test ignores were removed from HTTP/3 tests in August 2021 - dotnet/aspnetcore#35070

Test:
https://github.com/dotnet/aspnetcore/blob/70d7f5f9e02c84084daec2f6666decd343e56b46/src/Servers/Kestrel/test/Interop.FunctionalTests/Http3/Http3TlsTests.cs#L154-L213

AllowAnyClientCertificate() configures SslServerAuthenticationOptions.RemoteCertificateValidationCallback. The SslServerAuthenticationOptions are passed to the QUIC listener. I debugged the test and SslServerAuthenticationOptions.RemoteCertificateValidationCallback isn't called by S.N.Q when the test executes.

Is there a test for this scenario in the runtime repo?

@Tratcher @wtgodbe @adityamandaleeka

Reproduction Steps

Run test Interop.FunctionalTests.Http3.Http3TlsTests.ClientCertificate_AllowOrRequire_Available_Invalid_Refused(mode: AllowCertificate, serverAllowInvalid: True)

Failure log:

 Interop.FunctionalTests.Http3.Http3TlsTests.ClientCertificate_AllowOrRequire_Available_Invalid_Refused(mode: AllowCertificate, serverAllowInvalid: True)
   Source: Http3TlsTests.cs line 162
   Duration: 6 sec

  Message: 
System.Net.Http.HttpRequestException : Connection has been shutdown by transport. Error Code: ALPN_NEG_FAILURE (127.0.0.1:54972)
---- System.Net.Quic.QuicException : Connection has been shutdown by transport. Error Code: ALPN_NEG_FAILURE

  Stack Trace: 
ConnectHelper.ConnectQuicAsync(HttpRequestMessage request, QuicImplementationProvider quicImplementationProvider, DnsEndPoint endPoint, SslClientAuthenticationOptions clientAuthenticationOptions, CancellationToken cancellationToken)
HttpConnectionPool.GetHttp3ConnectionAsync(HttpRequestMessage request, HttpAuthority authority, CancellationToken cancellationToken)
HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken)
HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
TaskExtensions.TimeoutAfter[T](Task`1 task, TimeSpan timeout, String filePath, Int32 lineNumber) line 83
Http3TlsTests.ClientCertificate_AllowOrRequire_Available_Invalid_Refused(ClientCertificateMode mode, Boolean serverAllowInvalid) line 208
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
MsQuicConnection.HandleEventShutdownInitiatedByTransport(State state, ConnectionEvent& connectionEvent)
MsQuicConnection.NativeCallbackHandler(IntPtr connection, IntPtr context, ConnectionEvent* connectionEvent)
--- End of stack trace from previous location ---
ConnectHelper.ConnectQuicAsync(HttpRequestMessage request, QuicImplementationProvider quicImplementationProvider, DnsEndPoint endPoint, SslClientAuthenticationOptions clientAuthenticationOptions, CancellationToken cancellationToken)

  Standard Output: 
| [0.005s] TestLifetime Information: Starting test ClientCertificate_AllowOrRequire_Available_Invalid_Refused-AllowCertificate-True at 2022-05-06T12:27:49
| [0.298s] Microsoft.Extensions.Hosting.Internal.Host Debug: Hosting starting
| [5.831s] Microsoft.Hosting.Lifetime Information: Now listening on: https://[::]:54972
| [5.833s] Microsoft.AspNetCore.Hosting.Diagnostics Debug: Loaded hosting startup assembly Interop.FunctionalTests
| [5.833s] Microsoft.Hosting.Lifetime Information: Application started. Press Ctrl+C to shut down.
| [5.843s] Microsoft.Hosting.Lifetime Information: Hosting environment: Production
| [5.843s] Microsoft.Hosting.Lifetime Information: Content root path: C:\Development\Source\aspnetcore\artifacts\bin\Interop.FunctionalTests\Debug\net7.0\
| [5.843s] Microsoft.Extensions.Hosting.Internal.Host Debug: Hosting started
| [6.012s] Microsoft.AspNetCore.Server.Kestrel.Transport.Quic Debug: Listener has aborted with exception: Operation aborted.
| [6.039s] Interop.FunctionalTests.Http3.Http3TlsTests Error: Test threw an exception.
| System.Net.Http.HttpRequestException: Connection has been shutdown by transport. Error Code: ALPN_NEG_FAILURE (127.0.0.1:54972)
|  ---> System.Net.Quic.QuicException: Connection has been shutdown by transport. Error Code: ALPN_NEG_FAILURE
|    at System.Net.Quic.Implementations.MsQuic.MsQuicConnection.HandleEventShutdownInitiatedByTransport(State state, ConnectionEvent& connectionEvent)
|    at System.Net.Quic.Implementations.MsQuic.MsQuicConnection.NativeCallbackHandler(IntPtr connection, IntPtr context, ConnectionEvent* connectionEvent)
| --- End of stack trace from previous location ---
|    at System.Net.Http.ConnectHelper.ConnectQuicAsync(HttpRequestMessage request, QuicImplementationProvider quicImplementationProvider, DnsEndPoint endPoint, SslClientAuthenticationOptions clientAuthenticationOptions, CancellationToken cancellationToken)
|    --- End of inner exception stack trace ---
|    at System.Net.Http.ConnectHelper.ConnectQuicAsync(HttpRequestMessage request, QuicImplementationProvider quicImplementationProvider, DnsEndPoint endPoint, SslClientAuthenticationOptions clientAuthenticationOptions, CancellationToken cancellationToken)
|    at System.Net.Http.HttpConnectionPool.GetHttp3ConnectionAsync(HttpRequestMessage request, HttpAuthority authority, CancellationToken cancellationToken)
|    at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken)
|    at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
|    at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
|    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
|    at Microsoft.AspNetCore.Testing.TaskExtensions.TimeoutAfter[T](Task`1 task, TimeSpan timeout, String filePath, Int32 lineNumber) in C:\Development\Source\aspnetcore\src\Shared\TaskExtensions.cs:line 83
|    at Interop.FunctionalTests.Http3.Http3TlsTests.ClientCertificate_AllowOrRequire_Available_Invalid_Refused(ClientCertificateMode mode, Boolean serverAllowInvalid) in C:\Development\Source\aspnetcore\src\Servers\Kestrel\test\Interop.FunctionalTests\Http3\Http3TlsTests.cs:line 208
|    at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_1.<<InvokeTestMethodAsync>b__1>d.MoveNext() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 264
| --- End of stack trace from previous location ---
|    at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\ExecutionTimer.cs:line 48
|    at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 90
| [6.048s] TestLifetime Information: Finished test ClientCertificate_AllowOrRequire_Available_Invalid_Refused-AllowCertificate-True in 6.0457029s

Expected behavior

A connection with invalid client certificate is accepted by the server.

Actual behavior

See earlier exception.

Regression?

Yes. It worked at one point.

Known Workarounds

No response

Configuration

No response

Other information

No response

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

An ASP.NET Core HTTP/3 test is broken. It configures the server to allow a client certificate and to accept invalid certs. However, invalid certs continue to fail.

Unfortunately, we don't know the exact date it started breaking as all our test failures were ignored. It was only noticed when test ignore was removed in August 2021 - dotnet/aspnetcore#35070

Test:
https://github.com/dotnet/aspnetcore/blob/70d7f5f9e02c84084daec2f6666decd343e56b46/src/Servers/Kestrel/test/Interop.FunctionalTests/Http3/Http3TlsTests.cs#L154-L213

AllowAnyClientCertificate() configures SslServerAuthenticationOptions.RemoteCertificateValidationCallback which is then used when the listener is created. I debugged the test and SslServerAuthenticationOptions.RemoteCertificateValidationCallback isn't called by S.N.Q when the test executes.

Is there a test for this scenario in the runtime repo?

@Tratcher @wtgodbe @adityamandaleeka

Reproduction Steps

Run test Interop.FunctionalTests.Http3.Http3TlsTests.ClientCertificate_AllowOrRequire_Available_Invalid_Refused(mode: AllowCertificate, serverAllowInvalid: True)

Failure log:

 Interop.FunctionalTests.Http3.Http3TlsTests.ClientCertificate_AllowOrRequire_Available_Invalid_Refused(mode: AllowCertificate, serverAllowInvalid: True)
   Source: Http3TlsTests.cs line 162
   Duration: 6 sec

  Message: 
System.Net.Http.HttpRequestException : Connection has been shutdown by transport. Error Code: ALPN_NEG_FAILURE (127.0.0.1:54972)
---- System.Net.Quic.QuicException : Connection has been shutdown by transport. Error Code: ALPN_NEG_FAILURE

  Stack Trace: 
ConnectHelper.ConnectQuicAsync(HttpRequestMessage request, QuicImplementationProvider quicImplementationProvider, DnsEndPoint endPoint, SslClientAuthenticationOptions clientAuthenticationOptions, CancellationToken cancellationToken)
HttpConnectionPool.GetHttp3ConnectionAsync(HttpRequestMessage request, HttpAuthority authority, CancellationToken cancellationToken)
HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken)
HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
TaskExtensions.TimeoutAfter[T](Task`1 task, TimeSpan timeout, String filePath, Int32 lineNumber) line 83
Http3TlsTests.ClientCertificate_AllowOrRequire_Available_Invalid_Refused(ClientCertificateMode mode, Boolean serverAllowInvalid) line 208
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
MsQuicConnection.HandleEventShutdownInitiatedByTransport(State state, ConnectionEvent& connectionEvent)
MsQuicConnection.NativeCallbackHandler(IntPtr connection, IntPtr context, ConnectionEvent* connectionEvent)
--- End of stack trace from previous location ---
ConnectHelper.ConnectQuicAsync(HttpRequestMessage request, QuicImplementationProvider quicImplementationProvider, DnsEndPoint endPoint, SslClientAuthenticationOptions clientAuthenticationOptions, CancellationToken cancellationToken)

  Standard Output: 
| [0.005s] TestLifetime Information: Starting test ClientCertificate_AllowOrRequire_Available_Invalid_Refused-AllowCertificate-True at 2022-05-06T12:27:49
| [0.298s] Microsoft.Extensions.Hosting.Internal.Host Debug: Hosting starting
| [5.831s] Microsoft.Hosting.Lifetime Information: Now listening on: https://[::]:54972
| [5.833s] Microsoft.AspNetCore.Hosting.Diagnostics Debug: Loaded hosting startup assembly Interop.FunctionalTests
| [5.833s] Microsoft.Hosting.Lifetime Information: Application started. Press Ctrl+C to shut down.
| [5.843s] Microsoft.Hosting.Lifetime Information: Hosting environment: Production
| [5.843s] Microsoft.Hosting.Lifetime Information: Content root path: C:\Development\Source\aspnetcore\artifacts\bin\Interop.FunctionalTests\Debug\net7.0\
| [5.843s] Microsoft.Extensions.Hosting.Internal.Host Debug: Hosting started
| [6.012s] Microsoft.AspNetCore.Server.Kestrel.Transport.Quic Debug: Listener has aborted with exception: Operation aborted.
| [6.039s] Interop.FunctionalTests.Http3.Http3TlsTests Error: Test threw an exception.
| System.Net.Http.HttpRequestException: Connection has been shutdown by transport. Error Code: ALPN_NEG_FAILURE (127.0.0.1:54972)
|  ---> System.Net.Quic.QuicException: Connection has been shutdown by transport. Error Code: ALPN_NEG_FAILURE
|    at System.Net.Quic.Implementations.MsQuic.MsQuicConnection.HandleEventShutdownInitiatedByTransport(State state, ConnectionEvent& connectionEvent)
|    at System.Net.Quic.Implementations.MsQuic.MsQuicConnection.NativeCallbackHandler(IntPtr connection, IntPtr context, ConnectionEvent* connectionEvent)
| --- End of stack trace from previous location ---
|    at System.Net.Http.ConnectHelper.ConnectQuicAsync(HttpRequestMessage request, QuicImplementationProvider quicImplementationProvider, DnsEndPoint endPoint, SslClientAuthenticationOptions clientAuthenticationOptions, CancellationToken cancellationToken)
|    --- End of inner exception stack trace ---
|    at System.Net.Http.ConnectHelper.ConnectQuicAsync(HttpRequestMessage request, QuicImplementationProvider quicImplementationProvider, DnsEndPoint endPoint, SslClientAuthenticationOptions clientAuthenticationOptions, CancellationToken cancellationToken)
|    at System.Net.Http.HttpConnectionPool.GetHttp3ConnectionAsync(HttpRequestMessage request, HttpAuthority authority, CancellationToken cancellationToken)
|    at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken)
|    at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
|    at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
|    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
|    at Microsoft.AspNetCore.Testing.TaskExtensions.TimeoutAfter[T](Task`1 task, TimeSpan timeout, String filePath, Int32 lineNumber) in C:\Development\Source\aspnetcore\src\Shared\TaskExtensions.cs:line 83
|    at Interop.FunctionalTests.Http3.Http3TlsTests.ClientCertificate_AllowOrRequire_Available_Invalid_Refused(ClientCertificateMode mode, Boolean serverAllowInvalid) in C:\Development\Source\aspnetcore\src\Servers\Kestrel\test\Interop.FunctionalTests\Http3\Http3TlsTests.cs:line 208
|    at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_1.<<InvokeTestMethodAsync>b__1>d.MoveNext() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 264
| --- End of stack trace from previous location ---
|    at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\ExecutionTimer.cs:line 48
|    at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 90
| [6.048s] TestLifetime Information: Finished test ClientCertificate_AllowOrRequire_Available_Invalid_Refused-AllowCertificate-True in 6.0457029s

Expected behavior

A connection with invalid client certificate is accepted by the server.

Actual behavior

See earlier exception.

Regression?

Yes. It worked at one point.

Known Workarounds

No response

Configuration

No response

Other information

No response

Author:	JamesNK
Assignees:	-
Labels:	area-System.Net.Http
Milestone:	-

[wfurt]

This is client certificate, right? And Windows? May be dup of #57308. That one was blocked on underlying MsQuic work.

[wfurt]

Can you take a look @rzikm?

[rzikm]

Seems like this is the same issue we are hitting in #67442

You are listening on IPv6Any and try to connect using IPv4 loopback address, MsQuic does not do this (apparently by design, see microsoft/msquic#2704 (comment)). The ALPN_NEG_FAILURE Error is sent without even attempting a handshake (due to some MsQuic implementation details)

This should get fixed by #69182

[Tratcher]

Ah, https://github.com/dotnet/aspnetcore/blob/70d7f5f9e02c84084daec2f6666decd343e56b46/src/Servers/Kestrel/test/Interop.FunctionalTests/Http3/Http3TlsTests.cs#L191

[JamesNK]

Isn't the server listening on IPv4 and IPv6? https://github.com/dotnet/aspnetcore/blob/70d7f5f9e02c84084daec2f6666decd343e56b46/src/Servers/Kestrel/test/Interop.FunctionalTests/Http3/Http3TlsTests.cs#L170

Edit: Or is ListenAnyIP use IPv6 if possible, then fallback to IPv4? https://docs.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.server.kestrel.core.kestrelserveroptions.listenanyip?view=aspnetcore-6.0

[rzikm]

Isn't the server listening on IPv4 and IPv6?
Edit: Or is ListenAnyIP use IPv6 if possible, then fallback to IPv4?

ListenAnyIP prioritizes IPv6, and since that works on the machine, it is what the server ends up doing in this case. The problem is that if you instruct MsQuic to listen on IPv6Any, then it will not accept IPv4 connections (a difference from how TCP behaves, see microsoft/msquic#2704).

Luckily, MsQuic can be configured to accept both IPv4 and IPv6 connections, but it needs to be requested explicitly by using the "Unspecified" address family when configuring the listen endpoint. In #69182, we are going to make sure this is done transparently to the user so they don't need to be aware of this difference.

If you have any concerns, feel free to comment on the mentioned MsQuic issue, but from the discussion it seems they can't change this behavior, as it would be a breaking change and "mimicking TCP behavior" isn't good enough reason to do that.

[rzikm]

@JamesNK Can you check again? Latest main should now accept IPv4 connections when listening on IPv6Any.

[JamesNK]

Failing test has started passing: