HttpClient.GetAsync(...) throws Win32Exception when request sent first to https://FQDN and then to https://FQDN:8000

[davidkaya]

Description

Hello,

we have encountered a weird behaviour where if we create a HttpClient and first send a request (e.g. using HttpClient.GetAsync) to https://<redacted>.com and then to https://<redacted>.com:8000, we get a System.ComponentModel.Win32Exception (0x80090304): The Local Security Authority cannot be contacted. But if we do it the other way around, i.e. sending the request first to :8000 and then to uri without port (i.e. :443), it works just fine.

Is this an expected behaviour?

Reproduction Steps

// The FQDN is the same in all cases

// Does not work (throws exception)
/*
 * Unhandled exception. System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> System.ComponentModel.Win32Exception (0x80090304): The Local Security Authority cannot be contacted
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Program.<Main>$(String[] args) in C:\workspace\work\temp\ConsoleApp3\ConsoleApp3\Program.cs:line 5
 */
var firstHttpClient = new HttpClient();
await firstHttpClient.GetAsync("https://<redacted>.com");
await firstHttpClient.GetAsync("https://<redacted>.com:8000"); // <- this call throws the exception above

// Works (no exception)
var secondHttpClient = new HttpClient();
await secondHttpClient.GetAsync("https://<redacted>.com:8000");
await secondHttpClient.GetAsync("https://<redacted>.com");

Expected behavior

No exception is thrown and it doesn't depend in which order we send requests.

Actual behavior

An exception is thrown (see reproduction steps).

Regression?

No response

Known Workarounds

The workaround for us is to use separate HttpClient instances.

Configuration

net8 running on x64 Windows 11.

Other information

No response

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Hello,

we have encountered a weird behaviour where if we create a HttpClient and first send a request (e.g. using HttpClient.GetAsync) to https://<redacted>.com and then to https://<redacted>.com:8000, we get a System.ComponentModel.Win32Exception (0x80090304): The Local Security Authority cannot be contacted. But if we do it the other way around, i.e. sending the request first to :8000 and then to uri without port (i.e. :443), it works just fine.

Is this an expected behaviour?

Reproduction Steps

// The FQDN is the same in all cases

// Does not work (throws exception)
/*
 * Unhandled exception. System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> System.ComponentModel.Win32Exception (0x80090304): The Local Security Authority cannot be contacted
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Program.<Main>$(String[] args) in C:\workspace\work\temp\ConsoleApp3\ConsoleApp3\Program.cs:line 5
 */
var firstHttpClient = new HttpClient();
await firstHttpClient.GetAsync("https://<redacted>.com");
await firstHttpClient.GetAsync("https://<redacted>:8000"); // <- this call throws the exception above

// Works (no exception)
var secondHttpClient = new HttpClient();
await secondHttpClient.GetAsync("https://<redacted>.com:8000");
await secondHttpClient.GetAsync("https://<redacted>.com");

Expected behavior

No exception is thrown and it doesn't depend in which order we send requests.

Actual behavior

An exception is thrown (see reproduction steps).

Regression?

No response

Known Workarounds

The workaround for us is to use separate HttpClient instances.

Configuration

net8 running on x64 Windows 11.

Other information

No response

Author:	davidkaya
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

[wfurt]

Do you have same configuration on both @davidkaya? I suspect this is because cached credentials on Windows. The SNI e.g.the TargetHostName is same so picks same for both cases. If the configuration is different it may not work as expected. I run to something similar very long time ago. I think schannel has also cache.

We can inspect the packet captures. I would think we can reproduce it with simple SslStream app.

[davidkaya]

@wfurt If you by "configuration" mean the configuration of the HttpClient, then yes, I can reproduce it with the code in the Repro steps, which is a clean HttpClient. Both ports (443 and 8000) use the same SSL certificate.

I just reproduced it also with a SslStream app (same output for two different colleagues):

var hostname = "<redacted>.com";

// Works
Console.WriteLine("---Connecting to 8000---");
RunClient(hostname, hostname, 8000);
Console.WriteLine("---Connecting to 443---");
RunClient(hostname, hostname, 443);

// Throws
Console.WriteLine("---Connecting to 443---");
RunClient(hostname, hostname, 443);
Console.WriteLine("---Connecting to 8000---");
RunClient(hostname, hostname, 8000);


static void RunClient(string machineName, string serverName, int port)
{
    var client = new TcpClient(machineName, port);
    var sslStream = new SslStream(client.GetStream());
    try
    {
        sslStream.AuthenticateAsClient(serverName);
    }
    catch (AuthenticationException e)
    {
        Console.WriteLine(e);
        client.Close();
        return;
    }

    sslStream.Write(Encoding.UTF8.GetBytes(string.Format("GET https://{0}/  HTTP/1.1\r\nHost: {0}\r\n\r\n", serverName)));
    sslStream.Flush();
    var serverMessage = ReadMessage(sslStream);
    Console.WriteLine(
        $"""
         Response:
         -----------------
         {serverMessage}
         -----------------
         """
    );
    client.Close();
    Console.WriteLine("Client closed.");
    Console.WriteLine();
}

static string ReadMessage(SslStream sslStream)
{
    var buffer = new byte[2048];
    var bytes = sslStream.Read(buffer, 0, buffer.Length);
    return Encoding.UTF8.GetString(buffer[..bytes]);
}

and the output is

---Connecting to 8000---
Response:
-----------------
HTTP/1.1 404 Not Found
Content-Length: 0
Date: Wed, 21 Feb 2024 10:01:26 GMT
Server: Kestrel


-----------------
Client closed.

---Connecting to 443---
Response:
-----------------
HTTP/1.1 400 Bad Request
Content-Type: text/plain; charset=utf-8
Connection: close

400 Bad Request
-----------------
Client closed.

---Connecting to 443---
Response:
-----------------
HTTP/1.1 400 Bad Request
Content-Type: text/plain; charset=utf-8
Connection: close

400 Bad Request
-----------------
Client closed.

---Connecting to 8000---
System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> System.ComponentModel.Win32Exception (0x80090304): The Local Security Authority cannot be contacted
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.AuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions)
   at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost)
   at Program.<<Main>$>g__RunClient|0_0(String machineName, String serverName, Int32 port) in C:\workspace\work\temp\ConsoleApp3\ConsoleApp3\Program.cs:line 57

Tagging subscribers to this area: @dotnet/ncl, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Hello,

we have encountered a weird behaviour where if we create a HttpClient and first send a request (e.g. using HttpClient.GetAsync) to https://<redacted>.com and then to https://<redacted>.com:8000, we get a System.ComponentModel.Win32Exception (0x80090304): The Local Security Authority cannot be contacted. But if we do it the other way around, i.e. sending the request first to :8000 and then to uri without port (i.e. :443), it works just fine.

Is this an expected behaviour?

Reproduction Steps

// The FQDN is the same in all cases

// Does not work (throws exception)
/*
 * Unhandled exception. System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> System.ComponentModel.Win32Exception (0x80090304): The Local Security Authority cannot be contacted
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Program.<Main>$(String[] args) in C:\workspace\work\temp\ConsoleApp3\ConsoleApp3\Program.cs:line 5
 */
var firstHttpClient = new HttpClient();
await firstHttpClient.GetAsync("https://<redacted>.com");
await firstHttpClient.GetAsync("https://<redacted>.com:8000"); // <- this call throws the exception above

// Works (no exception)
var secondHttpClient = new HttpClient();
await secondHttpClient.GetAsync("https://<redacted>.com:8000");
await secondHttpClient.GetAsync("https://<redacted>.com");

Expected behavior

No exception is thrown and it doesn't depend in which order we send requests.

Actual behavior

An exception is thrown (see reproduction steps).

Regression?

No response

Known Workarounds

The workaround for us is to use separate HttpClient instances.

Configuration

net8 running on x64 Windows 11.

Other information

No response

Author:	davidkaya
Assignees:	-
Labels:	area-System.Net.Security, untriaged
Milestone:	-

[wfurt]

right, this is what I suspected. But I don't think we can make much progress without more details about the server or at least packet captures. If you do not want to share it publicly contact me or @rzikm privately.

[wfurt]

It is hard to see with the default TLS 1.3 but the the 443 site sends out only leaf cert while the 8000 site sends whole chain as it should.

port 443

and port 8000

You can see that by forcing TLS 1.2 or something like openssl s_client -tls1_2 -connect x.y.z:443
You would see more of the handshake clearly and you can notice difference in size of the certificate chunk server sends back.

Since .NET will cache intermediates (when it can) it will work if you connect first to the site that sends the chain @davidkaya.
HttpClient (SslStream/X509Chain) should try to fetch the missing intermediates. Either to download is blocked or something else is going on. It feel like the easiest fix is to update the 443 to send the chain as it should.

let us know.

This issue has been marked needs-author-action and may be missing some important information.

[davidkaya]

@wfurt We have tried to configure the service on :443 to send the full certificate chain but it still fails on the same error.

Port 443 sends whole chain including root:

Port 8000 sends whole chain except root:

There is one thing that I've noticed. When I forced SslStream to use TLS 1.2, I do not get the error anymore, so the error occurs only in the case of TLS 1.3:

This works:

1. Send request to :8000 (TLS 1.2)
2. Send request to :443 (TLS 1.2)

This works:

1. Send request to :443 (TLS 1.2)
2. Send request to :8000 (TLS 1.2)

This works:

1. Send request to :8000 (TLS 1.3)
2. Send request to :443 (TLS 1.3)

This fails:

1. Send request to :443 (TLS 1.3)
2. Send request to :8000 (TLS 1.3)

[rzikm]

Triage: we should have a look at 9.0 timeframe.