HttpClient SSL Connection with Client Certificate Fails After Code Change

[sflanker]

Description

I have a C# console app that makes an HttpClient connection that uses a Client Certificate for authentication. It works fine when I initially run it. However after making a completely unrelated/innocuous code change, the next time I run it I get the following error:

Unhandled exception. System.AggregateException: One or more errors occurred. (The SSL connection could not be established, see inner exception.)
 ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> Interop+AppleCrypto+SslException: The user name or passphrase you entered is not correct.
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.HttpConnectionWaiter`1.WaitForConnectionAsync(Boolean async, CancellationToken requestCancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.GetStringAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at Program.<Main>$(String[] args) in Program.cs:line 19

If I run dotnet clean before running again, it resolves the issue.

Reproduction Steps

Here is my test application:

using System.Security.Authentication;
using System.Security.Cryptography.X509Certificates;

var handler = new HttpClientHandler();

var url = args[0];
var clientCert = File.ReadAllText(args[1]);
var clientKey = File.ReadAllText(args[2]);

handler.ClientCertificateOptions = ClientCertificateOption.Manual;
handler.SslProtocols = SslProtocols.Tls12;
handler.ClientCertificates.Add(
  X509Certificate2.CreateFromPem(clientCert, clientKey)
);

var httpClient = new HttpClient(handler);
var responseTask = httpClient.GetStringAsync(url);
responseTask.Wait();
var response = responseTask.Result;
Console.WriteLine(response);

I run this program with something like the following:

dotnet run -- "https://127.0.0.1:57720/api/v1/namespaces" client_cert.crt client_cert.key

I'm connecting to a run-of-the-mill k3d cluster using Docker Desktop for Mac on Mac OS Monterey. I generate the crt and key with the following commands:

yq '.users[0].user.client-certificate-data' ~/.kube/config | base64 -d > client_cert.crt
yq '.users[0].user.client-key-data' ~/.kube/config | base64 -d > client_cert.key

Expected behavior

Running the program after changing the code should successfully connect and authenticate with the client certificate, even after changing the code, without running dotnet clean.

Actual behavior

Unhandled exception. System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> Interop+AppleCrypto+SslException: The user name or passphrase you entered is not correct.

Regression?

No response

Known Workarounds

running dotnet clean after every code change.

Configuration

MacOS 12.6.3
.Net 7.0.202
Docker Desktop 4.16.2
k3d version v5.4.1
k3s version v1.22.7-k3s1

Other information

No response

Tagging subscribers to this area: @dotnet/ncl, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I have a C# console app that makes an HttpClient connection that uses a Client Certificate for authentication. It works fine when I initially run it. However after making a completely unrelated/innocuous code change, the next time I run it I get the following error:

Unhandled exception. System.AggregateException: One or more errors occurred. (The SSL connection could not be established, see inner exception.)
 ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> Interop+AppleCrypto+SslException: The user name or passphrase you entered is not correct.
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.HttpConnectionWaiter`1.WaitForConnectionAsync(Boolean async, CancellationToken requestCancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.GetStringAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at Program.<Main>$(String[] args) in Program.cs:line 19

If I run dotnet clean before running again, it resolves the issue.

Reproduction Steps

Here is my test application:

using System.Security.Authentication;
using System.Security.Cryptography.X509Certificates;

var handler = new HttpClientHandler();

var url = args[0];
var clientCert = File.ReadAllText(args[1]);
var clientKey = File.ReadAllText(args[2]);

handler.ClientCertificateOptions = ClientCertificateOption.Manual;
handler.SslProtocols = SslProtocols.Tls12;
handler.ClientCertificates.Add(
  X509Certificate2.CreateFromPem(clientCert, clientKey)
);

var httpClient = new HttpClient(handler);
var responseTask = httpClient.GetStringAsync(url);
responseTask.Wait();
var response = responseTask.Result;
Console.WriteLine(response);

I run this program with something like the following:

dotnet run -- "https://127.0.0.1:57720/api/v1/namespaces" client_cert.crt client_cert.key

I'm connecting to a run-of-the-mill k3d cluster using Docker Desktop for Mac on Mac OS Monterey (full

Expected behavior

Running the program after changing the code should successfully connect and authenticate with the client certificate, even after changing the code, without running dotnet clean.

Actual behavior

Unhandled exception. System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> Interop+AppleCrypto+SslException: The user name or passphrase you entered is not correct.

Regression?

No response

Known Workarounds

running dotnet clean after every code change.

Configuration

MacOS 12.6.3
.Net 7.0.202
Docker Desktop 4.16.2
k3d version v5.4.1
k3s version v1.22.7-k3s1

Other information

No response

Author:	sflanker
Assignees:	-
Labels:	area-System.Net.Security
Milestone:	-

[wfurt]

Could you hook up FirstChanceException and check if there is any hidden exception? From what I can see it seems like the app is failing to read the certificate. That is managed out of the application process and it should work unless the app is signed with different key.

You an also try: log stream | grep securityd. That can show some more insight.

.NET creates temporary keychains and they have random passwords AFAIK.
But I would assume that would be overridden on new start.
If you have simple console app you can try to run it with dtruss to see what files it is accessing. And check if the clean removes them.

Any more ideas @vcsjones?

[sflanker]

I don't see any exceptions besides the aforementioned one being passed to the FirstChanceException event handler:

Interop+AppleCrypto+SslException: The user name or passphrase you entered is not correct.
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)

[sflanker]

I have attached the logs generated by for both the successful first run and the subsequent run with the error. I injected some blank lines to help vimdiff show the notable variations. Specifically the following is only present in the second run that has the error:

securityd: [com.apple.securityd:security_exception] MacOS error: -67063
securityd: [com.apple.securityd:security_exception] MacOS error: -67063
securityd: [com.apple.securityd:clientid] code requirement check failed (-67063), client is not Apple-signed
securityd: [com.apple.securityd:kcacl] Keychain query for process 45482 (UID 501)
securityd: [com.apple.securityd:security_exception] MacOS error: -67063
securityd: [com.apple.securityd:kcacl] client is invalid, suppressing prompt
securityd: [com.apple.securityd:kcacl] suppressing keychain prompt for invalidly signed client /Users/pwheeler/Code/experiments/ClientCertTest/bin/Debug/net7.0/ClientCertTest(45482)
securityd: (Security) [com.apple.securityd:security_exception] MacOS error: -25337
securityd: [com.apple.securityd:security_exception] CSSM Exception: 32 unknown error 32=20
ClientCertTest: (Security) [com.apple.securityd:security_exception] CSSM Exception: -2147416032 CSSMERR_CSP_OPERATION_AUTH_DENIED
ClientCertTest: (Security) [com.apple.securityd:security_exception] CSSM Exception: -2147416032 CSSMERR_CSP_OPERATION_AUTH_DENIED

Success log.txt
Error log.txt

[sflanker]

Note: this issue is only reproducible with dotnet run if I use dotnet build followed by dotnet path/to/dll it does not happen.