Unix: machine certificate store throws on load when it contains certificates the user has no read permission for

[tmds]

On Fedora 27:

       static async Task Main(string[] args)
        {
            using (var httpClient = new HttpClient())
            {
                var response = await httpClient.GetAsync("https://www.microsoft.com");
                System.Console.WriteLine(response.StatusCode);
            }
        }

With 2.1.300-rc1-008673 gives

OK

But with the 2.1.300 (https://github.com/aspnet/Home/wiki/2.1.0-Early-Access-Downloads) throws:

Unhandled Exception: System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> Interop+Crypto+OpenSslCryptographicException: error:2006D002:BIO routines:BIO_new_file:system lib
   at Interop.Crypto.CheckValidOpenSslHandle(SafeHandle handle)
   at Internal.Cryptography.Pal.StorePal.LoadMachineStores()
   at Internal.Cryptography.Pal.StorePal.FromSystemStore(String storeName, StoreLocation storeLocation, OpenFlags openFlags)
   at System.Security.Cryptography.X509Certificates.X509Store.Open(OpenFlags flags)
   at Internal.Cryptography.Pal.OpenSslX509ChainProcessor.FindCandidates(X509Certificate2 leaf, X509Certificate2Collection extraStore, HashSet`1 downloaded, HashSet`1 systemTrusted, TimeSpan& remainingDownloadTime)
   at Internal.Cryptography.Pal.ChainPal.BuildChain(Boolean useMachineContext, ICertificatePal cert, X509Certificate2Collection extraStore, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationMode revocationMode, X509RevocationFlag revocationFlag, DateTime verificationTime, TimeSpan timeout)
   at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate, Boolean throwOnException)
   at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate)
   at System.Net.Security.CertificateValidation.BuildChainAndVerifyProperties(X509Chain chain, X509Certificate2 remoteCertificate, Boolean checkCertName, String hostName)
   at System.Net.Security.SecureChannel.VerifyRemoteCertificate(RemoteCertValidationCallback remoteCertValidationCallback, ProtocolToken& alertToken)
   at System.Net.Security.SslState.CompleteHandshake(ProtocolToken& alertToken)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Security.SslState.ThrowIfExceptional()
   at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__47_1(IAsyncResult iar)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at console.Program.Main(String[] args) in /tmp/console/Program.cs:line 13
   at console.Program.<Main>(String[] args)
/tmp/console$ dotnet run

Unhandled Exception: System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> Interop+Crypto+OpenSslCryptographicException: error:2006D002:BIO routines:BIO_new_file:system lib
   at Interop.Crypto.CheckValidOpenSslHandle(SafeHandle handle)
   at Internal.Cryptography.Pal.StorePal.LoadMachineStores()
   at Internal.Cryptography.Pal.StorePal.FromSystemStore(String storeName, StoreLocation storeLocation, OpenFlags openFlags)
   at System.Security.Cryptography.X509Certificates.X509Store.Open(OpenFlags flags)
   at Internal.Cryptography.Pal.OpenSslX509ChainProcessor.FindCandidates(X509Certificate2 leaf, X509Certificate2Collection extraStore, HashSet`1 downloaded, HashSet`1 systemTrusted, TimeSpan& remainingDownloadTime)
   at Internal.Cryptography.Pal.ChainPal.BuildChain(Boolean useMachineContext, ICertificatePal cert, X509Certificate2Collection extraStore, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationMode revocationMode, X509RevocationFlag revocationFlag, DateTime verificationTime, TimeSpan timeout)
   at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate, Boolean throwOnException)
   at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate)
   at System.Net.Security.CertificateValidation.BuildChainAndVerifyProperties(X509Chain chain, X509Certificate2 remoteCertificate, Boolean checkCertName, String hostName)
   at System.Net.Security.SecureChannel.VerifyRemoteCertificate(RemoteCertValidationCallback remoteCertValidationCallback, ProtocolToken& alertToken)
   at System.Net.Security.SslState.CompleteHandshake(ProtocolToken& alertToken)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Security.SslState.ThrowIfExceptional()
   at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__47_1(IAsyncResult iar)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at console.Program.Main(String[] args) in /tmp/console/Program.cs:line 13
   at console.Program.<Main>(String[] args)

[tmds]

CheckValidOpenSslHandle throws in LoadMachineStores:

https://github.com/dotnet/corefx/blob/cf195c1d0b782e1c5af8af8cf73f57b01cb41d95/src/System.Security.Cryptography.X509Certificates/src/Internal/Cryptography/Pal.Unix/StorePal.cs#L253-L262

This was added in dotnet/corefx#29351

Instead of skipping the cert file we are now throwing.

The handle is not valid for files that the user can't read.

$ ls -lah /etc/pki/tls/certs/
total 24K
drwxr-xr-x. 2 root root 4.0K Apr 24 11:17 .
drwxr-xr-x. 5 root root 4.0K Apr  3 13:18 ..
-rw-r-----. 1 root root 1.5K Oct 28  2016 2015-RH-IT-Root-CA.pem
lrwxrwxrwx. 1 root root   49 Feb  6 14:48 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx. 1 root root   55 Feb  6 14:48 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
-rw-r--r--. 1 root root 2.5K Apr  3 13:18 Makefile
-rw-r-----. 1 root root 1.4K Oct 28  2016 newca.crt
-rw-r-----. 1 root root 1019 Oct 28  2016 oracle_ebs.crt

[stephentoub]

cc: @pjanotti

[tmds]

The files that trigger the issue are not part of the default system installation. So it does not affect every system. You need to have installed a package that brings such files. In my case, they come with the package that configures my system for access to the corporate vpn.

[danmoseley]

@tmds for shiproom do you have data suggesting how commonly systems are configured this way? Is there a feasible workaround? Do you know how long this was regressed?

[tmds]

for shiproom do you have data suggesting how commonly systems are configured this way?

For me it is due to the company vpn package adding certificates that are only readable by root.
I'm not sure if this is specific to the way we have packed up those certificates, or if you'd see this for other vpn packages too.
It won't affect many users.

Is there a feasible workaround?

The workaround is to change permissions on the system certificates. You need to have root permissions and be willing to change permissions on those files. This may have security side-effects.

Do you know how long this was regressed?

It regressed after 2.1-rc1.

If you're not including it for servicing we may patch our .NET Core builds for Fedora/RHEL/CentOS. Our preference is to minimize such differences, so that's why I'm asking here first.

CC @omajid

[danmoseley]

That's a very recent regression - you're sure the certificates were present at the time you tested with RC1?

@stephentoub any idea where the regression might have been?

[stephentoub]

any idea where the regression might have been?

https://github.com/dotnet/corefx/pull/29351/files#diff-28edeba8fcd86130f28b8c85afd19f37R257

[pjanotti]

I missed this earlier... yes, the recent change to keep the SSL error queue clean introduced this.

[saluton-mundo]

We've also encountered this issue on CentOS 7.5.1804. Our install of Apache (2.4.6) and mod_ssl created a default localhost.crt which the user we are running the site under did not have access to.

To avoid using a permissions workaround (running the site as root or giving read access to the certificate file) we simply moved the localhost.crt to a different folder and updated the Apache config.