"The certificate data cannot be read with the provided password, the password may be incorrect." in `X509Certificate2` after updating to .NET 5.0

[codeaid]

Description

I upgraded my application earlier today to .NET 5.0 and everything worked well apart from one thing - I'm getting the error specified in the title of this issue as soon I start the project up (through dotnet run) and get to the point of instantiating an X509 certificate for use with IdentityServer4. The certificate file is the same one I had before the upgrade and hasn't been changed for weeks.

The key file was originally generated using the following commands by accepting all defaults (just Enter, Enter, Enter...):

openssl req \
    -x509 \
    -newkey rsa:4096 \
    -sha256 \
    -nodes \
    -days 3650 \
    -keyout .MyCertificate.key \
    -out .MyCertificate.crt

openssl pkcs12 \
    -export \
    -in .MyCertificate.crt \
    -inkey .MyCertificate.key \
    -certfile .MyCertificate.crt \
    -out .MyCertificate.pfx

rm .MyCertificate.crt .MyCertificate.key

...which is then added to the Identity Server in the Startup.cs file:

public void ConfigureServices(IServiceCollection services)
{
    IIdentityServerBuilder builder = services.AddIdentityServer(options => ...);

    string certPath = Path.Combine(Environment.ContentRootPath, "Certificates", "MyCertificate.pfx");

    // This is the line that fails
    builder.AddSigningCredential(new X509Certificate2(certPath));

    // Changing it to just the instantiation (without calling `AddSigningCredential`) results in the same exception:
    // new X509Certificate2(certPath)
}

Configuration

Running:

• .NET 5.0
• Ubuntu 20.04 (5.4.0-52-generic #57-Ubuntu SMP Thu Oct 15 10:57:00 UTC 2020 x86_64 x86_64 x86_64)

Regression?

This worked in 3.1 with no issues even earlier today before the upgrade. Tested a moment ago by reverting to 3.1 just to confirm.

Other information

Here's the console output of the exception:

Host terminated unexpectedly
System.Security.Cryptography.CryptographicException: The certificate data cannot be read with the provided password, the password may be incorrect.
 ---> System.Security.Cryptography.CryptographicException: A certificate referenced a private key which was already referenced, or could not be loaded.
   at Internal.Cryptography.Pal.UnixPkcs12Reader.BuildCertsWithKeys(CertBagAsn[] certBags, AttributeAsn[][] certBagAttrs, CertAndKey[] certs, Int32 certBagIdx, SafeBagAsn[] keyBags, RentedSubjectPublicKeyInfo[] publicKeyInfos, AsymmetricAlgorithm[] keys, Int32 keyBagIdx)
   at Internal.Cryptography.Pal.UnixPkcs12Reader.Decrypt(ReadOnlySpan`1 password, ReadOnlyMemory`1 authSafeContents)
   at Internal.Cryptography.Pal.UnixPkcs12Reader.VerifyAndDecrypt(ReadOnlySpan`1 password, ReadOnlyMemory`1 authSafeContents)
   at Internal.Cryptography.Pal.UnixPkcs12Reader.Decrypt(SafePasswordHandle password)
   --- End of inner exception stack trace ---
   at Internal.Cryptography.Pal.UnixPkcs12Reader.Decrypt(SafePasswordHandle password)
   at Internal.Cryptography.Pal.PkcsFormatReader.TryReadPkcs12(OpenSslPkcs12Reader pfx, SafePasswordHandle password, Boolean single, ICertificatePal& readPal, List`1& readCerts)
   at Internal.Cryptography.Pal.PkcsFormatReader.TryReadPkcs12(ReadOnlySpan`1 rawData, SafePasswordHandle password, Boolean single, ICertificatePal& readPal, List`1& readCerts, Exception& openSslException)
   at Internal.Cryptography.Pal.OpenSslX509CertificateReader.FromFile(String fileName, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags)
   at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(String fileName, String password, X509KeyStorageFlags keyStorageFlags)
   at System.Security.Cryptography.X509Certificates.X509Certificate2..ctor(String fileName, String password)
   at MyProject.Startup.ConfigureIdentityServer(IServiceCollection services) in /var/www/MyProject/Startup.cs:line 197
   at MyProject.Startup.ConfigureServices(IServiceCollection services) in /var/www/MyProject/Startup.cs:line 53
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass9_0.<Invoke>g__Startup|0(IServiceCollection serviceCollection)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass8_0.<Build>b__0(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(Type startupType, HostBuilderContext context, IServiceCollection services, Object instance)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass13_0.<UseStartup>b__0(HostBuilderContext context, IServiceCollection services)
   at Microsoft.Extensions.Hosting.HostBuilder.CreateServiceProvider()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at MyProject.Program.Main(String[] args) in /var/www/MyProject/Program.cs:line 23

Tagging subscribers to this area: @bartonjs, @vcsjones, @krwq, @jeffhandley
See info in area-owners.md if you want to be subscribed.

---

Issue meta data

Issue content:	### Description I upgraded my application earlier today to .NET 5.0 and everything worked well apart from one thing - I'm getting the error specified in the title of this issue as soon I start the project up (through dotnet run) and get to the point of instantiating an X509 certificate for use with IdentityServer4. The certificate file is the same one I had before the upgrade and hasn't been changed for weeks. The key file was originally generated using the following commands by accepting all defaults (just Enter, Enter, Enter...): openssl req \ -x509 \ -newkey rsa:4096 \ -sha256 \ -nodes \ -days 3650 \ -keyout .MyCertificate.key \ -out .MyCertificate.crt openssl pkcs12 \ -export \ -in .MyCertificate.crt \ -inkey .MyCertificate.key \ -certfile .MyCertificate.crt \ -out .MyCertificate.pfx rm .MyCertificate.crt .MyCertificate.key ...which is then added to the Identity Server in the Startup.cs file: public void ConfigureServices(IServiceCollection services) { IIdentityServerBuilder builder = services.AddIdentityServer(options => ...); string certPath = Path.Combine(Environment.ContentRootPath, "Certificates", "MyCertificate.pfx"); // This is the line that fails builder.AddSigningCredential(new X509Certificate2(certPath)); // Changing it to just the instantiation (without calling `AddSigningCredential`) results in the same exception: // new X509Certificate2(certPath) } Configuration Running: .NET 5.0 Ubuntu 20.04 (5.4.0-52-generic #57-Ubuntu SMP Thu Oct 15 10:57:00 UTC 2020 x86_64 x86_64 x86_64) Regression? This worked in 3.1 with no issues even earlier today before the upgrade. Tested a moment ago by reverting to 3.1 just to confirm. Other information Here's the console output of the exception: Host terminated unexpectedly System.Security.Cryptography.CryptographicException: The certificate data cannot be read with the provided password, the password may be incorrect. ---> System.Security.Cryptography.CryptographicException: A certificate referenced a private key which was already referenced, or could not be loaded. at Internal.Cryptography.Pal.UnixPkcs12Reader.BuildCertsWithKeys(CertBagAsn[] certBags, AttributeAsn[][] certBagAttrs, CertAndKey[] certs, Int32 certBagIdx, SafeBagAsn[] keyBags, RentedSubjectPublicKeyInfo[] publicKeyInfos, AsymmetricAlgorithm[] keys, Int32 keyBagIdx) at Internal.Cryptography.Pal.UnixPkcs12Reader.Decrypt(ReadOnlySpan`1 password, ReadOnlyMemory`1 authSafeContents) at Internal.Cryptography.Pal.UnixPkcs12Reader.VerifyAndDecrypt(ReadOnlySpan`1 password, ReadOnlyMemory`1 authSafeContents) at Internal.Cryptography.Pal.UnixPkcs12Reader.Decrypt(SafePasswordHandle password) --- End of inner exception stack trace --- at Internal.Cryptography.Pal.UnixPkcs12Reader.Decrypt(SafePasswordHandle password) at Internal.Cryptography.Pal.PkcsFormatReader.TryReadPkcs12(OpenSslPkcs12Reader pfx, SafePasswordHandle password, Boolean single, ICertificatePal& readPal, List`1& readCerts) at Internal.Cryptography.Pal.PkcsFormatReader.TryReadPkcs12(ReadOnlySpan`1 rawData, SafePasswordHandle password, Boolean single, ICertificatePal& readPal, List`1& readCerts, Exception& openSslException) at Internal.Cryptography.Pal.OpenSslX509CertificateReader.FromFile(String fileName, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags) at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(String fileName, String password, X509KeyStorageFlags keyStorageFlags) at System.Security.Cryptography.X509Certificates.X509Certificate2..ctor(String fileName, String password) at MyProject.Startup.ConfigureIdentityServer(IServiceCollection services) in /var/www/MyProject/Startup.cs:line 197 at MyProject.Startup.ConfigureServices(IServiceCollection services) in /var/www/MyProject/Startup.cs:line 53 at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions) at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services) at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass9_0.<Invoke>g__Startup|0(IServiceCollection serviceCollection) at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection services) at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass8_0.<Build>b__0(IServiceCollection services) at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(Type startupType, HostBuilderContext context, IServiceCollection services, Object instance) at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass13_0.<UseStartup>b__0(HostBuilderContext context, IServiceCollection services) at Microsoft.Extensions.Hosting.HostBuilder.CreateServiceProvider() at Microsoft.Extensions.Hosting.HostBuilder.Build() at MyProject.Program.Main(String[] args) in /var/www/MyProject/Program.cs:line 23
Issue author:	codeaid
Assignees:	-
Milestone:	-

[vcsjones]

This is because there are two (identical) certificates in the PKCS12 bag that are referencing the same key. If you change the OpenSSL command to not include the certificate twice, then the issue no longer occurs:

openssl pkcs12 \
    -export \
    -inkey .MyCertificate.key \
    -in .MyCertificate.crt \
    -out .MyCertificate.pfx

(Note the removal of -certfile parameter.) This will work in .NET Core 3.1 and .NET 5.

As for why this changed / regressed in .NET 5 I am not sure yet, but I don't think it was an intentional change, though @bartonjs is probably the right person to say for certain.

[vcsjones]

As for why this changed / regressed in .NET 5 I am not sure yet

Ah, this is because UnixPkcs12Reader didn't exist in .NET Core 3.1, it looks like the PKCS12 handling was still being done by native OpenSSL. I confused myself by thinking the native to managed change happened between 2.1 and 3.1.

@bartonjs it looks like the managed implemented explicitly only allows a key to be used once. Once it's used, it's marked as null to make the key ineligible for use again. The behavior seems intentional then, but it diverges from what OpenSSL appears to allow.

runtime/src/libraries/System.Security.Cryptography.X509Certificates/src/Internal/Cryptography/Pal.Unix/UnixPkcs12Reader.cs

Lines 553 to 559 in 1923bba

	if (keys[matchingKeyIdx] == null)
	{
	throw new CryptographicException(SR.Cryptography_Pfx_BadKeyReference);
	}
	certs[certBagIdx].Key = keys[matchingKeyIdx];
	keys[matchingKeyIdx] = null;

[bartonjs]

Part of the purpose of that change was to be consistent across the three OS families. Presumably that means that Windows would fail to read a PFX in that state.

[codeaid]

(Note the removal of -certfile parameter.) This will work in .NET Core 3.1 and .NET 5.

Can confirm that this does indeed fix the issue. Regenerated the certificate file using the same commands minus the -certfile parameter and it now loads the certificate without any problems.

[vcsjones]

@bartonjs

Presumably that means that Windows would fail to read a PFX in that state.

Hm, this particular example appears to work in Windows, at least on Windows 10 2004.

This test seems to support that Windows doesn't like two certificates and one key:

runtime/src/libraries/System.Security.Cryptography.X509Certificates/tests/PfxFormatTests.cs

Line 783 in e33ba5d

public void CertTwice_KeyOnce(bool addLocalKeyId)

But something more nuanced seems to be going on, I shall dig in.

@codeaid

Can confirm that this does indeed fix the issue.

Great! Does this mean this particular issue is no longer blocking you from moving to .NET 5?

[codeaid]

@vcsjones It does! I don't have too many dependencies in my project but they all upgraded without any issues apart from Pomelo.EntityFrameworkCore.MySql, for which I need to use an alpha version until they release the final one.

Thanks all for the great work you're doing! Really appreciate that. Makes using everything .NET a joy, especially coming from other languages.

[mano-cz]

We have same issue with password protected ceritificate on our linux azure pipeline but it works on windows dev environment.

[vcsjones]

@mano-cz Are you getting the exact same error? In particular, is this the exception message of the inner exception?

System.Security.Cryptography.CryptographicException: A certificate referenced a private key which was already referenced, or could not be loaded.

Does the work around work for you in the meantime?

[mano-cz]

@mano-cz Are you getting the exact same error? In particular, is this the exception message of the inner exception?

System.Security.Cryptography.CryptographicException: A certificate referenced a private key which was already referenced, or could not be loaded.

Does the work around work for you in the meantime?
Yes the message is same.
System.Security.Cryptography.CryptographicException : The certificate data cannot be read with the provided password, the password may be incorrect. ----> System.Security.Cryptography.CryptographicException : A certificate referenced a private key which was already referenced, or could not be loaded. Stack Trace: at Internal.Cryptography.Pal.UnixPkcs12Reader.Decrypt(SafePasswordHandle password) at Internal.Cryptography.Pal.PkcsFormatReader.TryReadPkcs12(OpenSslPkcs12Reader pfx, SafePasswordHandle password, Boolean single, ICertificatePal& readPal, List1& readCerts)
at Internal.Cryptography.Pal.PkcsFormatReader.TryReadPkcs12(ReadOnlySpan1 rawData, SafePasswordHandle password, Boolean single, ICertificatePal& readPal, List1& readCerts, Exception& openSslException)
at Internal.Cryptography.Pal.OpenSslX509CertificateReader.FromBlob(ReadOnlySpan1 rawData, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags) at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(Byte[] rawData, String password, X509KeyStorageFlags keyStorageFlags) at System.Security.Cryptography.X509Certificates.X509Certificate2..ctor(Byte[] rawData, String password) ...<our code>... --CryptographicException at Internal.Cryptography.Pal.UnixPkcs12Reader.BuildCertsWithKeys(CertBagAsn[] certBags, AttributeAsn[][] certBagAttrs, CertAndKey[] certs, Int32 certBagIdx, SafeBagAsn[] keyBags, RentedSubjectPublicKeyInfo[] publicKeyInfos, AsymmetricAlgorithm[] keys, Int32 keyBagIdx) at Internal.Cryptography.Pal.UnixPkcs12Reader.Decrypt(ReadOnlySpan1 password, ReadOnlyMemory1 authSafeContents) at Internal.Cryptography.Pal.UnixPkcs12Reader.VerifyAndDecrypt(ReadOnlySpan1 password, ReadOnlyMemory1 authSafeContents) at Internal.Cryptography.Pal.UnixPkcs12Reader.Decrypt(SafePasswordHandle password)
This is a "unit test" with saved certificate, core3.1 was ok. But our customers may have issues with their certificates. We try the workaround tomorrow.

[vcsjones]

This test seems to support that Windows doesn't like two certificates and one key:

Okay, I dug a bit more at this and I'm less clear what is going on, but it seems that Windows permits this as well, unless you are importing with X509KeyStorageFlags.EphemeralKeySet, then it fails?

e.g. this will fail:

using System;
using System.Security.Cryptography;
using System.Security.Cryptography.Pkcs;
using System.Security.Cryptography.X509Certificates;

using RSA rsa = RSA.Create();
CertificateRequest req = new CertificateRequest("CN=blah", rsa, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
using X509Certificate2 cert = req.CreateSelfSigned(DateTimeOffset.Now, DateTimeOffset.Now);

Pkcs12Builder builder = new Pkcs12Builder();
Pkcs12SafeContents keyContents = new Pkcs12SafeContents();
Pkcs12SafeContents certContents = new Pkcs12SafeContents();

string password = "YabbaDabbaDoo";
PbeParameters pbeParameters = new PbeParameters(PbeEncryptionAlgorithm.TripleDes3KeyPkcs12, HashAlgorithmName.SHA1, 2000);

Pkcs12SafeBag keyBag = keyContents.AddShroudedKey(rsa, password, pbeParameters);
Pkcs12SafeBag certBag = certContents.AddCertificate(cert);
Pkcs12SafeBag certBag2 = certContents.AddCertificate(cert);

builder.AddSafeContentsEncrypted(keyContents, password, pbeParameters);
builder.AddSafeContentsUnencrypted(certContents);

builder.SealWithMac(password, HashAlgorithmName.SHA1, 2000);

byte[] pfxBytes = builder.Encode();

X509KeyStorageFlags flags = X509KeyStorageFlags.EphemeralKeySet;
using X509Certificate2 reOpened = new X509Certificate2(pfxBytes, password, flags);

but change the flags to DefaultKeySet, and it won't fail.

@bartonjs any clues what might be the cause of that behavior?

I don't know that the CertTwice_KeyOnce is passing (failing to open the PFX) for the reason we think it is.