Issues with using NSS PKCS#11 Provider in FIPS mode

[jkakavas]

NSS >= 3.27 cannot be used

When using libnss >= 3.27 loading the PKCS11 token fails with CKR_ATTRIBUTE_READ_ONLY errors. There is an open BUG JDK-8180837 that affects all Java versions and the RCA is not yet complete, nor is there an indication of when this will be resolved.

TLS1.2 cannot be used with PKCS11-NSS

See unresolved jdk bug

Loading system keystore/truststore from a PKCS11 token

PKCS#11 tokens can be used as keystores and truststores. To use a PKCS#11 token as the JDK Default keystore and truststore ( a.k.a. javax.net.ssl.keyStore and javax.net.ssk.TrustStore ), one needs to set jaxax.net.ssl.keyStore and javax.net.ssl.trustStore to NONE (case sensitive) according to the PKCS#11 Reference Guide.

We do not handle the NONE parameter correctly, and this gets interpreted as the name of the keystore and truststore, relevant to $ES_CONF_PATH.

Issue addressed in a PR shortly (See : #33460)

Multiple PKCS#11 keystores

We can't have multiple PKCS#11 keystores. PKCS#11 is supposed to be a wrapper to access HW/SW tokens and the token is configured in the provider's configuration that is referenced in java.security. As such, each JVM can only point to one token, i.e. one Keystore/Trustore. PKCS#11 KeyStores/TrustStores cannot be stored on disk and when loading the KeyStore via KeyStore.load() we need to pass null as the InputStream in order to access the token configured in the security properties.

In practice this means that keystores cannot be used with the NSS FIPS provider and only the System Truststore(PKCS#11 token) can be used, adding trusted certificates to it, as needed. This is NOT an option for trust configuration for TLS on the transport layer as it means that all certificates signed by the JVM trusted CAs will be able to connect to an Elasticsearch node.
It can't either be used for storing key material, unless we introduce the option to define the key alias to be selected from a keystore when a keystore is configured, instead of assuming there will be only one key as we do now. (This might be a nice todo irrespective to the FIPS effort)

This is not an issue per se, just a configuration limitation.

PKCS#11 tokens as password protected keystores

PKCS#11 tokens need to be password protected (NOTE: NSS tooling by default doesn't add a password to the db), otherwise loading the store fails with

javax.security.auth.login.LoginException: no password provided, and no callback handler available for retrieving password

PKCS#11 keystores cannot be loaded from disk and in effect one can only use a single store so this only applies to the default PKCS#11 token (accessed as the system default keystore) that needs to be password protected. The password for it is passed using javax.net.ssl.keyStorePassword and javax.net.ssl.TrustStorePassword that cannot be empty.
Again, a limitation rather than an issue.

Another way this affects us is that if no trust configuration is defined, we resolve to using the default JDK Trust Configuration and we do that by initializing an TrustManager with a null Keystore and no password. We need to detect that we're using a PKCS11-NSS Provider and that the default truststore is acually a PKCS#11 token and pass the javax.net.ssl.TrustStorePassword when loading the null Keystore.

Issue addressed in a PR shortly

In memory keystores

When using PEM files for key and trust material, we create in memory KeyStores (of applicable type - in this case PKCS#11) and load these keys and certificates so that we can then initialize TrustManager and KeyManager objects using the keystores. We pass null as the InputStream parameter to indicate that it should be initialized as an empty keystore, but in the case of PKCS#11 this will mean that the default PKCS#11 token will be loaded in the KeyStore. This, in turn, adds a number of complications:

• Adding a certificate using KeyStore#setCertificateEntry() doesn't work as the CKA_ID attribute which is required for the PKCS#11 format is not set, and this fails with an NPE. We could code around this if needed.
• The TrustManager that will be created, will be trusting all certificates that are in the default token, plus the PEM one we add, instead of just the one we add, which would be the intended behavior.
• The KeyManager will be created containing all certificate entries plus the private key entry we add. This is not problematic, unless we need to load many private key entries (one for transport TLS, one for http TLS, etc. ), when the problem becomes how to select the correct entry, which can be solved with specifying aliases in the configuration as mentioned above.

PEM files cannot be used as is.

PKCS#12 and FIPS 140 compliance

Looking for possible solutions for the above, I revisited why PKCS#12 keystores are not FIPS 140-2 compliant and came to the conclusion that a PKCS#12 keystore can be FIPS 140-2 compliant.

To give some context, the original idea is that a PKCS#12 is not FIPS 140-2 compliant
(for example BCFIPS disallows the use of PKCS12 stores in fips-approved mode) because the algorithms required for PBE key generation (for the key that is generated from the store's password and is used to encrypt the actual keys that are stored in the keystore) are not FIPS 140-2 compliant. However, reading through the RFC, PKCS#12 defines that PKCS#5 should be used and in particular

Specifically, PBES2 should be used as encryption scheme, with PBKDF2 as the key derivation function.

Now, PBES2 combines a password-based key derivation function (PBKDF2 is called out explicitly) with an underlying encryption scheme. Appendix B2 of RFC2898 defines DES-EDE3-CBC-Pad ( three-key triple-DES in CBC mode with the RFC 1423 padding operation) as one of the possible schemes of encryption. Triple-DES is a NIST Approved algorithm and as such FIPS 140-2 compliant. Additionally, common tooling (i.e. openssl) allows for creating PKCS#12 keystores with arbitrary (even not specified in PKCS#12 RFC) (FIPS 140-2 compliant) algorithms and encryption schemes.
For example

openssl pkcs12 -inkey key.pem -in certificate.pem -macalg SHA256 -keypbe AES-192-CBC -certpbe AES-256-CFB -export -out test.p12

will produce a PKCS#12 store with the following information

openssl pkcs12 -info -in test.p12 -noout

MAC:sha256 Iteration 2048
PKCS7 Encrypted data: PBES2, PBKDF2, AES-256-CFB, Iteration 2048, PRF hmacWithSHA256
Certificate bag
PKCS7 Data
Shrouded Keybag: PBES2, PBKDF2, AES-192-CBC, Iteration 2048, PRF hmacWithSHA256

which for all effects and purposes is FIPS 140-2 compliant since it only uses approved algorithms (AES for encryption and PBKDF2 as the key derivation function.

To remain compliant to PKCS#12 and PKCS#5 2.0, one could create a keystore with

openssl pkcs12 -inkey key.pem -in certificate.pem -macalg SHA256 -keypbe DES-EDE3-CBC -certpbe DES-EDE3-CBC -export -out test.p12

Granted, existing PKCS12 stores are not FIPS 140 compliant since the default algorithms schemes are pbeWithSHA1And3-KeyTripleDES-CBC for the inner (private key encryption) and pbeWithSHA1And40BitRC2-CBC for the outer (PKCS#7) encryption.

The problem with the above is that the KeyManagerImpl specifically requires that the keystore used to initialize a KeyManager, must be created by the same Security Provider that is actually being used

https://github.com/frohoff/jdk8u-jdk/blob/da0da73ab82ed714dc5be94acd2f0d00fbdfe2e9/src/share/classes/sun/security/ssl/KeyManagerFactoryImpl.java#L65

Unfortunately PKCS#12 stores created by openssl do not have a provider set and if they did, it wouldn't be the SunPKCS-NSS that will be in use in runtime, and using the SunPCKS11-NSS provider with keytool , i.e.

/usr/lib/jvm/java-8-openjdk-amd64-fips-nss/bin/keytool -genkeypair -keystore instance.p12 -storetype pkcs12 -storepass password -alias cert0 -keyalg RSA -keysize 2048 -validity 99999 -dname "CN=My SSL Certificate" -providerName SunPKCS11-NSS

yields an error as this provider can't handle PKCS12 keystores.

Summary

• NONE keyword needs to be handled accordingly ( Correctly handle PKCS#11 tokens for system keystore #33460)
• PKCS#12 keystores , JKS keystores and PEM files cannot be used for key configuration.
• PKCS#12 keystores can be used for trust configuration and as shown above can be FIPS 140 compliant.
• The HW/SW PKCS#11 token that is used as the system keystore and truststore must be password protected.

Actions

• Focus on supporting the use of PEM files for key configuration. This would require specific handling for adding keys and certificates to the PKCS#11 token and to allow to specify in configuration the alias that should be used to read a key from a PKCS#11 token that might contain multiple private key entries (edited see below for justification.)
• Allow for selecting a Security Provider to use ( and in PKCS#11 case, the associated keystore/token ) when configuring a keystore or key/certificate pair in any related configuration option.

[elasticmachine]

Pinging @elastic/es-security

[jaymode]

From the PKCS#11 guide:

To use a PKCS#11 token as a keystore or trust store, set the javax.net.ssl.keyStoreType and javax.net.ssl.trustStoreType system properties, respectively, to "PKCS11", and set the javax.net.ssl.keyStore and javax.net.ssl.trustStore system properties, respectively, to NONE. To specify the use of a specific provider instance, use the javax.net.ssl.keyStoreProvider and javax.net.ssl.trustStoreProvider system properties (e.g., "SunPKCS11-SmartCard").

What do you think about adding support for keyStoreProvider and trustStoreProvider?

[jkakavas]

JSSE assumes that there is only one PrivateKeyEntry in the keystore that is used to initialize the X509ExtendedKeyManager (or that we are only interested in the first entry), see https://github.com/frohoff/jdk8u-jdk/blob/da0da73ab82ed714dc5be94acd2f0d00fbdfe2e9/src/share/classes/sun/security/ssl/SunX509KeyManagerImpl.java#L50

The JSSE handshake code currently calls into this class via
chooseClientAlias() and chooseServerAlias() to find the certificates to
use. As implemented here, both always return the first alias returned by
getClientAliases() and getServerAliases(). In turn, these methods are
implemented by calling getAliases(), which performs the actual lookup.

If we were to support keystores that contain multiple PrivateKeyEntry and the possibility to select the appropriate alias to use via configuration (ie xpack.ssl.keystore.key_alias , then we'd have to implement our own KeyManager extending X509ExtendedKeyManager and implement the chooseServerAlias() method accordingly. Unfortunately we cannot do that as, as we figured out during the FIPS effort, we cannot use KeyManagers that extend the SunJSSE ones

Since we can only use onePKCS#11 Keystore per Provider with and the KeyStore can only contain one entry, we can effectively use only one Key throughout our configuration.

An alternative to the above is to support multiple Security Providers, configured in the Java security properties and use a different one for each (KeyManager-KeyStore-Key) that needs to be used in a configuration. This will require the introduction of an optional configuration setting that can be used wherever a certificate/key or a keystore can be configured in order to denote the provider ID to use for the KeyManagerFactory. We should probably limit this to be allowed only when PKCS#11 keystores/tokens are in use and also verify that a provider ID is valid when we parse the newly introduced setting's value.

[jkakavas]

It turns out that multiple PKCS#11 Security Providers cannot be used, as only one can be configured at a given time.
This basically means that there is no FIPS Compliant way to bypass the single keystore ( == single private key and certificate for TLS ) limitation.
A PR that allows the use of PKCS#11 tokens is opened : #34063

[jkakavas]

It turns out that multiple PKCS#11 Security Providers cannot be used, as only one can be configured at a given time.

That was misleading. Mulitple PKCS#11 (of type sun.security.pkcs11.SunPKCS11) can be configured both statically ( in java.security ) or dynamically. For example using the security.properties one can define:

...
...
security.provider.10=sun.security.pkcs11.SunPKCS11 nss.cfg
security.provider.11=sun.security.pkcs11.SunPKCS11 nss2.cfg

with nss.cfg and nss.cfg pointing to different databases ( or HW tokens ), ie.

name=NSS
nssLibraryDirectory=/usr/lib/x86_64-linux-gnu
nssSecmodDirectory=/path/to/nss/fips/db
nssModule=fips

and

name=NSS
nssLibraryDirectory=/usr/lib/x86_64-linux-gnu
nssSecmodDirectory=/path/to/nss2/fips/db
nssModule=fips

What cannot be done is to initialize both. An error:

Secmod directory /path/to/nss2/fips/db invalid, NSS already initialized with /path/to/nss/fips/db

is thrown. See

https://github.com/frohoff/jdk8u-jdk/blob/master/src/share/classes/sun/security/pkcs11/SunPKCS11.java#L172

A minimal example that reproduces the above behavior is:

public class MultipleNssProviders {
    private static final String PROVIDER_ID_1 = "SunPKCS11-NSS";
    private static final String PROVIDER_ID_2 = "SunPKCS11-NSSTrust";
    private static final String KEYSTORE_TYPE = "PKCS11";
    private static final String KEYSTORE_PWD = "ThePassword";

    public static void main(String[] args) throws Exception{
        KeyStore ks1 = KeyStore.getInstance(KEYSTORE_TYPE, PROVIDER_ID_1);
        ks1.load(null, KEYSTORE_PWD.toCharArray());
        KeyStore ks2 = KeyStore.getInstance(KEYSTORE_TYPE, PROVIDER_ID_2);
        // throws java.security.ProviderException
        ks2.load(null, KEYSTORE_PWD.toCharArray());
    }
}

As such we cannot use two PKCS#11 tokens simultaneously , i.e one for the http layer keystore and one for the transport layer one.

[jaymode]

That is really an unfortunate limitation. I agree then we don't need the provider support.