Allow PKI backend to use PKCS#8 format instead of (in addition to?) PKCS#1 for private keys

[jchambers]

Unless I'm missing something (which is very possible!), it seems like the PKI backend is always going to send private keys to clients as PEM-encoded PKCS#1 (or "traditional format") files. It seems that PKCS#8 is actually the hip new thing. OpenSSL's docs recommend using PKCS#8 instead of PKCS#1:

Note this command uses the traditional SSLeay compatible format for private key encryption: newer applications should use the more secure PKCS#8 format using the pkcs8 utility.

Additionally, Java (and probably others?) includes support for PKCS#8-encoded keys, but not PKCS#1. I'd like to request a feature where the PKI backend can return PEM-encoded PKCS#8 private keys in addition to or instead of PEM-encoded PKCS#1 keys.

Some additional background for readers who may not already be knee-deep in this stuff: we'd still be using the same key, it'd just be formatted a little differently. PEM-encoded PKCS#1 keys begin with -----BEGIN RSA PRIVATE KEY-----, while PEM-encoded PKCS#8 keys begin with -----BEGIN PRIVATE KEY-----.

Thanks!

[jefferai]

The backend supports reading PKCS8, but what exactly is the use-case here? Are you seeing software in the wild that is only able to read PKCS8 but not PKCS1?

PKCS8 is only more secure if you're encrypting the data. Vault doesn't generate encrypted PEM blobs (there would be no point, since it'd have to send the password in the same JSON blob over the wire).

[jchambers]

Are you seeing software in the wild that is only able to read PKCS8 but not PKCS1?

Yes, exactly. We certainly have workarounds (make a call to openssl to convert from PKCS1 to PKCS8), but they're a bit inconvenient. I should clarify that this is 100% a convenience thing and not a security thing.

[jefferai]

Can I just ask what, so I can file it away into my knowledge-hole? It's good to be aware of this kind of thing.

[jchambers]

Oh, sorry! Didn't mean to be vague. It's, uh, Java. All of it ;)

The Java SE API has a PKCS8EncodedKeySpec, but no good equivalent for PKCS1. A number of third-party libraries provide tools for working with PKCS1 keys, but it'd be cool to not have to add new dependencies.

[jchambers]

…and, more specifically, we're working with gRPC, which wants a "private key PEM file," but it turns out that under the hood, it'll only work with PKCS#8 keys. I emphasize again that we have workarounds and gRPC's internals are definitely not your problem, but offer it for context.

[jefferai]

I don't think gRPC is the problem here; more likely that you're using gRPC on Java. gRPC tends to use language-specific methods of acquiring certs and other information.

Good on Java for ignoring widely-supported standards in favor of something random.

[jefferai]

FWIW, you don't even need to use openssl to convert. You literally just need to do a string replacement on the PEM header.

[jchambers]

FWIW, you don't even need to use openssl to convert. You literally just need to do a string replacement on the PEM header.

With respect, I'm not sure if that's true in the general case. To test, I generated a private key with openssl genrsa -out pkcs1.pem, which gives us:

-----BEGIN RSA PRIVATE KEY-----
MIIBOwIBAAJBANxybyAkx0HtoT2VcT+5HRDFj0oeuMfmjgt1V8T6qrr5WRq5H1z8
M4Oxe4k6x83hc5DDsacFJHY6MZB2Phe7qFMCAwEAAQJBAKnEHrwcVq7492FbwvkV
78JIwk6MVjTr4pwEY/E4MV8G1ZjaJBDaLBzsJ0b3XkrZw6gW9ugvwQqdmDPYrqqY
rhECIQDyBPPKqWF4YJ836HxsU/Y0Z80TQhaefTFfk8dx7u5WdwIhAOkudtZN8ZrU
tnl1t4YKw4m758MNGJjZn5iNCVspccgFAiAZpKywLIJHXvV/1wJX+A1+QAFaqQX8
hQdWOTJIwChAwQIhAM1rF1l3VK43UYDyiPhWpyfBF0ynTtHodUYoiON8QTYZAiAm
601ZuUhE+fXHrS28pzaxNw3jTEBEnBRH+eirjpx6vQ==
-----END RSA PRIVATE KEY-----

Trying to read it as a PKCS8 file doesn't work right off the bat, as expected:

$ openssl pkcs8 -nocrypt -in pkcs1.pem 
Error decrypting key
51124:error:0906D06C:PEM routines:PEM_read_bio:no start line:/BuildRoot/Library/Caches/com.apple.xbs/Sources/OpenSSL098/OpenSSL098-64/src/crypto/pem/pem_lib.c:648:Expecting: PRIVATE KEY

Then I replaced BEGIN/END RSA PRIVATE KEY with BEGIN/END PRIVATE KEY:

$ openssl pkcs8 -nocrypt -in pkcs1.pem 
Error decrypting key
50901:error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:/BuildRoot/Library/Caches/com.apple.xbs/Sources/OpenSSL098/OpenSSL098-64/src/crypto/asn1/tasn_dec.c:1344:
50901:error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:/BuildRoot/Library/Caches/com.apple.xbs/Sources/OpenSSL098/OpenSSL098-64/src/crypto/asn1/tasn_dec.c:387:Type=X509_ALGOR
50901:error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:/BuildRoot/Library/Caches/com.apple.xbs/Sources/OpenSSL098/OpenSSL098-64/src/crypto/asn1/tasn_dec.c:768:Field=pkeyalg, Type=PKCS8_PRIV_KEY_INFO
50901:error:0906700D:PEM routines:PEM_ASN1_read_bio:ASN1 lib:/BuildRoot/Library/Caches/com.apple.xbs/Sources/OpenSSL098/OpenSSL098-64/src/crypto/pem/pem_oth.c:83:

So there's definitely something structural about the content that's different. If we ask OpenSSL to convert to a PKCS8 private key, everything is fine:

$ openssl pkcs8 -topk8 -nocrypt -in pkcs1.pem 
-----BEGIN PRIVATE KEY-----
MIIBVQIBADANBgkqhkiG9w0BAQEFAASCAT8wggE7AgEAAkEA3HJvICTHQe2hPZVx
P7kdEMWPSh64x+aOC3VXxPqquvlZGrkfXPwzg7F7iTrHzeFzkMOxpwUkdjoxkHY+
F7uoUwIDAQABAkEAqcQevBxWrvj3YVvC+RXvwkjCToxWNOvinARj8TgxXwbVmNok
ENosHOwnRvdeStnDqBb26C/BCp2YM9iuqpiuEQIhAPIE88qpYXhgnzfofGxT9jRn
zRNCFp59MV+Tx3Hu7lZ3AiEA6S521k3xmtS2eXW3hgrDibvnww0YmNmfmI0JWylx
yAUCIBmkrLAsgkde9X/XAlf4DX5AAVqpBfyFB1Y5MkjAKEDBAiEAzWsXWXdUrjdR
gPKI+FanJ8EXTKdO0eh1RiiI43xBNhkCICbrTVm5SET59cetLbynNrE3DeNMQESc
FEf56KuOnHq9
-----END PRIVATE KEY-----

Taking it one step further, if we run the two private keys through an ASN.1 decoder, we see that the PKCS1 data looks like this:

SEQUENCE (9 elem)
    INTEGER 0
    INTEGER (512 bit) 1154574665582692612624485476148287950912867945632723620269744370043575…
    INTEGER 65537
    INTEGER (512 bit) 8891371800961059578430166467980544254783167556660140243681279026519256…
    INTEGER (256 bit) 1094684593335189506116646701915991020538498901518504493119673805123258…
    INTEGER (256 bit) 1054709888685868191386453447511853880265483304311328541769865614741121…
    INTEGER (253 bit) 1159877598321491055506659550804071234053944546156292659142623999080810…
    INTEGER (256 bit) 9291334774766912455789173037925045687664588414347657358521357615005862…
    INTEGER (254 bit) 1760363115980350320698367789927621772628435898551421235596133963997109…

…while the PKCS8 data looks like this:

SEQUENCE (3 elem)
    INTEGER 0
    SEQUENCE (2 elem)
        OBJECT IDENTIFIER 1.2.840.113549.1.1.1
        NULL
    OCTET STRING (1 elem)
        SEQUENCE (9 elem)
            INTEGER 0
            INTEGER (512 bit) 1154574665582692612624485476148287950912867945632723620269744370043575…
            INTEGER 65537
            INTEGER (512 bit) 8891371800961059578430166467980544254783167556660140243681279026519256…
            INTEGER (256 bit) 1094684593335189506116646701915991020538498901518504493119673805123258…
            INTEGER (256 bit) 1054709888685868191386453447511853880265483304311328541769865614741121…
            INTEGER (253 bit) 1159877598321491055506659550804071234053944546156292659142623999080810…
            INTEGER (256 bit) 9291334774766912455789173037925045687664588414347657358521357615005862…
            INTEGER (254 bit) 1760363115980350320698367789927621772628435898551421235596133963997109…

So it's the same key, but nested deeper inside a more complicated data structure. There may be tools smart enough to infer the format of a key, but it looks like OpenSSL isn't one of them.

[jefferai]

Yeah, I misspoke, I forgot that it's a different function call in the Go libraries.

[spodkowinski]

Missing PKCS8 support makes it really painful to work with Vault keys on the Java platform. Looks like a MarshalPKCS8PrivateKey will be added in Go 1.10 (see golang/go@0a1a65c), which also has already been ported into Packer.

[jefferai]

Closing as Vault has supported this for four months now (see #3518)