Encryption with GnuPG 2.3 (RFC4880bis) causes compatibility issues with GnuPG 2.2

[cameronkerrnz]

I'm in the unhappy situation of trying to get my GPG keypair from Windows bind-mounted into a Docker container so I can use ksops (Kustomize-SOPS) on Windows (because kustomize+ksops on Windows is not a happy path, although sops works wonderfully well)

If I take a SOPS encrypted file (encrypted with just GPG; haven't tried anything else), and attempt to decrypt it in a container that has GPG but does not have my keypair, then I get a very unhelpful error message.

root@dafc786e620b:/work# sops --decrypt kustomize/overrides/demo/secrets/secrets.py.sops
Error decrypting tree: Error walking tree: Could not decrypt value: crypto/aes: invalid key size 0

The same command works well from outside the container on Windows where my key material is present (but note that this key is not present on any keyserver, as it was generated just for a demonstration purpose).

With debugging:

root@dafc786e620b:/work# sops --verbose --decrypt kustomize/overrides/demo/secrets/secrets.py.sops
[PGP]    INFO[0000] Decryption succeeded                          fingerprint=6CBBBFF08395F8E19F185780222A65AD721294E8
[SOPS]   INFO[0000] Data key recovered successfully
[SOPS]   DEBU[0000] Decrypting tree
Error decrypting tree: Error walking tree: Could not decrypt value: crypto/aes: invalid key size 0

Testing with just some gpg, I see the problem seems to be there, and makes me think that sops suffers from gpg compatibility issues, particularly when a GPG key was generated on a newer (2.3 in my case) version of GPG, and also used on earlier versions (2.2 in my case)

From Windows (where I generated this keypair), encrypting a simple message to myself:

❯ gpg --version
gpg (GnuPG) 2.3.1
libgcrypt 1.9.3
Copyright (C) 2021 g10 Code GmbH
License GNU GPL-3.0-or-later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: C:\Users\kerca54p\AppData\Roaming\gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
AEAD: EAX, OCB
Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

❯ gpg -k
C:\Users\kerca54p\AppData\Roaming\gnupg\pubring.kbx
---------------------------------------------------
pub   ed25519 2021-06-29 [SC]
      6CBBBFF08395F8E19F185780222A65AD721294E8
uid           [ultimate] SOPS Testing (Throwaway key for testing SOPS/KSOPS) <cameron.kerr.nz+sops-testing@gmail.com>
sub   cv25519 2021-06-29 [E]

❯ gpg --encrypt --recipient 6CBBBFF08395F8E19F185780222A65AD721294E8 --verbose --output test.txt.enc test.txt.in
gpg: Note: RFC4880bis features are enabled.
gpg: using subkey 1B5CBE5831039829 instead of primary key 222A65AD721294E8
gpg: using pgp trust model
gpg: This key belongs to us
gpg: reading from 'test.txt.in'
File 'test.txt.enc' exists. Overwrite? (y/N) y
gpg: writing to 'test.txt.enc'
gpg: ECDH/AES256.OCB encrypted for: "1B5CBE5831039829 SOPS Testing (Throwaway key for testing SOPS/KSOPS) <cameron.kerr.nz+sops-testing@gmail.com>"

Decrypting the same, on the same machine (to test the round-trip)

❯ gpg --decrypt --output test.txt.out test.txt.enc
gpg: encrypted with cv25519 key, ID 1B5CBE5831039829, created 2021-06-29
      "SOPS Testing (Throwaway key for testing SOPS/KSOPS) <cameron.kerr.nz+sops-testing@gmail.com>"

I can verify that text.txt.out matches test.txt.in

This very same file text.txt.enc, being in a directory that is bind-mounted into a Debian Buster container with an earlier version of GPG:

root@dafc786e620b:/work# gpg --version
gpg (GnuPG) 2.2.12
libgcrypt 1.8.4
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /root/.gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

root@dafc786e620b:/work# gpg --list-secret-keys
/root/.gnupg/pubring.kbx
------------------------
sec   ed25519 2021-06-29 [SC]
      6CBBBFF08395F8E19F185780222A65AD721294E8
uid           [ultimate] SOPS Testing (Throwaway key for testing SOPS/KSOPS) <cameron.kerr.nz+sops-testing@gmail.com>
ssb   cv25519 2021-06-29 [E]

root@dafc786e620b:/work# gpg --decrypt -vvd --output test.txt.out-buster test.txt.enc
# off=0 ctb=84 tag=1 hlen=2 plen=94
:pubkey enc packet: version 3, algo 18, keyid 1B5CBE5831039829
        data: [263 bits]
        data: [392 bits]
gpg: public key is 1B5CBE5831039829
gpg: no running gpg-agent - starting '/usr/bin/gpg-agent'
gpg: waiting for the agent to come up ... (5s)
gpg: connection to agent established
gpg: using subkey 1B5CBE5831039829 instead of primary key 222A65AD721294E8
gpg: pinentry launched (938 curses 1.1.0 /dev/pts/0 xterm -)
gpg: public key encrypted data: good DEK
# off=96 ctb=d4 tag=20 hlen=2 plen=101 new-ctb
:unknown packet: type 20, length 101
dump: 01 09 02 15 1c 24 82 e3  4b a1 f6 2e 99 b5 4a d3  bc 00 56 76 14 05 7c 06
  24: 5d 1b c1 f9 b3 de bf de  88 6c 1f 93 43 03 e9 ac  9e d9 93 c9 40 93 95 16
  48: 3f 3b 90 d6 f4 17 2f ac  5e 3c e5 e9 aa 01 e5 39  a1 19 69 4b d4 2c 95 b0
  72: 25 64 2a cd dc 82 76 65  95 5e 69 14 8d 3f 38 5b  55 b5 4e 4e d3 17 ff e7
  96: 90 61 53 81 b4

A packet tag of 20 seems new (https://datatracker.ietf.org/doc/html/rfc4880#section-4.3 stops at 19), and tag 20 seems to be https://tools.ietf.org/id/draft-ietf-openpgp-rfc4880bis-06.html#rfc.section.5.16, and this would tally with the Note in the output:

gpg: Note: RFC4880bis features are enabled.

If I encrypt a document in GPG 2.2, then I can happily decrypt it in GPG 2.2 and in GPG 2.3. So its not a matter of where the key is created, but the (implicit) settings used at encrypt-time.

Testing this hypothesis in sops, to evaluate whether encryption by sops (with GPG 2.2) will decrypt with the same version of sops (with GPG 2.3), and I can confirm that it does.

So it seems some consideration needs to be given to how sops should encrypt content with regard to compatibility in decryption, as this could easily lock out team members from decrypting content if it gets encrypted by someone with a newer version.

https://www.gnupg.org/documentation/manuals/gnupg/Compliance-Options.html

... path of least resistance might just be to specify '--rfc4880' if calling out to gpg directly? This makes my problem go away.

[cameronkerrnz]

Ah, that was easy to find, the fix would appear to be to add --rfc4880 to https://github.com/mozilla/sops/blob/master/pgp/keysource.go#L65-L74

[cameronkerrnz]

pull request #910 submitted

[jarrettprosser]

I also came across this issue when using sops 3.7.1/gpg 2.3.1 on Mac to encrypt files, trying to decrypt files in Ubuntu with gpg 2.2.20. Glad to see there's a fix in the pipeline!

[EdgeJ]

Until #924 can be merged up, there's another workaround I've found to this issue. The main problem with compatibility between RFC4880 and RFC4880bis is in how the keys themselves are generated. I've found that when generating new keys to use with SOPS on a Mac with GPG v2.3, passing the --rfc4880 flag when generating the key itself eliminates this problem (effectively emulating the way keys are generated with GPG 2.2). You can also edit already existing keys to change their preferences to match the RFC4880 standard, if you desire to keep the same fingerprint. This, of course, requires re-encrypting all secrets with the new key.

[cameronkerrnz]

[...] use with SOPS on a Mac with GPG v2.3, passing the --rfc4880 flag when generating the key [...]

Good to know that this affects GPG v2.3 on a Mac as well, and is not specific to Debian Buster. Thanks.

[badrow]

Same problem with Fedora 35 which uses gpg version 2.3.3

[neoakris]

This may be controversial, but it might be worth giving serious consideration of the next version of sops just dropping support for gpg entirely.

• It'd send a clear message to new users that they should adopt age.
• It also shouldn't be that crazy hard for people to do a 1 off manual migration from gpg to age if they really want to use the latest version of sops in the future.

gpg and age seem to solve the exact same use case, but age seems easier to maintain, test, install, and is more likely to be bug free / stay bug free since it's written in go vs written in c.

update: just realized my idea is stupid, AGE's crypto's are safe + better UX, but not recognized by NIST so it's not automatically best in 100% of cases.

[neoakris]

Tip for non-interactive copy pasteable command to generate gpg key pair that works consistently on both federa 35++ / centos 9 (gpg 2.3.x) and centos 8 (gpg 2.2.x):

gpg --list-secret-keys

gpg --batch --full-generate-key --rfc4880 --allow-weak-key-signatures <<EOF
    %no-protection
    Key-Type: RSA
    Key-Length: 4096
    Subkey-Type: RSA
    Subkey-Length: 4096
    Expire-Date: 0
    Name-Real: my-gpg-2.0.x_and_2.2.x-compatible-test-key-pair
    Name-Comment: Test
EOF

gpg --list-secret-keys

Note: When using gpg 2.3.x in non-interactive mode --rfc4880 alone is insufficient. You'll see a failure message and gpg --list-secret-keys will show the key didn't get generated. So gpg 2.3.x needs both flags flags --rfc4880 --allow-weak-key-signatures added

I also verified that the exact same non-interactive generation command can be copy pasted into Centos 8, so you can have a consistent experience.

update: the above is a poor workaround
gpg --list-secret-keys
(showed me the fingerprint)
export fp=E75A046BFE318854C4BF7CB54B84FC7893E24710
gpg --armor --export $fp | gpg --list-packets --verbose
(mentions algo 1(RSA & RSA) and digest algo 2(SHA-1)), I'll try to find a better copy paste-able workaround

[neoakris]

Improved version tested against CentOS 9 (gpg --version 2.3.4) & CentOS 8 (gpg --version = 2.2.20) vagrant boxes

# Notice: 
# By default, gpg 2.3.x generated keys, will generate errors when 
# consumed by clients running gpg 2.0.x - 2.2.x 
# 
# Workaround: 
# When the following gpg key gen command can be copy pasted and ran against
# RHEL 8 (which uses gpg 2.2.x) or Mac / RHEL 9 (which uses gpg 2.3.x)
# The resulting generated key will work correctly for gpg 2.0.x - 2.3.x clients
# (Technically the --rfc4880 & --digest-algo flags are only needed for gpg 2.3.x;
# however, they don't hurt when added to gpg 2.2.x, so it's recommended to 
# always use them for the sake of consistency) 
# gpg 2.0.x users (RHEL 7 & Amazon Linux) substitute --full-generate-key for --gen-key
# 
gpg --batch --full-generate-key --rfc4880 --digest-algo sha512 --cert-digest-algo sha512 <<EOF
    %no-protection
    # %no-protection: means the private key won't be password protected 
    # (no password is a fluxcd requirement, it might also be true for argo & sops)
    Key-Type: RSA
    Key-Length: 4096
    Subkey-Type: RSA
    Subkey-Length: 4096
    Expire-Date: 0
    Name-Real: Example
    Name-Comment: Example
EOF

[ajdergute]

#%no-protection: means the private key won't be password protected # (no password is a fluxcd requirement, it might also be true for argo & sops)

This doesn't hold true for sops and I recommend to remove this from your template.

[ganto]

This story goes into a new round:

If one has a GnuPG 2.4 version then even the --rfc4880 flag doesn't disable AEAD support in a new GPG key. I'm not fully sure if intended or not. As a positive side effect the removal of rfc4880bis support as done in e.g. Fedora keeps this flag from "working correctly" also with a recent GPG version.

Once AEAD is supported by the key it's prefered over the old algorithm:

AEAD is a modern and faster way to do authenticated encryption than the old MDC method. [...] The MDC is always used unless the keys indicate that an AEAD algorithm can be used in which case AEAD is used. But note: If the creation of a legacy non-MDC message is exceptionally required, the option --rfc2440 allows for this.

This eventually results in old GPG releases (especially 2.2.x found in e.g. RHEL 8) from not being able to decrypt a file anymore as discussed in this thread.

If with GPG 2.4 a key should be created that is compatible with the old GPG releases one must explicitly remove the AEAD feature as described in the Arch Linux wiki page linked above or discussed in keybase/keybase-issues#4025

[tyrannosaurus-becks]

Thanks for this ticket!

Just to state the workaround I used in shorter terms - I found that:

• If I decrypted the file somewhere it works (like on a Mac with any version of sops)
• Then re-encrypted it inside a Linux container with sops 3.9.0

The problem disappeared. I tested that the file was usable across both Mac and Linux, with sops 3.7, 3.8, and 3.9, and it was.