[xenial] Journalist replies are not displayed in source interface

[emkll]

Steps to Reproduce

• Spin up a Xenial staging environment (make build-debs-xenial and molecule converge -s libvirt-staging-xenial)
• Submit a document or message from the source interface
• Reply to said document or message
• Observe empty reply on the source interface

Expected Behavior

Source should be able to view journalist reply

Actual Behavior

Journalist reply is blank in source interface

Comments

Several changes were made in gpg 2.1 which is being used by Xenial [0], specifically the removal of the secret keyring

[0] : https://www.gnupg.org/faq/whats-new-in-2.1.html

[redshiftzero]

I took a look at this today, here's where I'm at so far: I can reproduce this bug in a Xenial-based SecureDrop dev container.

The secret keys for decrypting the replies are available, e.g.:

>>> self.gpg.list_keys(secret=True)
gnupg._parsers.ListKeys([{'dummy': u'', 'keyid': u'53ABF186E68948B9', 'expires': u'', 'rev': {}, 'sigs': {u'Autogenerated Key <JO2JDJSLGIPNTKEDZUECNFMUG2QYXVAS6VAMNXIMIAAIORSFJSG  : []}, 'subkeys': [], 'length': u'4096', 'ownertrust': u'u', 'algo': u'1', 'fingerprint': u'DE410BBA310AA8745E3BE7E653ABF186E68948B9', 'date': u'1368489600', 'trust': u'u', 'type': u'sec', 'uids': [u'Autogenerated Key <JO2JDJSLGIPNTKEDZUECNFMUG2QYXVAS6VAMNXIMIAAIORSFJSG  ]  }, {'dummy': u'', 'keyid': u'5761A79A11779651', 'expires': u'', 'rev': {}, 'sigs': {u'Autogenerated Key <3OQPA6YBNFBM3BEQH7KHYWPNLDBHKHP76HP2SCW6324K45QDHVD  : []}, 'subkeys': [], 'length': u'4096', 'ownertrust': u'u', 'algo': u'1', 'fingerprint': u'D90BD4822154FF63A54B95A55761A79A11779651', 'date': u'1368489600', 'trust': u'u', 'type': u'sec', 'uids': [u'Autogenerated Key <3OQPA6YBNFBM3BEQH7KHYWPNLDBHKHP76HP2SCW6324K45QDHVD  ]  }, {'dummy': u'', 'keyid': u'B4863A093C809921', 'expires': u'', 'rev': {}, 'sigs': {u'Autogenerated Key <ZOO3FPHBLKQQREIQ7ECPMGTPVAFXUSLEDUY2ZENQSWRWABALMPS  : []}, 'subkeys': [], 'length': u'4096', 'ownertrust': u'u', 'algo': u'1', 'fingerprint': u'2E945059A3A56EAE4F15DD7DB4863A093C809921', 'date': u'1368489600', 'trust': u'u', 'type': u'sec', 'uids': [u'Autogenerated Key <ZOO3FPHBLKQQREIQ7ECPMGTPVAFXUSLEDUY2ZENQSWRWABALMPS  ]  }])

But indeed, when I try decrypting a test file, the decrypted content is empty (one sees self.gpg below because I put a breakpoint in crypto_util.py):

>>> input_data = self.gpg.gen_key_input(name_email='jen+test@freedom.press', passphrase='test')
>>> key = self.gpg.gen_key(input_data)
>>> key.fingerprint
u'E27203CAB6B15F1B4A19260276A8A236B3C334CA'
>>> encrypted = self.gpg.encrypt('test data', key.fingerprint)
>>> encrypted.data
'-----BEGIN PGP MESSAGE-----\n\nhQEMA8k3DSXIftBUAQf/YZg9yiIRoWv8NEtwLFR2j  
>>> encrypted.ok
True
>>> decrypted = self.gpg.decrypt(str(encrypted), passphrase='test')
>>> decrypted.data
''
>>> decrypted.ok
False
>>> decrypted.stderr
u'[GNUPG:] ENC_TO C9370D25C87ED054 1 0\n[GNUPG:] PINENTRY_LAUNCHED 984\ngpg: encrypted with 2048-bit RSA key, ID C87ED054, created 2019-01-11\n      "Autogenerated Key <jen+test@freedom.press>"\ngpg: public key decryption failed: Inappropriate ioctl for device\n[GNUPG:] ERROR pkdecrypt_failed 83918950\n[GNUPG:] BEGIN_DECRYPTION\n[GNUPG:] DECRYPTION_FAILED\ngpg: decryption failed: No secret key\n[GNUPG:] END_DECRYPTION\n'  
>>> 

This looks to be the PINENTRY_LAUNCHED issue described in isislovecruft/python-gnupg#98, isislovecruft/python-gnupg#122, and isislovecruft/python-gnupg#137. The version of python-gnupg we are using includes the patches that correspond to those issues (released in 2.2.0, we're on 2.3.1). In those issues modifying gpg-agent.conf to allow the loopback option resolved the issue. I haven't tried yet, but will try this next. If anyone beats me to it, please comment on this issue.

[redshiftzero]

Running the full application test suite on Xenial indicates that there are a number of other test failures which need to be investigated and resolved (will scope as part of this issue and file followups where appropriate).

[kushaldas]

Adding the loopback pinentry configuration did not help in my system. It still failed to decrypt.

[redshiftzero]

Upstream issue I just filed with a minimal reproducer: isislovecruft/python-gnupg#249

I tested this issue in the SD dev container via the following:

BASE_OS=xenial DOCKER_RUN_ARGUMENTS='-p127.0.0.1:8080:8080 -p127.0.0.1:8081:8081 -p127.0.0.1:5901:5901' ./bin/dev-shell

and running the prerequisites. Then, following the recommendation in the earlier mentioned python-gnupg tickets, and here, I added the following options:

echo allow-loopback-pinentry > /var/lib/securedrop/keys/gpg-agent.conf
echo pinentry-mode loopback > /var/lib/securedrop/keys/gpg.conf

I ensured permissions were correct (chmod 700 /var/lib/securedrop/keys and chmod 600 /var/lib/securedrop/keys/*) and then killed any existing gpg-agent processes.

Next, I start the dev server and navigate to /lookup. Decryption of replies continue to fail, with the following stderr (acquiring by setting a breakpoint after decryption in crypto_util.py):

u'[GNUPG:] ENC_TO BD07E5639109FB27 1 0\n[GNUPG:] ENC_TO C3E7C4C0A2201B2A 1 0\ngpg: encrypted with 4096-bit RSA key, ID A2201B2A, created 2013-10-12\n      "SecureDrop Test/Development (DO NOT USE IN PRODUCTION)"\n[GNUPG:] NO_SECKEY C3E7C4C0A2201B2A\ngpg: encrypted with 4096-bit RSA key, ID 9109FB27, created 2013-05-14\n      "Autogenerated Key <VRRWLOTKWYIDQQ5MUSVDD7FZAFJTT356B2X5SNJIA2C6I3BBT7DY32YKWGKOPWCBLQRY7YTXN6T7IRJA2YFQC7LZIYALP33NI3HRIAI=>"\n[GNUPG:] NO_SECKEY BD07E5639109FB27\n[GNUPG:] BEGIN_DECRYPTION\n[GNUPG:] DECRYPTION_FAILED\ngpg: decryption failed: No secret key\n[GNUPG:] END_DECRYPTION\n'

[kushaldas]

I saw similar errors in my test last week.

[redshiftzero]

So I have this particular issue resolved in trusty and xenial staging now, see this branch. This is in a Xenial staging VM:

However - the remaining issue is that resolving this bug via the pinentry loopback option apparently introduces an issue with deleting keypairs. One reference claims that a passphrase is required. The man page claims that using --batch with --yes while passing the fingerprint should work, but in testing today it didn't work for me. I'm going to research this a bit more and figure out what to do (the good-ish news is that being unable to delete keypairs is not a huge showstopper like being unable to decrypt journalist replies would be, but would still be an unfortunate regression).