Veracrypt support

[rocodes]

Description

Per discussion, opening new monorepo-friendly PR for Veracrypt work.

Refs freedomofpress/securedrop-export#126

Fixes #1730
Fixes #1462
Fixes freedomofpress/securedrop-workstation#265
Fixes #1605
Fixes #1738
Fixes #1021
Fixes #1553
Fixes #1019
Fixes #990

Towards #1269 (fixed for ExportWizard, print not yet implemented)
Towards #991 (fixed for ExportWizard, print not yet implemented)

Supersedes #1669

Checklist:

• Address remaining functional test failures
• GUI portion WIP
• tcrypt.conf file (see note below)

Test Plan

• CI passes (note: remaining functional test failures need to be addressed)
• Visual review
• Manual test plan below

Manual testing test plan

Setup

• Prereq: Qubes SDW machine
• Provision three USBs: a VeraCrypt-encrypted USB without a hidden volume (instructions), a LUKS-encrypted USB, and an invalid device of your choosing. (Invalid devices have != 1 encrypted partitions at the main device or first partition level).
• Using build-debs.sh, build securedrop-client packages from the tip of this branch. You will need to check out this branch of the builder repo to pull in the pexpect wheel if it hasn't been merged yet.) Install the securedrop-export deb in sd-large-bullseye-template, or clone sd-large-bullseye-template and set the cloned template as template for sd-devices-dvm. if you don't want to mess with your sdw install. You will have to adjust rpc policies to copy the .deb into the template; prepend changes to /etc/qubes/policy.d/60-securedrop-workstation.policy. Shut down the template and sd-devices.
• Either install the securedrop-client deb in sd-app, and run LOGLEVEL=DEBUG securedrop-client, or give sd-app a netvm, install git, clone the client repo into sd-app, check out this branch, activate the venv, and then start the client with LOGLEVEL=DEBUG ./run.sh (observe log in ~/.securedrop_client/logs/client.log or in the logging tmpdir indicated by console output).

The business

USB detection

• Clicking export with no USB inserted: preflight window shown, click "next", prompted to insert a usb.
• >1 usbs attached to sd-devices yields "too many usbs" error
• Invalid usb attached to sd-devices yields "invalid device" error
• Can't proceed past "Insert a USB" page until exactly 1 valid device detected
• Specific hint appears to differentiate error states

LUKS USB unlocking

• 1 locked LUKS drive attached to sd-devices before export is initiated leads from preflight to passphrase prompt screen
• 1 locked LUKS drive attached to sd-devices during export wizard ("please insert a usb drive") leads to passphrase prompt screen
• Wrong unlock password does not allow you to pass passphrase screen, and an error message ("that passphrase didn't work") is shown

Successful export

• Unlocked (or unlocked and mounted) Veracrypt drive yields successful export, "Export successful" page is shown ((When testing VeraCrypt devices, unlock them, eg using nautilus in sd-devices or using udisksctl unlock -b /path/to/device at commandline in sd-devices. Locked VC drives are not currently supported)) ( [securedrop-export] [spike] Support Veracrypt #1730)
• Locked, unlocked, or unlocked and mounted LUKS drive yields successful export, as above
• Drives that are already unlocked before starting the export go from the Preflight message to the Export Successful screen, skipping the intermediate prompts ( [securedrop-export] if device already unlocked, don't ask for passphrase #1734)
• On successful exports, both luks and veracrypt drives are locked and unmounted at the end of export

Error behaviour

• On 'unrecoverable' errors (mount error, drive too full, QProcess error, etc) the "Next" button is disabled, "Finish" is present, and an error screen is shown
• Failed exports show human-readable message (Support granular error messaging when exporting conversations #1605)
• 'Recoverable' errors (bad passphrase, too many/too few devices, invalid device) show "hint" (additional text description in pink, aka error_details)

When good people do bad things

• After initiating export and reaching passphrase prompt screen, removing USB drive from sd-devices, then clicking "next" returns you to "insert a device" screen
• After initiating export and reaching the passphrase prompt screen, inserting another USB drive then clicking "next" returns you to "insert a device" screen with hint about too many USBs
• Error "hints" are not shown after resolving the issue, pressing Next, then pressing the back button

Checklist

If these changes modify code paths involving cryptography, the opening of files in VMs or network (via the RPC service) traffic, Qubes testing in the staging environment is required. For fine tuning of the graphical user interface, testing in any environment in Qubes is required. Please check as applicable:

• I have tested these changes in the appropriate Qubes environment
• I do not have an appropriate Qubes OS workstation set up (the reviewer will need to test these changes)
• These changes should not need testing in Qubes

If these changes add or remove files other than client code, the AppArmor profile may need to be updated. Please check as applicable:

• I have updated the AppArmor profile
• No update to the AppArmor profile is required for these changes
• I don't know and would appreciate guidance

If these changes modify the database schema, you should include a database migration. Please check as applicable:

• I have written a migration and upgraded a test database based on main and confirmed that the migration is self-contained and applies cleanly
• I have written a migration but have not upgraded a test database based on main and would like the reviewer to do so
• I need help writing a database migration
• No database schema changes are needed

[cfm]

@rocodes, re: the new sd-devices:/etc/udisks2/tcrypt.conf: I got a bee in my bonnet about how this works in the monorepository, so I added cfm/securedrop-client@52db899. Feel free to cherry-pick and squash it in here.

I don't see any evidence that securedrop-export installs an AppArmor profile, only securedrop-client....

[rocodes]

you're right, sorry- the PR template tripped me up into thinking maybe that had changed. Amended the description, ty

[cfm]

As of 5e681ae, the export-side test plan from #1669 checks out!

1. With "device": "disk-test": No USBs connected: NO_DEVICE_DETECTED

2. With "device": "disk-test": > 1 USBs connected: MULTI_DEVICE_DETECTED

3. With "device": "disk-test": Unsupported device connected (could be: device has 2 or more partitions, device has unrecognized encryption scheme, device is a locked VeraCrypt drive, which we won't support for now due to difficulty with error handling, or malformed device): INVALID_DEVICE_DETECTED

4. With "device": "disk-test": 1 LUKS-formatted USB connected and it's locked: DEVICE_LOCKED

5. With "device": "disk-test": 1 LUKS-formatted device (partition or whole device encrypted) connected and it's unlocked: DEVICE_WRITABLE (and device will be mounted in /media/user), or SUCCESS_EXPORT (depending on the command in metadata.json).

6. With "device": "disk-test": 1 Veracrypt-formatted device (partition or whole device encrypted) and it's unlocked: same as above

7. Other error statuses are fairly self explanatory (ERROR_MOUNT, ERROR_CLEANUP, etc)

@rocodes, I'll stay tuned for when to restart end-to-end testing from the Client.

[rocodes]

@cfm: There are a couple known issues I should flag to you that I'm trying to debug, one of which I reintroduced.

• The mimetypes handler issue: The mimetype error dialog still pops up, and export succeeds after you dismiss the dialog.
• A mounting issue (this is the one I reintroduced) - it looks like there's a slight regression where it takes 2 clicks to export after unlocking a drive, because the mounted device name isn't detected til the second try.

[cfm]

I've reviewed the marginal diff 05b8ad3...0555520 after #1777 (comment). Final review steps:

1. Resolve open review threads
2. Finish reading through the overall diff (mostly tests)
3. Repeat test plan

[cfm]

Fabulous. Here are some optional nit-level suggestions if it's easy to tack them on or rebase them in (as you prefer). Nothing substantive, however, so I'll proceed with a final round of testing this afternoon.

[cfm]

Marginal diff reviewed from 0555520...0367a08: all looks good to me, and CI is happy. So is the test plan, with one asterisk in the "Successful Export" section that I'm nonetheless comfortable approving.

USB detection

• Clicking export with no USB inserted: preflight window shown, click "next", prompted to insert a usb.
• >1 usbs attached to sd-devices yields "too many usbs" error
• Invalid usb attached to sd-devices yields "invalid device" error
• Can't proceed past "Insert a USB" page until exactly 1 valid device detected
• Specific hint appears to differentiate error states

LUKS USB unlocking

• 1 locked LUKS drive attached to sd-devices before export is initiated leads from preflight to passphrase prompt screen
• 1 locked LUKS drive attached to sd-devices during export wizard ("please insert a usb drive") leads to passphrase prompt screen
• Wrong unlock password does not allow you to pass passphrase screen, and an error message ("that passphrase didn't work") is shown

Successful export

• Unlocked (or unlocked and mounted) Veracrypt drive yields successful export, "Export successful" page is shown ((When testing VeraCrypt devices, unlock them, eg using nautilus in sd-devices or using udisksctl unlock -b /path/to/device at commandline in sd-devices. Locked VC drives are not currently supported)) ( [securedrop-export] [spike] Support Veracrypt #1730)

On the machine where I've done all previous testing, the wizard hangs on Working with this error sequence on sd-devices.

Feb 21 15:22:48 sd-devices send-to-usb[2194]: 2024-02-21 15:22:48,705 - securedrop_export.disk.cli:110(get_volume) DEBUG: Checking removable, writable device /dev/sda
Feb 21 15:22:48 sd-devices send-to-usb[2194]: 2024-02-21 15:22:48,715 - securedrop_export.disk.cli:199(_get_supported_volume) DEBUG: sda1 is unlocked but unmounted; attempting mount
Feb 21 15:22:48 sd-devices send-to-usb[2194]: 2024-02-21 15:22:48,716 - securedrop_export.disk.cli:354(_mount_volume) DEBUG: Check to make sure udisks identified /dev/sda1 (unlocked as /dev/mapper/tcrypt-2049)
Feb 21 15:22:48 sd-devices send-to-usb[2194]: 2024-02-21 15:22:48,827 - securedrop_export.disk.cli:367(_mount_volume) DEBUG: udisks found /dev/sda1
Feb 21 15:22:48 sd-devices send-to-usb[2194]: 2024-02-21 15:22:48,827 - securedrop_export.disk.cli:370(_mount_volume) INFO: Mount /dev/mapper/tcrypt-2049 using udisksctl
Feb 21 15:22:48 sd-devices dbus-daemon[736]: [session uid=1000 pid=736] Activating via systemd: service name='org.freedesktop.Tracker1' unit='tracker-store.service' requested by ':1.2' (uid=1000 pid=719 comm="/>
Feb 21 15:22:48 sd-devices systemd[697]: Starting Tracker metadata database store and lookup manager...
Feb 21 15:22:48 sd-devices dbus-daemon[736]: [session uid=1000 pid=736] Successfully activated service 'org.freedesktop.Tracker1'
Feb 21 15:22:48 sd-devices systemd[697]: Started Tracker metadata database store and lookup manager.
Feb 21 15:22:49 sd-devices send-to-usb[2194]: 2024-02-21 15:22:49,116 - securedrop_export.disk.cli:379(_mount_volume) INFO: Successfully mounted device at /media/user/7e9fd0b1-dc72-4cab-8665-cece31cb6a0b
Feb 21 15:22:49 sd-devices send-to-usb[2194]: 2024-02-21 15:22:49,119 - securedrop_export.disk.cli:391(_mount_volume) DEBUG: Close pexpect process
Feb 21 15:22:49 sd-devices send-to-usb[2194]: 2024-02-21 15:22:49,219 - securedrop_export.disk.cli:131(get_volume) DEBUG: Export target is /dev/sda1
Feb 21 15:22:49 sd-devices send-to-usb[2194]: 2024-02-21 15:22:49,220 - securedrop_export.disk.service:52(export) DEBUG: Mounted volume detected, exporting files
Feb 21 15:22:49 sd-devices qubes.OpenInVM+-sd-app[2141]: mkdir: cannot create directory ‘/media/user/7e9fd0b1-dc72-4cab-8665-cece31cb6a0b/sd-export-20240221-152248’: Permission denied
Feb 21 15:22:49 sd-devices send-to-usb[2194]: 2024-02-21 15:22:49,222 - securedrop_export.disk.cli:432(write_data_to_device) ERROR: Command '['mkdir', '/media/user/7e9fd0b1-dc72-4cab-8665-cece31cb6a0b/sd-export>
Feb 21 15:22:49 sd-devices send-to-usb[2194]: 2024-02-21 15:22:49,222 - securedrop_export.disk.cli:459(cleanup) DEBUG: Syncing filesystems
Feb 21 15:22:49 sd-devices send-to-usb[2194]: 2024-02-21 15:22:49,234 - securedrop_export.disk.cli:524(_remove_temp_directory) DEBUG: Deleting temporary directory /tmp/tmptff_mkc8
Feb 21 15:22:49 sd-devices send-to-usb[2194]: 2024-02-21 15:22:49,236 - securedrop_export.disk.cli:476(_close_volume) DEBUG: Unmounting drive /dev/mapper/tcrypt-2049 from /media/user/7e9fd0b1-dc72-4cab-8665-cec>
Feb 21 15:22:49 sd-devices systemd[697]: media-user-7e9fd0b1\x2ddc72\x2d4cab\x2d8665\x2dcece31cb6a0b.mount: Succeeded.
Feb 21 15:22:49 sd-devices send-to-usb[2194]: 2024-02-21 15:22:49,527 - securedrop_export.disk.cli:497(_close_volume) DEBUG: Closing drive /dev/sda1
Feb 21 15:22:49 sd-devices send-to-usb[2194]: 2024-02-21 15:22:49,584 - securedrop_export.disk.service:77(export) DEBUG:
Feb 21 15:22:49 sd-devices send-to-usb[2194]: 2024-02-21 15:22:49,584 - securedrop_export.disk.service:79(export) ERROR: Enountered ERROR_EXPORT while trying to export

However, on a fresher machine, where I've not been tampering with sd-devices for weeks, this works as expected. Marking as passing.

• Locked, unlocked, or unlocked and mounted LUKS drive yields successful export, as above
• Drives that are already unlocked before starting the export go from the Preflight message to the Export Successful screen, skipping the intermediate prompts ( [securedrop-export] if device already unlocked, don't ask for passphrase #1734)
• On successful exports, both luks and veracrypt drives are locked and unmounted at the end of export

Error behaviour

• On 'unrecoverable' errors (mount error, drive too full, QProcess error, etc) the "Next" button is disabled, "Finish" is present, and an error screen is shown
• Failed exports show human-readable message (Support granular error messaging when exporting conversations #1605)
• 'Recoverable' errors (bad passphrase, too many/too few devices, invalid device) show "hint" (additional text description in pink, aka error_details)

When good people do bad things

• After initiating export and reaching passphrase prompt screen, removing USB drive from sd-devices, then clicking "next" returns you to "insert a device" screen
• After initiating export and reaching the passphrase prompt screen, inserting another USB drive then clicking "next" returns you to "insert a device" screen with hint about too many USBs
• Error "hints" are not shown after resolving the issue, pressing Next, then pressing the back button

[rocodes]

Thanks @cfm. Re export hanging: I haven't seen that before on any machine. Does the behaviour persist across reboots of sd-devices?

[...] cannot create directory ‘/media/user/7e9fd0b1-dc72-4cab-8665-cece31cb6a0b/sd-export-20240221-152248’: Permission denied

This is what I would expect to see if for example the drive were unlocked by root, since the export process now does not use root, although I know we talked about this and you said it wasn't the case. Anyways, maybe we can do some followup on that next week.

Thank you for all your patience and your careful testing and review.