Converts all SDW AppVMs to Debian 9

[conorsch]

All of the sd-* VMs for the SecureDrop Workstation now use Debian 9 as
the base template. This will allow us to use deb packages across the
board for all AppVMs, as well as the hardened based image with a custom
kernel in the near future, as well.

Updates the test suites to accommodate for gpg v1/v2 output variance, as
well as removes Fedora 28 as a permitted base for the AppVMs in the
platform tests.

Testing

1. Run make all in dom0 and confirm no errors.
2. Run make test in dom0 and confirm no errors.
3. Run Qubes Manager and visually confirm all the sd- VMs are running Debian 9 (or Whonix 14, which is itself based on Debian 9).
4. Confirm you can download and view a test submission.

Regarding 4, there's a bit of manual work that may be required, in order to get a file browser for in the SVS.

1. Run sudo apt install -y nautilus inside the debian-9 template.
2. Via Qubes menu, navigate to Template: Debian-9 > Qubes Settings > Applications and click ** Refresh Applications.
3. Via Qubes menu, navigate to Domain: sd-svs > Qubes Settings > Applications and click ** Refresh Applications.

I propose not automating this step since it's a temporary workaround. We're planning to implement TemplateVMs in #131, as well as ditching file browsing in favor of the SecureDrop Client on the SVS.

[emkll]

I can confirm that all tests pass and that submissions are decrypted, and files can be opened in the sd-svs-disp based disposable VM (after installing nautilus in the debian-9 template, using the Files application).

I am, however, experiencing some regressions with the processing of default applications in debian-9 based VMs:

1. xdg-open invoked by the command line in sd-svs does not open in a disposable VM (using gnome/nautilus, however, correctly opens the file is a disposable vm). This is a regression, current master under fedora28 would open these files in disposable VMs when using xdg-open.
2. Text submissions (i.e. messages) fail to open in a new window (within the dispvm) due to what appears to be the default editor in debian-9 (nano). Using something like update-alternatives should work, but the change must be applied in the template (we could apply this in the securedrop-workstation template):
   sudo update-alternatives --install /usr/bin/editor editor /usr/bin/gedit 50

This PR obviously does not change the template for sys-net and sys-firewall, and we should perhaps open a follow-up issue to discuss how we will keep those vms up-to-date.

[conorsch]

Hadn't tested with xdg-open on the CLI before, but now that I have, I can confirm @emkll's report: it opens text files in nano. Spent a fair amount of time tracing down the chain of calls that xdg-open makes, and it seems that the root problem is xdg-mime query filetype <text file> returns nothing—whereas file -b --mime-type <text file> returns "text/plain" as expected.

There's an automatic fallback to run-mailcap --action=view <text file>, which uses /etc/mailcap to determine the handler. Editing that file is painful: it's not clear to me in what order the declarations are resolved; it appears to be the first successful handler per mimetype wins. Testing config munging logic to confirm that an edit to /etc/mailcap resolves on a fresh build.

It's not technically necessary that we resolve the xdg-open logic right now, since we don't actually explicitly call xdg-open, and we'll have much more control once the native client lands (as well as via templates in #131). Still, I do agree we should understand the mechanics here, and testing xdg-open behavior is a sane fallback (we may reasonable call out to it via the client code). Rather than punt to a separate issue immediately, I'll try to resolve here as requested, but if the config munging doesn't work, let's discuss deferring.

[conorsch]

Testing config munging logic to confirm that an edit to /etc/mailcap resolves on a fresh build.

Ah, of course: we need #131 to make that config change persistent, since sd-svs will erase it on shutdown. It does work if you make all && make sd-svs, which is a bit goofy. Will push up that change for you to review, @emkll. Still worth including here so that we can port it to the template setups in #131.

[conorsch]

Take another look, @emkll. The mailcap hack only applies to text/plain mimetypes. We may want to force all types via mailcap in order to coerce xdg-open via CLI; however, in my testing, running in a full desktop environment seems to be sufficient for getting the mimetypes to be handled by open-in-dvm. May still be worth another issue to investigate more completely; certainly we'll want tests of some kind for the handling logic.

[conorsch]

Marking as WIP until we shake out the remaining issues on the mimetype handling regression. We'll almost certainly need to open a separate issue to track mimetype coverage in Debian 9, but the near-term issue blocking merge is that files opened in Nautilus in sd-svs are not actually displayed—presumably due to mimetype mishandling, as @emkll noted above.

Previously I'd incorrectly assigned the mailcap mimetype override to sd-svs, when it should have been applied to sd-svs-disp. We cannot apply directly to sd-svs-disp, however, because the config will never persist—cue #131, which we badly need, and which is up next, after we sort out this temporary base template transition.

More recently I've applied the mailcap mimetype override to debian-9, which is (as of this PR) the underlying template for sd-svs-disp. I can confirm that the override persists on sd-svs-disp, but it's insufficient to view a submission. Suggested test plan for further investigation is the same as that in the PR submission text, with crucial bit being, of course, "Confirm you can download and view a test submission". When I try that, everything works except the final display; that is:

1. Downloading a submission via sd-journalist properly triggers the custom process handler.
2. GUI feedback is displayed about in-progress submission processing.
3. Files are available on sd-svs within ~/Sources/ subdirectories.
4. Opening a submission via Nautilus in sd-svs triggers creation of an sd-svs-disp-based DispVM.
5. 🔴 Broken, does not occur: File is displayed in sd-svs-disp-based DispVM.

It's step 5 that we need to resolve for merge here. I agree with @emkll that it's still mimetypes biting us. So far I've observed that mimetypes are handled "correctly" (i.e. as they've been in fedora-28) in debian-9 only when there's an active desktop environment. Invocations from terminal—which may include the open-in-dvm call—fallback to mailcap, ignoring the desktop file handlers @joshuathayer has so meticulously prepared.

@joshuathayer can you take a look at this and try to resolve step 5 above?

[conorsch]

Ready for re-review. The mailcap override recently inserted wasn't taking precedence because it wasn't in the correct section of the file. From /etc/mailcap:

###############################################################################
#
#  User section follows:  Any entries included in this section will take
#  precedence over those created by "update-mime".  DO NOT CHANGE the
#  "User Section Begins" and "User Section Ends" lines, or anything outside
#  of this section!
#

# ----- User Section Begins ----- #
# -----  User Section Ends  ----- #

###############################################################################

Indeed, the overrides must land between those headings. Updated the insertion logic to be more precise (rather than prepending to the top of the file, which does not trigger override behavior) and can confirm: "File is displayed in sd-svs-disp-based DispVM".

We'll likely need to port the entirety of the mimetypes already declared in desktop files to the mailcap setup. That work should happen as part of the overall conversion to custom templates (#131). Will open a separate issue to track.

[conorsch]

Remarking as WIP. ☹️ During discussion with @emkll, we discovered that we already have divergence of the debian-9 TemplateVM on our respective workstations—which vastly complicates review. We should revisit hosting the securedrop-workstation template via an RPM installed in dom0.

The issue at present is that the Salt config always reportes debian-9: ERROR, which means that the mailcap changes I made aren't ever applied to the debian-9 template. At first glance, I suspected this was the same problem we resolved in #143, but applying that same fix was insufficient to resolve here. So forget about the changes presented here for a moment, and let's figure out why it's not possible to manage the debian-9 template via Salt. Answering that will unblock this PR for final review and merge.

[msheiny]

@kushaldas how do you feel here? This seems very similar to complaints you've made against using salt as a software distribution mechanism which I strongly agree with. Falls in line with a lot of the problems that have bit us with using ansible on the securedrop server vs pushing around large binaries (in this case, pre-built templates)

[joshuathayer]

I traced through xdg-mime, which, it turns out, is just a big shell script. Long story short, the xdg world on Stretch under Qubes (at least on my machine) thinks it's running Gnome (because $GNOME_SESSION_ID is non-null) thus it tries to use gnomevfs-info to actually detect file mimetypes. Sadly that binary is not installed on our barebones Stretch VMs. After installing it by hand with

sudo apt install libgnomevfs2-bin

xdg-mime query filetype foo.txt works as expected. I suspect that will go a long ways towards addressing other issues in this above discussion, but I have not investigated further yet.

I suspect there's a better way to fix this problem. I don't know much about desktop environments and stuff, but I don't think we're running under Gnome at all, right? I suspect we'd be happier if xdg-mime fell back to a very generic tool like mimetype or file to do its filetype identification (which it does if it fails to identify the running desktop environment).

[joshuathayer]

Also I feel like we should break this PR out into a couple of issues, then revisit when those are sorted

* Getting `xdg-mime` to play nicely in Stretch
* Handling template divergence / salt vs RPM template management

Does that breakdown make sense to people?

[eloquence]

Is it even worth aiming for landing this PR, or should we move straight on to #131, taking the lessons learned here into account (including the potential need for RPM-packaging the base template)?

As for the xdg-open issue, I would suggest that we need to be very precise what we're solving for right now. If it's just making sure that we don't break master by landing the switch to Debian, then a more temporary quick fix seems acceptable, since it would be replaced once the client lands. If it's an approach the client will also use to match MIME types and applications, then we should make sure it's the correct one. Personally, it seems to me that a quick fix is acceptable for now, to get us into Debian land as soon as possible.

[joshuathayer]

@eloquence, I think we'd be best off finding a real fix for the xdg-* issue, since even when we have a real client app we'll still want to attempt to keep users opening files directly in the client VM. I just pushed onto this branch (hope that's OK, @conorsch ) what I think is a good initial step in that direction- it should fix the xdg-open issue on sd-journalist, and also fixes a missing Python library that was a victim of the python 2 to 3 migration.

@conorsch and @emkll feel free to try the xdg stuff with this latest commit applied (note that it updates the whonix-ws-14 template, so make sure to shut down that template before running sd-journalist!)

[joshuathayer]

Oops, upon rereading the comments on this PR, I see the problems people were experiencing were in sd-svs and sd-svs-disp, while my commit above addresses a related problem in sd-journalist (where we currently don't want anybody opening anything). The fix for sd-svs*, I think for both the "xdg-open doesn't work on sd-svs" problem and (at least partially) the "text files are not opened correctly in sd-svs-disp" problem should be the same: ensure that gvfs-bin and libgnomevfs2-bin are installed on sd-svs and sd-svs-disp.

I just manually installed those on my sd-svs VM and text/plain files are now identified correctly by xdg-mime and attempted to be opened on a new disposable VM when opened with xdg-open.

Looking deeper into the disposable VM's handling of text/plain files: debian-9 is configured to use vim explicitly (not /usr/bin/editor) to open text files:

user@debian-9:~$ xdg-mime query default text/plain
vim.desktop

It's pretty easy to update that (we just add an entry to ~/.config/mimeapps.list, associating text/plain with whatever app we want to use), but presently that would mean altering the disp-vm's template, which is debian-9.

Since fixes to all these problems involve changes to templates, I agree with @eloquence and others that we should put this on ice and focus instead on #131 now. I think we've identified a number of changes which we'll want to integrate with our new templates, at least:

• install gvfs-bin and libgnomevfs2-bin
• for the viewing disp-vm template, update the the mime database to associate text/plain to gedit
• install nautilus in the client VM?
• I don't totally understand the mailcap discussion, but I think its need is obviated by fixing xdg-open?

[conorsch]

Great info here, @joshuathayer, thanks for the detail! I'll try to reproduce the success you've had with the new package inclusions and report back.

It's pretty easy to update that (we just add an entry to ~/.config/mimeapps.list, associating text/plain with whatever app we want to use), but presently that would mean altering the disp-vm's template, which is debian-9.

Recall that editing files in $HOME won't work in TemplateVMs, since every AppVM gets its own homedir by default. So in the partial progress for #131, I've switched use of ~/.config/mimeapps.list to /usr/share/applications/mimeapps.list throughout. Ensuring that file is never clobbered by upstream changes should be handled by the separate xdg/mimetype issue you propose.

I don't totally understand the mailcap discussion, but I think its need is obviated by fixing xdg-open?

Agreed, but warrants confirmation via testing. I'll open the new issues you described, along with two others—use a dedicated template, e.g. securedrop-workstation; and ship that template via PRM—to guide next steps. Sounds like the best path forward here is to close this PR, and fold its lessons into the WIP toward #131. Let's discuss at the sprint planning meeting today.

[emkll]

Thanks @joshuathayer, installing gvfs-bin and libgnomevfs2-bin is the key to fixing these issues.

Installing these packages does indeed fix the xdg-open issue for sd-svs, despite the xdg-mime query command returning the same thing before and after installing those packages:

user@sd-svs:~$ xdg-mime query default text/plain
open-in-dvm.desktop;

xdg-open ~/Sources/TheSource/file will now open in a sd-svs-disp-based disposable vm. However, the default file in sd-svs-disp for text/plain is vim.

Running the following command in sd-svs-disp will associate text/plain to gedit (and the change appears to persist accross dispvms spawned, if gvfs-bin and libgnomevsf2-bin are installed in the template, per the above, since sd-svs-disp is not itself a template):

xdg-mime default org.gnome.gedit.desktop text/plain

With that, plaintext submissions are now properly opened in gedit on the disposable vm, either invoked by nautilus or by xdg-open on the command line of sd-svs.

[conorsch]

For the record, this command:

xdg-mime default org.gnome.gedit.desktop text/plain

adds customizations to $HOME, which would not persist in TemplateVMs. We can instead declare the resultant rules in a system location at /usr/share/applications/mimeapps.list, the same way we already do for sd-svs (to trigger the open-in-dvm logic). Will follow-up with such an implementation shortly, via a separate PR.

[conorsch]

We learned a lot of lessons in this attempt. Superseded by changes in #160, so closing.