Switch to fedora-33 for sys-* VMs

[eloquence]

Status

Ready for review

Description of Changes

Fixes #691 (co-authored with @conorsch).

Test plan

Update scenario

1. Provision a staging environment from current main, or enforce the state of an existing, up-to-date staging environment via sdw-admin --apply.
2. Ensure that you do not have a fedora-33 template present on your system from prior runs or a manual update. If necessary, remove the template with sudo dnf remove qubes-template-fedora-33.
3. make clone this branch into dom0 and sudo dnf reinstall the freshly built RPM from rpm-build/RPMS/noarch.
4. Run the updater with /opt/securedrop/launcher/sdw-launcher.py --skip-delta 0.
5. • Observe that the update completes successfully
6. • Observe that the fedora-33 template is now installed
7. • Observe that qvm-ls | grep fedora shows that sys-net, sys-usb, and sys-firewall are all based on fedora-33.
8. • Observe that qvm-prefs default-mgmt-dvm template shows fedora-33
9. • Observe that dom0 tests are passing (ensure that config.json and sd-journalist.sec are present).

Fresh install scenario

1. Uninstall the SecureDrop Workstation via sdw-admin --uninstall
2. Ensure that you do not have a fedora-33 template present on your system from prior runs or a manual update. If necessary, remove the template with sudo dnf remove qubes-template-fedora-33.
3. make clone this branch into dom0 and run make staging
4. • Observe that the installation completes successfully
5. • Observe that the fedora-33 template is now installed
6. • Observe that qvm-ls | grep fedora shows that sys-net, sys-usb, and sys-firewall are all based on fedora-33.
7. • Observe that qvm-prefs default-mgmt-dvm template shows fedora-33
8. • Observe that dom0 tests are passing (ensure that config.json and sd-journalist.sec are present).

[conorsch]

we attempt to run an update before having changed the default system VM

You're absolutely right! I pushed a commit that resolves on my end. Can you take it for a spin? You'll have to undo all the fedora-33-related changes from your system, and sudo dnf remove qubes-template-fedora-33, in order to test that it works OK.

[eloquence]

With your fix, I was able to re-create a fedora-33 environment without errors during this part of the migration. I did encounter our friend libxenlight at the end, which may or may not have been memory-related. It didn't cause the update to fail, though, and the system seemed to be in a happy state at the end.

I think at this point it would be good for someone else to take it for a spin, for either the update or fresh install scenario, so I added a test plan and am declaring it ready for review.

[conorsch]

I'm not seeing the libxenlight error (64GB RAM on this machine, so I don't expect to see it), and I'm able to migrate F32 -> F33 separately, manually resetting in between. That much is exactly as we'd expect. However, these changes introduce a concerning regression in the salt logic:

Set up logging VMs early
sd-small-buster-template:
      An Exception occurred while executing state.highstate: Unable to render top file. No targets found.
sd-log:
      An Exception occurred while executing state.highstate: Unable to render top file. No targets found.
whonix-gw-15:
      An Exception occurred while executing state.highstate: Unable to render top file. No targets found.
Provision all SecureDrop Workstation VMs with service-specific configs
sd-small-buster-template:
      An Exception occurred while executing state.highstate: Unable to render top file. No targets found.
sd-large-buster-template:
      An Exception occurred while executing state.highstate: Unable to render top file. No targets found.
securedrop-workstation-buster:
      An Exception occurred while executing state.highstate: Unable to render top file. No targets found.

I made sure to compare with latest main, and the exception messages are not printed there. Even though the changes in this PR do what they say on the tin, and successfully migrate the Fedora-based VMs, I'd like to understand that new warning message and hopefully address it prior to merge. I volunteer to look into this a bit more, @eloquence.

[eloquence]

By all means, thanks @conorsch. It may be worth landing #690 first to aid collaborative debugging.

[eloquence]

I don't see these errors in my logs, but that may be partially because I launched the updater from the GUI. That was intentional - to capture all Salt output in journalctl, the updater has to be launched via the XFCE file manager (where the Salt output shows up as org.xfce.FileManager service events). That said, I do not see Exceptions reported in the Salt output from my test yesterday.

#690 should give us more reliable and complete logs -- the combination of running the updater from the CLI and looking at launcher-detail.log should help get consistent error output.

[conorsch]

#690 should give us more reliable and complete logs -- the combination of running the updater from the CLI and looking at launcher-detail.log should help get consistent error output.

#690 is merged. Now that I've spent more time with those changes, I'm no longer convinced the warnings shown above will make it into the detailed log, but still glad to have them in. I haven't investigated the warning changes here further, but will do so this sprint.

[conorsch]

Took a closer look at this today. Turns out it's not a problem with the provisioning logic we're changing here, but rather a known error caused by using fedora-33 as the default-mgmt-dvm. Unfortunately, not setting fedora-33 as the default-mgmt-dvm also causes an error, although a different one. So our hands are tied here, for the time being, until the upstream patch lands in the stable repos. See related discussion here: QubesOS/qubes-issues#6580

One thing we can do near-term is enable the testing repos and confirm that the patch presented there resolves the problem in conjunction with these changes. Let's keep a close on the stable repos, since we're currently blocked on the transition until that's resolved.

[conorsch]

Specifically, we want qubes-mgmt-salt-4.0.25-1.fc25 in dom0 and qubes-mgmt-salt-4.0.25-1.fc33 in fedora-33. Once those are live, the problem should be resolved. Will test with the rcs already available in the "testing" channels, and if results are good, then we can at least proceed with review and merge. We'd still be blocked a bit longer on final release until the packages in are stable.

[conorsch]

After manually pulling in the salt fixes from the Qubes testing channels, highstate functionality is restored, and subsequent updater runs work as expected, using F33 across the board. It's important to note that after installing the F33 image, it must be updated in order to get the salt fixes, before it can be used to update other machines. Fortunately the logic here is already doing just that, so we're covered. That's enough for us to approve these changes and merge in.

Judging by past experience, we can expect the packages to land in stable sometime early next week. That gives us time to work on the keyring updates such as #697 before the next release of the securedrop-workstation-dom0-config package.

There's one remaining concern with merging here: once this PR lands in main, running make dev will force F33 everywhere, without the testing repos, thereby breaking highstate functionality for the system. That's not a showstopper—notably the qubes gui updater will continue to work, since that uses a state apply, not a highstate—but mentioning it for team visibility.

[conorsch]

Approving. We're still waiting on stable packages, due next week, and until then make dev will break highstate functionality. Merging despite that shortcoming to continue with other work.

[eloquence]

Noting that qubes-mgmt-salt-4.0.25-1.fc33 has landed in fedora-33 and I was able to run sdw-admin --apply without errors after applying the update.