Avoid re-downloading dom0 update metadata after every restart of template-based UpdateVM

[mattmccutchen]

Qubes OS version: R4.0

Affected component(s) or functionality: qubes-dom0-update

Brief summary: If the UpdateVM is a template-based VM, then the dom0 update metadata cache in /var/lib/qubes/dom0-updates/var/cache is lost every time the UpdateVM restarts. This makes qubes-dom0-update periodically re-download tens of megabytes of Fedora 25 repository metadata that are no longer changing, which is slow and unnecessarily burdens the Fedora servers.

How Reproducible: Always (under the conditions previously described)

To Reproduce

Steps to reproduce the behavior:

1. Choose an UpdateVM that is a template-based VM.
2. Run sudo qubes-dom0-update in a dom0 terminal.
3. Restart the UpdateVM.
4. Run sudo qubes-dom0-update in a dom0 terminal again.

Expected behavior: Unchanged repository metadata such as that of the Fedora 25 repositories is not re-downloaded.

Actual behavior: The metadata is re-downloaded, as one can see from the progress output.

Screenshots: I didn't think to take one before manually fixing the problem on my system, and I suspect reverting the fix to take one would have little value in this case.

Additional context: None.

Solutions you've tried: Manually create a directory under /rw in the UpdateVM, and symlink /var/lib/qubes/dom0-updates to it in the TemplateVM. (FTR, I found two problems with symlinking only /var/lib/qubes/dom0-updates/var/cache: the find command doesn't traverse the symlink (though that could easily be changed) and the ln command fails because it is cross-filesystem. It seems harmless to me to symlink the entire /var/lib/qubes/dom0-updates.)

Relevant documentation you've consulted: https://www.qubes-os.org/doc/software-update-dom0/

Related, non-duplicate issues: None found.

I'd be interested in trying my hand at a fix if the maintainers give some tips about the approach they would accept, but this may be low on my long task list.

[DemiMarie]

Fixing this is quite simple: use a file under /etc/qubes-bind-dirs.d to ensure that /var/lib/qubes/dom0-updates is bind-mounted to a directory under /rw. A symlink would also work, but getting RPM to handle it correctly seems somewhat tricky.

[andrewdavidwong]

This makes qubes-dom0-update periodically re-download tens of megabytes of Fedora 25 repository metadata that are no longer changing, which is slow and unnecessarily burdens the Fedora servers.

I would think that the obvious solution is to disable the EOL Fedora 25 repos, which can only serve as an attack vector at this point (related: #6186).

By contrast, a persistent cache would have the side effect of consuming disk space, which we've found many users to be extremely sensitive about.

I think the difference between the issue title and issue description represents an XY problem as currently stated and risks being a "won't do" due to the aforementioned side effect, so I'll update the title to match the actual problem stated in the description.

[mattmccutchen]

I would think that the obvious solution is to disable the EOL Fedora 25 repos, which can only serve as an attack vector at this point (related: #6186).

Are we assuming that by the time each future Qubes OS version is released, the dom0 Fedora version will be EOL, or that users will never install any dom0 Fedora packages beyond the ones that come with the installer? I for one am one of those lower-class Qubes OS users who installs things like cdparanoia in dom0 because there isn't another practical way on Qubes OS and I can't be bothered to dual boot just for this, so I'll be reverting the change locally.

Also, I cited the unchanging Fedora repos as the most egregious example, but the qubes-dom0-current metadata is also 21 MB and I restart my UpdateVM much more frequently than the metadata_expire setting of the repo (7 days). (I wonder whether 7 days is too long for security updates, but that's another story.)

By contrast, a persistent cache would have the side effect of consuming disk space, which we've found many users to be extremely sensitive about.

Is this any worse than the persistent cache that each system volume will have of the in-VM package metadata under any mainstream setup?

I think the difference between the issue title and issue description represents an XY problem as currently stated and risks being a "won't do" due to the aforementioned side effect, so I'll update the title to match the actual problem stated in the description.

I updated it again to reflect the more general problem I intended to report.

[andrewdavidwong]

Are we assuming that by the time each future Qubes OS version is released, the dom0 Fedora version will be EOL

Not necessarily. Historically, this has not always been the case.

or that users will never install any dom0 Fedora packages beyond the ones that come with the installer? I for one am one of those lower-class Qubes OS users who installs things like cdparanoia in dom0 because there isn't another practical way on Qubes OS and I can't be bothered to dual boot just for this, so I'll be reverting the change locally.

The need to install packages in dom0 is a fair point, but it is done very rarely compared to checking for Qubes dom0 updates, so it still seems wasteful and time-consuming to download Fedora repo metadata every time after that upstream version has reached EOL. It seems like it would be better just to enable the repo whenever you need to use it to install a package, but I suppose most users won't know how to do that, even if we document it.

Is this any worse than the persistent cache that each system volume will have of the in-VM package metadata under any mainstream setup?

I don't know, but I doubt that would matter to the users who are very sensitive about disk space. On the other hand, #1957 seems somewhat popular, so perhaps there's hope.

[mattmccutchen]

@andrewdavidwong OK. I think I've presented all the relevant facts, so I'll trust your judgment on the best default for Qubes OS and do whatever I need to do on my own system. If there's a simple contribution you'd like me to try to make, let me know.

[andrewdavidwong]

@andrewdavidwong OK. I think I've presented all the relevant facts, so I'll trust your judgment on the best default for Qubes OS and do whatever I need to do on my own system.

I was just sharing my opinion. It'll be up to the devs, especially @marmarek, to decide.

If there's a simple contribution you'd like me to try to make, let me know.

Thank you! I'll leave it to the devs to let you know if there's anything you can do.

[iacore]

This is still slow. Fedora 32 metadata is 90MB in size; dnf is impossible use under poor Internet connection. I can't even install security updates.

Workaround: Use standalone VM as updatevm

[DemiMarie]

@marmarek can we only re-download the Fedora 32 metadata when we need it?

Edit: What if we shipped it as a file:/// URL?

[logoerthiner1]

I have just tried cd /etc/yum.repos.d && sudo mkdir old && sudo mv fedora* old to disable fedora 32 update in dom0. Are there any security consequences or other types of consequences by doing this, and is this suggested or not suggested?

[iacore]

@logoerthiner1 You probably can't install xen then.

[andrewdavidwong]

I have just tried cd /etc/yum.repos.d && sudo mkdir old && sudo mv fedora* old to disable fedora 32 update in dom0. Are there any security consequences or other types of consequences by doing this, and is this suggested or not suggested?

Please note that the issue tracker (qubes-issues) is not intended to be a place for fielding questions. Instead, we have other venues meant for asking questions, asking for help, and having discussions. (By contrast, the issue tracker is more of a technical tool intended to support our developers in their work.) Thank you for your understanding!

[andrewdavidwong]

Is this still a problem in 4.1?

[mattmccutchen]

Is this still a problem in 4.1?

Yes: on my main system (which is now running Qubes 4.1), I temporarily reverted my workaround and performed the original steps to reproduce, and all of the dom0 repository metadata (both Fedora 32 and Qubes 4.1) was re-downloaded.

[andrewdavidwong]

After skimming this issue and comments again to refresh my memory, this sounds more like an enhancement request than a bug report. It doesn't sound like there's actually a bug in the current update process. It just hasn't been optimized and has a lot of room for improvement. If I've misunderstood, let me know.

[mattmccutchen]

I think the reasons I originally filed this as a bug were that I guessed that the current behavior was unintentional and that the fix would be obvious and uncontroversial and I felt the "steps to reproduce, expected behavior, actual behavior" format of the bug template was better suited to the information I had to convey than the open-ended enhancement request template. It is true that the only impact is poor performance. I imagine there are at least some cases in which we consider poor performance to constitute a bug, but I'm fine if you prefer to regard this one as an enhancement request given the apparent controversy around the proposed fix due to some users' disk space concerns.