Unused kernels can fill up template disk - distro equivalent of autoremove should be applied automatically.

[eloquence]

SecureDrop Workstation templates can accumulate a large amount of used diskspace. We should consider if we can run certain maintenance tasks automatically as part of update runs. For example, one one system, just running sudo apt autoremove freed up ~3.5 GB of system space (in 3 different templates, each). With a default size of 10GB, that can be the difference between a working template or a broken one.

[eloquence]

@marmarek Disk space inflation inside templates is biting both developers and end users on a regular basis; as noted above, autoremove for Debian templates can often free up several GBs of space. While we can of course add our own autoremove runs to the SecureDrop Workstation updater, it seems better resolved at the OS level. Are there flags we should set during updates using update.qubes-vm with qubesctl, or improvements coming in Qubes 4.1? If not, perhaps this is something that we could usefully contribute to upstream?

[marmarek]

This is a very good point indeed. I think it should be easy enough to add autoremove call to update.qubes-vm salt file. I don't see proper abstraction for it in the pkg state, so it will likely need cmd.run state.

[conorsch]

Observed this in the wild today, recommended as a workaround: qvm-volume resize sd-large-buster-template:root 15G.

[conorsch]

And another one! Same resolution. We should definitely set 15G by default as part of the dom0 state—similar to what we already do for the sd-app private volume.

Anecdotally, @creviera ran apt autoremove tasks in her long-running large template, and it cleaned up >3GB (!) of space.

[eloquence]

@conorsch @creviera I'm wondering if we can do more in terms of how we ship the grsec package(s) and declare their dependencies to clean up old kernels after each update. At least in anecdotal observations, most of the autoremove diskspace cleanup has been due to old kernel packages that are no longer needed still hanging around.

[conorsch]

At least in anecdotal observations, most of the autoremove diskspace cleanup has been due to old kernel packages that are no longer needed still hanging around.

See also #442

[zenmonkeykstop]

Flagging for @deeplow - if this functionality is available in the new Qubes updater we can close this, if not it might make sense to raise it upstream again. Fallback would be to do it ourselves after the updater run finishes.

[deeplow]

Took a peek at the code. It seems that the updater runs apt-get dist-upgrade by default (as opposed to apt-get upgrade).

@legoktm debian question: can you confirm if dist-upgrade does in fact include something equivalent to autoremove?

[marmarek]

Took a peek at the code.

That's not entirely right place - if python3-apt is installed, the API version is used: https://github.com/QubesOS/qubes-core-admin-linux/blob/a954e6595948aad7cf33f450d6144da40461230e/vmupdate/agent/source/apt/apt_api.py

BTW, in bookworm apt dist-upgrade seems to automatically remove old kernels (but it isn't full autoremove). But, apt-get dist-upgrade doesn't do that. May be worth looking how it's done, and make the updater do that too: QubesOS/qubes-issues#6676 (comment)

[deeplow]

That's not entirely right place - if python3-apt is installed, the API version is used: https://github.com/QubesOS/qubes-core-admin-linux/blob/a954e6595948aad7cf33f450d6144da40461230e/vmupdate/agent/source/apt/apt_api.py

Makes sense. Thanks!

BTW, in bookworm apt dist-upgrade seems to automatically remove old kernels (but it isn't full autoremove). But, apt-get dist-upgrade doesn't do that. May be worth looking how it's done, and make the updater do that too: QubesOS/qubes-issues#6676 (comment)

Assuming that kernels are indeed the biggest size culprit, then that could be a good approach. However, with this cleanup approach we're relying on the cleanup side-effect of dist-upgrade, which could behavior at some point. But if autoremove is too dangerous, clean to weak this may be the only path forward.

[legoktm]

@deeplow wrote:

@legoktm debian question: can you confirm if dist-upgrade does in fact include something equivalent to autoremove?

The difference between dist-upgrade and a regular upgrade is that the former will remove packages if necessary to resolve conflicts. Typically this happens when upgrading major versions (e.g. bullseye -> bookworm), hence the name "distribution upgrade". dist-upgrade does not also do autoremove, except for kernel packages it seems.

@marmarek wrote:

BTW, in bookworm apt dist-upgrade seems to automatically remove old kernels (but it isn't full autoremove). But, apt-get dist-upgrade doesn't do that. May be worth looking how it's done, and make the updater do that too: QubesOS/qubes-issues#6676 (comment)

Huh. apt is intended for humans, so it can be more aggressive since the expectation is that the proposed installs/removals will be reviewed ahead of time rather than apt-get, which is for scripting (it's also for humans, but explicitly supported for scripting). I can reproduce the difference in behavior between apt/apt-get as well, though I haven't (yet) been able to track down what is causing it.

[deeplow]

Thanks for all the context @legoktm!