DNF/yum removes ZFS on kernel updates (Fedora Core 38)

[tomchiverton]

dnf.log

2023-08-11T06:05:48+0100 DEBUG Installed: kernel-6.4.9-200.fc38.x86_64
2023-08-11T06:05:48+0100 DEBUG Installed: kernel-core-6.4.9-200.fc38.x86_64
2023-08-11T06:05:48+0100 DEBUG Installed: kernel-devel-6.4.9-200.fc38.x86_64
2023-08-11T06:05:48+0100 DEBUG Installed: kernel-modules-6.4.9-200.fc38.x86_64
2023-08-11T06:05:48+0100 DEBUG Installed: kernel-modules-core-6.4.9-200.fc38.x86_64 
2023-08-11T06:05:48+0100 DEBUG Removed: kernel-6.3.12-200.fc38.x86_64
2023-08-11T06:05:48+0100 DEBUG Removed: kernel-core-6.3.12-200.fc38.x86_64 
2023-08-11T06:05:48+0100 DEBUG Removed: kernel-devel-6.3.12-200.fc38.x86_64
2023-08-11T06:05:48+0100 DEBUG Removed: kernel-modules-6.3.12-200.fc38.x86_64
2023-08-11T06:05:48+0100 DEBUG Removed: kernel-modules-core-6.3.12-200.fc38.x86_64
2023-08-11T06:05:48+0100 DEBUG Removed: kmod-xtables-addons-6.3.12-200.fc38.x86_64-3.24-1.fc38.x86_64
2023-08-11T06:05:48+0100 DEBUG Removed: zfs-2.1.12-1.fc38.x86_64
2023-08-11T06:05:48+0100 DEBUG Removed: zfs-dkms-2.1.12-1.fc38.noarch

This should never be allowed to happen.

echo 'zfs' > /etc/dnf/protected.d/zfs.conf

Should be added to the RPM install script (and removed on removal)

[AllKind]

zfs-2.1.12
Supported Platforms

Linux: compatible with 3.10 - 6.3 kernels

[tomchiverton]

Note this isn't about what is or is not currently supported, though the above is a concrete example.

Adding two lines to the install/uninstall script would remove this footgun for any future incompatibility as well as the current one.

I'd submit a PR if I understood anything about how the RPMs are built.

[k-korn]

Adding two lines to the install/uninstall script would remove this footgun for any future incompatibility as well as the current one.

This will add the opposite footgun, e.g. "Having ZFS installed breaks updates on my Fedora!!11":

# dnf update
...
Error:
 Problem: The operation would result in removing the following protected packages: zfs-dkms
(try to add '--skip-broken' to skip uninstallable packages)

So yes, system administrator should be wary if they need ZFS non-removable or they need the latest updates w/o ZFS.

[tomchiverton]

That's better than rebooting and finding half your file system is missing? -- †øღ Sent from a super computer that fits in my pocket and is connected to the sum total of all human knowledge

…

On 3 September 2023 19:27:38 BST, Korn ***@***.***> wrote: > Adding two lines to the install/uninstall script would remove this footgun for any future incompatibility as well as the current one. This will add the opposite footgun, e.g. "Having ZFS installed breaks updates on my Fedora!!11": ``` # dnf update ... Error: Problem: The operation would result in removing the following protected packages: zfs-dkms (try to add '--skip-broken' to skip uninstallable packages) ``` So yes, system administrator should be wary if they need ZFS non-removable or they need the latest updates w/o ZFS. -- Reply to this email directly or view it on GitHub: #15188 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>

[AllKind]

Problem: The operation would result in removing the following protected packages: zfs-dkms
(try to add '--skip-broken' to skip uninstallable packages)

Then people would start using --skip-broken. -> The kernel gets updated to a version which is not compatible with the installed zfs version. Which could lead to disastrous results, if the dkms modules nevertheless successfully compile. If not, the result would be the same as you want to avoid. The zfs filesystem would not be accessible.

As a zfs user on a bleeding edge distro, you should know, that it always takes a few weeks until zfs catches up with the latest kernel versions.
This could only be solved, if zfs gets included in the kernel, which will likely never happen because of licensing reasons.

Edit: btw... never install updates unattended! ;-)

[tomchiverton]

It's been months now, not weeks, since ZFS from the repo stopped working with Fedora's latest kernel -- †øღ Sent from a super computer that fits in my pocket and is connected to the sum total of all human knowledge

…

On 4 September 2023 13:06:29 BST, AllKind ***@***.***> wrote: > Problem: The operation would result in removing the following protected packages: zfs-dkms > (try to add '--skip-broken' to skip uninstallable packages) Then people would start using --skip-broken. -> The kernel gets updated to a version which is not compatible with the installed zfs version. Which could lead to disastrous results, if the dkms modules nevertheless successfully compile. If not, the result would be the same as you want to avoid. The zfs filesystem would not be accessible. As a zfs user on a bleeding edge distro, you should know, that it always takes a few weeks until zfs catches up with the latest kernel versions. This could only be solved, if zfs gets included in the kernel, which will likely never happen because of licensing reasons. -- Reply to this email directly or view it on GitHub: #15188 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>

[mskarbek]

It's been months now, not weeks, since ZFS from the repo stopped working with Fedora's latest kernel -- †øღ Sent from a super computer that fits in my pocket and is connected to the sum total of all human knowledge

@tomchiverton are you paying for OpenZFS support or something like that? If not, what justifies your demanding behavior? And it's been one month, not months. Fedora released the latest 6.3 kernel in July.
It's your responsibility to track and match compatible kernel versions and skip unsupported updates until OpenZFS releases a new version.

[Gibson85]

I agree with @tomchiverton. I know it is or was vacation time, but these delays are annoying. Especially often one gets not a single hint at all that after update ZFS won't work anymore. With every shutdown I must be aware to uncheck the "install updates" combobox otherwise I would have a broken system again. And I had dozen broken systems in the last years because of ZFS. Fedora is one of the most used distributions if not the most used at all. There exists testing version for e.g. Fedora. So why is it so hard to implemented future kernels right in time? I don't understand it. And this has nothing to do whether it's free software or not. Its only a thing of "we don't want it to work better". And "more stable" distributions are not a solution also. E.g. I used Debian for years before. This "rock solid" (laughing) outdated distro had horrible bugs and even did not support encryption when I used it last.

[m-ueberall]

Especially often one gets not a single hint at all that after update ZFS won't work anymore. With every shutdown I must be aware to uncheck the "install updates" combobox otherwise I would have a broken system again.

As @mskarbek pointed out, in general, as long as you use software (packages) from different sources/repositories, it is your responsibility to track and match available versions.

@Gibson85: I'm using Ubuntu instead of Fedora on a daily basis, but the dnf documentation explicitly explains how to exclude (here: kernel) packages from transactions which is supposed to cover every direct/indirect interaction with dnf (including the combo box you mentioned) similar to what apt offers on Ubuntu/Debian. If this does not work as documented, this should IMHO be reported against dnf ASAP. OTOH, if your environment stops working because you cannot update certain sofware components for a couple of weeks due to the lack of matching available versions or whatever reason, maybe it makes sense to look into options how to circumvent this problem in the first place (this can/should be discussed elsewhere, though.)

Fedora might be one of the most used distributions, but it is after all but one distribution among all those distributions, operating systems that the OpenZFS project addresses. If a sufficient number of Fedora users need to use both current kernels and current OpenZFS versions, then why not convert this issue into a discussion, look for fellow campaigners, fork this repository, and establish/maintain a workflow that fits your needs? If this proves feasible over a certain period of time, maybe merging said fork with the main repository might be an option (if need be—that's debatable).

[tomchiverton]

I did offer to contribute fixes, if anyone could explain how the rpm is built, and where things like its pre/post scripts are sourced from -- †øღ Sent from a super computer that fits in my pocket and is connected to the sum total of all human knowledge

…

On 4 September 2023 19:46:03 BST, Markus Ueberall ***@***.***> wrote: > Especially often one gets not a single hint at all that after update ZFS won't work anymore. With every shutdown I must be aware to uncheck the "install updates" combobox otherwise I would have a broken system again. As @mskarbek pointed out, in general, as long as you use software (packages) from different sources/repositories, it is your responsibility to track and match available versions. I'm using Ubuntu instead of Fedora on a daily basis, but [the dnf documentation](https://docs.fedoraproject.org/en-US/quick-docs/dnf/#exclude-package) _explicitly_ explains how to exclude (here: kernel) packages from transactions which is supposed to cover every direct/indirect interaction with `dnf` (including the combo box you mentioned) similar to what `apt` offers on Ubuntu/Debian. If this does _not_ work as documented, this should IMHO be reported against `dnf` ASAP. OTOH, if your environment stops working because you cannot update certain sofware components for a couple of weeks due to the lack of matching available versions or whatever reason, maybe it makes sense to look into options how to circumvent this problem in the first case (this can/should be discussed elsewhere, though.) Fedora _might_ be one of the most used distributions, but it is after all _but one_ distribution among _all_ those distributions, operating systems that the OpenZFS project addresses. If a sufficient number of Fedora users _need_ to use both current kernels _and_ current OpenZFS versions, then why not convert this issue into a discussion, look for fellow campaigners, fork this repository, and establish/maintain a workflow that fits your needs? If this proves feasible over a certain period of time, maybe merging said fork with the main repository might be an option (if need be—that's debatable). -- Reply to this email directly or view it on GitHub: #15188 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

[ElCoyote27]

@tomchiverton to rebuild the rpms, I usually use this:
https://openzfs.github.io/openzfs-docs/Developer%20Resources/Custom%20Packages.html#get-the-source-code
the pre/post stuff in the rpm spec file.

[tomchiverton]

I looked into that, and that'd be how I'd test any change in the pre/post script or add an extra file to be installed to provide /etc/dnf/protected.d/zfs.conf

What I can't find in the repo is where those (or their sources) actually are.

[jkool702]

I have a good bit of experience with dnf and building custom RPM's from source. The behavior you describe that prompted this issue is sorta not a bug.

The logic the dnf uses here is that it will keep around RPM's that depend on a specific kernel version so long as you have at least 1 kernel installed that is a valid version for that RPM. You'll notice that dnf didnt try to uninstall zfs when you installed your first 6.4.x kernel. It did, however, [try to] uninstall it when you uninstalled your last remaining 6.3.x kernel.

If you think about it, this logic makes sense - it will keep zfs around so long as it can be used somewhere in the installed system (even if it cant be used with your current kernel), but as soon as there is no valid possible way to use it (because you dont have a kernel installed that it is compatible with) it automatically uninstalls it.

Funnily enough, I use Zfs on root and this problem is automatically a non-issue for me, since dnf will never remove the running kernel and i cant get into the system at all on a unsupported kernel, so I stay on the latest supported kernel until zfs catches up (since...well...I dont really have a choice in the matter), and as such dnf never tries to remove fs since there is always at least 1 zfs-supported kernel around. Its pretty well foolproof...youd have to do an offline dnf update from a live usb to accidentally delete zfs if you staty on using the last zfs-supported kernel.

As such, my advice would be to not change anything and either

1. keep on using the last zfs-supported kernel until a new zfs version is released, or
2. add the last zfs-supported kernel as a protected package until a new zfs version is released

Alternatively, if you dont care about not accessing zfs for a while, let dnf uninstall it and install it again when a new version comes out. uninstalling zfs shouldnt effect your pools...they will still be there when you reinstall. And its not like you could (or should) use it with an incompatible kernel (mean you sometimes can if you tweak the specfile and recompile it, and it actually compiles, but in this case you really shouldnt...assuming you dont want data corruption).

I see no reason to try and change this behavior...it seems fairly optimal; as-is.