Hold back incompatible kernel packages on Fedora

[tonyhutter]

System information

Type	Version/Name
Distribution Name	Fedora
Distribution Version	42
Kernel Version
Architecture
OpenZFS Version	master

Describe the problem you're observing

A user reported that when your upgrade your kernel packages on Fedora with ZFS installed, only the kernel-devel package gets held back to the ZFS-supported version, but not the other kernel packages. So if ZFS only supports the 6.13 kernel, Fedora will still happily upgrade the kernel RPMs to 6.14, but hold back kernel-devel at 6.13, for example.

I see d02257c makes mention of holding back the kernel version, but it only holds back kernel-devel. It seems we should be holding back the kernel packages as well.

Describe how to reproduce the problem

User reported upgrading kernel packages on Fedora causes the issue.

Include any warning/errors/backtraces from the system logs

[jkool702]

Thanks for creating this issue @tonyhutter .

I think the following change (adding explicit "Requires" dependencies for kernel and kernel-core in zfs-dkms.spec) would probably be sufficient to fix this:

jkool702@6e5e105

That said I havent tested it at all, nor have I deeply examined the RPM dependencies between the various kernel packages (which is why I didnt open a PR for it). But, it seems like holding back kernel, kernel-core and kernel-devel would have to result in basically all the core kernel packages being held back from DNF updates until zfs updates to a new version (perhaps some of the extra optional kernel packages like kernel-headers or kernel-tools might not be blocked?)

Just having "Requires" dependencies on the kernel package (which is a metapackage to pull in all the various required kernel packages) and on kernel-devel might be enough even, but Im not sure how exactly dependencies get propagated through the main kernel metapackage.

[jkool702]

@tonyhutter - Im not sure what openzfs's policy on backporting fixes to already released zfs versions is, but...assuming 2.3.2 isnt getting released in the next day or two...maybe it is worth doing the following:

1. backport tonyhutter@2c01d88 to zfs 2.3.0 and 2.3.1
2. run rpmdev-bumpspec the zfs-dkms.spec file in 2.3.0 and 2.3.1 (so dnf update will pull the new zfs-dkms package in for existing 2.3.0 and 2.3.1 users)
3. rebuild/replace the zfs-dkms.rpm packages for 2.3.0 and 2.3.1 in the openzfs public fedora repos

???

I mean, I completely understand not wanting to modify already released versions for the sake of stability, but...maybe this one is an exception? That commit cant really change anything about how zfs does zfs things, and it immediately solves a widespread and fairly major pain in the ass for people using zfs 2.3.x on Fedora.

[tonyhutter]

@jkool702 I would expect 99.9% of our Fedora users are going to be running zfs-2.3.1. It has usually taken about a week for everyone to auto-update to the new ZFS release RPMs due to dnf query settings. I think Fedora 40-41 are still running 6.13 kernels or less, so updating to the upcoming zfs-2.3.2 (hopefully with #17271 included) should fix the issue going forward.

[jkool702]

@tonyhutter - thats fair. I can confirm that Fedora 41's current kernel is 6.13.12.

Regarding that patch - you may actually want to explicitly set up a few more Requires/Conflicts dependencies. After some trial and error I manajed to generate a list of how all the kernel sub-packages depend on each other.

# COMMAND

dnf4 repoquery --deplist 'kernel' 'kernel-*' 2>/dev/null | grep -v 'provider' | grep -E '((^$)|(dependency: kernel)|(package: ))' | sed -E s/' ?[-=] ?6\.[0-9]+\.[0-9]+\-[0-9]+\.fc[0-9]+.*$'//| grep -v 'macros'| sed -zE s/'\n  dependency'/'  dependency'/g | sort -u | sed -zE 's/  dependency/\n  dependency/g; s/\npackage/\n\npackage/g'

#OUTPUT

package: kernel-core
  dependency: kernel-modules-core-uname-r

package: kernel-cross-headers

package: kernel-debug-core
  dependency: kernel-debug-modules-core-uname-r

package: kernel-debug
  dependency: kernel-debug-core-uname-r
  dependency: kernel-debug-modules-core-uname-r
  dependency: kernel-debug-modules-uname-r

package: kernel-debug-devel

package: kernel-debug-devel-matched
  dependency: kernel-debug-core
  dependency: kernel-debug-devel

package: kernel-debug-modules-core
  dependency: kernel-uname-r

package: kernel-debug-modules
  dependency: kernel-debug-modules-core-uname-r
  dependency: kernel-uname-r

package: kernel-debug-modules-extra
  dependency: kernel-debug-modules-core-uname-r
  dependency: kernel-debug-modules-uname-r
  dependency: kernel-uname-r

package: kernel-debug-modules-internal
  dependency: kernel-debug-modules-core-uname-r
  dependency: kernel-debug-modules-uname-r
  dependency: kernel-uname-r

package: kernel-debug-uki-virt-addons
  dependency: kernel-debug-uki-virt

package: kernel-debug-uki-virt
  dependency: kernel-debug-modules-core-uname-r

package: kernel
  dependency: kernel-core-uname-r
  dependency: kernel-modules-core-uname-r
  dependency: kernel-modules-uname-r

package: kernel-devel

package: kernel-devel-matched
  dependency: kernel-core
  dependency: kernel-devel

package: kernel-doc

package: kernel-headers

package: kernel-modules-core
  dependency: kernel-uname-r

package: kernel-modules
  dependency: kernel-modules-core-uname-r
  dependency: kernel-uname-r

package: kernel-modules-extra
  dependency: kernel-modules-core-uname-r
  dependency: kernel-modules-uname-r
  dependency: kernel-uname-r

package: kernel-modules-internal
  dependency: kernel-modules-core-uname-r
  dependency: kernel-modules-uname-r
  dependency: kernel-uname-r

package: kernel-selftests-internal

package: kernel-tools
  dependency: kernel-tools-libs

package: kernel-tools-libs

package: kernel-tools-libs-devel
  dependency: kernel-tools
  dependency: kernel-tools-libs

package: kernel-uki-virt-addons
  dependency: kernel-uki-virt

package: kernel-uki-virt
  dependency: kernel-modules-core-uname-r

Ignoring the debug kernel packages, I think the following list is the smallest set of kernel packages that (through their chained dependencies) will ensure all the possible kernel subpackages are blocked from being installed until zfs updates

kernel-devel-matched 
kernel
kernel-modules-core 
kernel-modules 
kernel-modules-extra 
kernel-modules-internal 
kernel-uki-virt-addons 
kernel-tools-libs-devel 
kernel-selftests-internal 
kernel-headers
kernel-cross-headers
kernel-doc

ALSO, I think that having zfs-dkms pull in literally all the kernel subpackages is probably a bit...heavy handed.Id imagine the ideal behavior here would be to list only the packages kernel; packages that zfs actually requires (which is just kernel-devel-matched i think??) as both "Requires" and "Conflicts" dependencies and everything else only as "Conflicts" dependencies. I think (but have NOT tested that) this will keep all the kernel subpackages that are installed on the same kernel version (since installing their update would conflict with zfs-dkms) but would NOT cause them all to be pulled in and installed. Which would make the final version of that part of the spec file look something like

%if 0%{?rhel}%{?fedora}%{?mageia}%{?suse_version}%{?openEuler}
Requires:       kernel-devel-matched >= @ZFS_META_KVER_MIN@, kernel-devel-matched <= @ZFS_META_KVER_MAX@.999
Requires(post): kernel-devel-matched >= @ZFS_META_KVER_MIN@, kernel-devel-matched <= @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-devel-matched < @ZFS_META_KVER_MIN@, kernel-devel-matched > @ZFS_META_KVER_MAX@.999
Conflicts:      kernel < @ZFS_META_KVER_MIN@, kernel > @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-modules-core < @ZFS_META_KVER_MIN@, kernel-modules-core > @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-modules < @ZFS_META_KVER_MIN@, kernel-modules > @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-modules-extra < @ZFS_META_KVER_MIN@, kernel-modules-extra > @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-modules-internal < @ZFS_META_KVER_MIN@, kernel-modules-internal > @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-uki-virt-addons < @ZFS_META_KVER_MIN@, kernel-uki-virt-addons > @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-tools-libs-devel < @ZFS_META_KVER_MIN@, kernel-tools-libs-devel > @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-selftests-internal < @ZFS_META_KVER_MIN@, kernel-selftests-internal > @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-headers < @ZFS_META_KVER_MIN@, kernel-headers > @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-cross-headers < @ZFS_META_KVER_MIN@, kernel-cross-headers > @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-doc < @ZFS_META_KVER_MIN@, kernel-doc > @ZFS_META_KVER_MAX@.999
Obsoletes:      spl-dkms <= %{version}
%endif

If, for reasons Im not currently seeing, the "Requires" part has to be there to make this all work then...i guess figure out which ones are not going to cause problems if a version for a newer kernel installed and then all the rest become explicit dependencies of zfs-dkms???

[jkool702]

@tonyhutter - I tested it and turns out just using "Requires" for kernel-devel-matched didnt work. However, expanding it to the following seems to work

%if 0%{?rhel}%{?fedora}%{?mageia}%{?suse_version}%{?openEuler}
# Hold back kernel upgrades if kernel is not supported by ZFS
Requires:       kernel-devel-matched >= @ZFS_META_KVER_MIN@, kernel-devel-matched <= @ZFS_META_KVER_MAX@.999
Requires(post): kernel-devel-matched >= @ZFS_META_KVER_MIN@, kernel-devel-matched <= @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-devel-matched < @ZFS_META_KVER_MIN@, kernel-devel-matched > @ZFS_META_KVER_MAX@.999
Requires:       kernel-devel >= @ZFS_META_KVER_MIN@, kernel-devel <= @ZFS_META_KVER_MAX@.999
Requires(post): kernel-devel >= @ZFS_META_KVER_MIN@, kernel-devel <= @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-devel < @ZFS_META_KVER_MIN@, kernel-devel > @ZFS_META_KVER_MAX@.999
Requires:       kernel-core >= @ZFS_META_KVER_MIN@, kernel-core <= @ZFS_META_KVER_MAX@.999
Requires(post): kernel-core >= @ZFS_META_KVER_MIN@, kernel-core <= @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-core < @ZFS_META_KVER_MIN@, kernel-core > @ZFS_META_KVER_MAX@.999
Requires:       kernel >= @ZFS_META_KVER_MIN@, kernel <= @ZFS_META_KVER_MAX@.999
Requires(post): kernel >= @ZFS_META_KVER_MIN@, kernel <= @ZFS_META_KVER_MAX@.999
Conflicts:      kernel < @ZFS_META_KVER_MIN@, kernel > @ZFS_META_KVER_MAX@.999
Requires:       kernel-modules >= @ZFS_META_KVER_MIN@, kernel-modules <= @ZFS_META_KVER_MAX@.999
Requires(post): kernel-modules >= @ZFS_META_KVER_MIN@, kernel-modules <= @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-modules < @ZFS_META_KVER_MIN@, kernel-modules > @ZFS_META_KVER_MAX@.999
Requires:       kernel-modules-core >= @ZFS_META_KVER_MIN@, kernel-modules-core <= @ZFS_META_KVER_MAX@.999
Requires(post): kernel-modules-core >= @ZFS_META_KVER_MIN@, kernel-modules-core <= @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-modules-core < @ZFS_META_KVER_MIN@, kernel-modules-core > @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-modules-extra < @ZFS_META_KVER_MIN@, kernel-modules-extra > @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-modules-internal < @ZFS_META_KVER_MIN@, kernel-modules-internal > @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-uki-virt-addons < @ZFS_META_KVER_MIN@, kernel-uki-virt-addons > @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-tools-libs-devel < @ZFS_META_KVER_MIN@, kernel-tools-libs-devel > @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-selftests-internal < @ZFS_META_KVER_MIN@, kernel-selftests-internal > @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-headers < @ZFS_META_KVER_MIN@, kernel-headers > @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-cross-headers < @ZFS_META_KVER_MIN@, kernel-cross-headers > @ZFS_META_KVER_MAX@.999
Conflicts:      kernel-doc < @ZFS_META_KVER_MIN@, kernel-doc > @ZFS_META_KVER_MAX@.999
Obsoletes:      spl-dkms <= %{version}
%endif

To test, i recompiled the zfs-2.3-release branch with the above code substituted in rpm/generic/zfs-dkms.spec.in, built the zfs-dkms.rpm package, then reinstalled zfs-dkms using the new modified one.

I then took the fedora 42 kernel (6.14.x) and re-compiled it into tthe various fedora 41 kernel rpms. I added these to a local DNF repository.

running dnf update currently gives me

Updating and loading repositories:
Repositories loaded.
Problem 1: installed package zfs-dkms-2.3.1-1.fc41.noarch conflicts with kernel > 6.13.999 provided by kernel-6.14.3-300.fc41.x86_64 from localrepo-all
  - cannot install the best update candidate for package zfs-dkms-2.3.1-1.fc41.noarch
  - cannot install the best update candidate for package kernel-6.12.15-200.fc41.x86_64
 Problem 2: problem with installed package
  - installed package zfs-dkms-2.3.1-1.fc41.noarch conflicts with kernel > 6.13.999 provided by kernel-6.14.3-300.fc41.x86_64 from localrepo-all
  - package zfs-dkms-2.3.1-1.fc41.noarch from localrepo-all conflicts with kernel > 6.13.999 provided by kernel-6.14.3-300.fc41.x86_64 from localrepo-all
  - package zfs-dkms-2.3.1-1.fc41.noarch from localrepo-built conflicts with kernel > 6.13.999 provided by kernel-6.14.3-300.fc41.x86_64 from localrepo-all
  - package zfs-dkms-2.3.1-1.fc41.noarch from localrepo-built-41 conflicts with kernel > 6.13.999 provided by kernel-6.14.3-300.fc41.x86_64 from localrepo-all
  - package zfs-dkms-2.3.1-1.fc41.noarch from localrepo-built-41-noarch conflicts with kernel > 6.13.999 provided by kernel-6.14.3-300.fc41.x86_64 from localrepo-all
  - cannot install the best update candidate for package kernel-6.13.11-200.fc41.x86_64

Package                                                                                                 Arch             Version                                                                                                 Repository                                                                             Size
Upgrading:
 ImageMagick                                                                                            x86_64           1:7.1.1.47-1.fc41                                                                                       updates                                                                            88.4 KiB
   replacing ImageMagick                                                                                x86_64           1:7.1.1.46-1.fc41                                                                                       updates                                                                            88.4 KiB
 ImageMagick-libs                                                                                       x86_64           1:7.1.1.47-1.fc41                                                                                       updates                                                                             9.1 MiB
   replacing ImageMagick-libs                                                                           x86_64           1:7.1.1.46-1.fc41                                                                                       updates                                                                             9.1 MiB
 akmod-nvidia                                                                                           x86_64           3:570.144-1.fc41                                                                                        rpmfusion-nonfree-nvidia-driver                                                    92.2 KiB
   replacing akmod-nvidia                                                                               x86_64           3:570.133.07-1.fc41                                                                                     rpmfusion-nonfree-nvidia-driver                                                    92.2 KiB
 hplip                                                                                                  x86_64           3.25.2-2.fc41                                                                                           updates                                                                            29.3 MiB
   replacing hplip                                                                                      x86_64           3.24.4-4.fc41                                                                                           updates                                                                            29.1 MiB
 hplip-common                                                                                           x86_64           3.25.2-2.fc41                                                                                           updates                                                                             1.5 MiB
   replacing hplip-common                                                                               x86_64           3.24.4-4.fc41                                                                                           updates                                                                             1.4 MiB
 hplip-libs                                                                                             x86_64           3.25.2-2.fc41                                                                                           updates                                                                           445.7 KiB
   replacing hplip-libs                                                                                 x86_64           3.24.4-4.fc41                                                                                           updates                                                                           445.7 KiB
 kmod-nvidia                                                                                            x86_64           3:570.144-1.fc41                                                                                        rpmfusion-nonfree-nvidia-driver                                                     0.0   B
   replacing kmod-nvidia                                                                                x86_64           3:570.133.07-1.fc41                                                                                     rpmfusion-nonfree-nvidia-driver                                                     0.0   B
 libsane-hpaio                                                                                          x86_64           3.25.2-2.fc41                                                                                           updates                                                                           185.5 KiB
   replacing libsane-hpaio                                                                              x86_64           3.24.4-4.fc41                                                                                           updates                                                                           185.5 KiB
 mock-rpmfusion-free                                                                                    noarch           42.1-1.fc41                                                                                             rpmfusion-free-updates                                                             16.1 KiB
   replacing mock-rpmfusion-free                                                                        noarch           42.0-1.fc41                                                                                             rpmfusion-free-updates                                                             16.1 KiB
 mock-rpmfusion-nonfree                                                                                 noarch           42.1-1.fc41                                                                                             rpmfusion-nonfree-updates                                                          14.8 KiB
   replacing mock-rpmfusion-nonfree                                                                     noarch           42.0-1.fc43                                                                                             rpmfusion-nonfree-rawhide                                                          14.8 KiB
 nvidia-modprobe                                                                                        x86_64           3:570.144-1.fc41                                                                                        rpmfusion-nonfree-nvidia-driver                                                    51.0 KiB
   replacing nvidia-modprobe                                                                            x86_64           3:570.133.07-1.fc41                                                                                     copr:copr.fedorainfracloud.org:kwizart:nvidia-driver-rawhide                       51.0 KiB
 nvidia-persistenced                                                                                    x86_64           3:570.144-1.fc41                                                                                        rpmfusion-nonfree-nvidia-driver                                                    58.5 KiB
   replacing nvidia-persistenced                                                                        x86_64           3:570.133.07-1.fc41                                                                                     copr:copr.fedorainfracloud.org:kwizart:nvidia-driver-rawhide                       58.5 KiB
 nvidia-settings                                                                                        x86_64           3:570.144-1.fc41                                                                                        rpmfusion-nonfree-nvidia-driver                                                     4.4 MiB
   replacing nvidia-settings                                                                            x86_64           3:570.133.07-1.fc41                                                                                     copr:copr.fedorainfracloud.org:kwizart:nvidia-driver-rawhide                        4.4 MiB
 nvidia-xconfig                                                                                         x86_64           3:570.144-1.fc41                                                                                        rpmfusion-nonfree-nvidia-driver                                                   202.7 KiB
   replacing nvidia-xconfig                                                                             x86_64           3:570.133.07-1.fc41                                                                                     copr:copr.fedorainfracloud.org:kwizart:nvidia-driver-rawhide                      202.7 KiB
 perf                                                                                                   x86_64           6.14.3-300.fc41                                                                                         localrepo-all                                                                      10.8 MiB
   replacing perf                                                                                       x86_64           6.13.12-200.fc41                                                                                        updates                                                                            10.8 MiB
 python3-perf                                                                                           x86_64           6.14.3-300.fc41                                                                                         localrepo-all                                                                       8.4 MiB
   replacing python3-perf                                                                               x86_64           6.13.12-200.fc41                                                                                        updates                                                                             8.8 MiB
 selinux-policy                                                                                         noarch           41.38-1.fc41                                                                                            updates                                                                            31.6 KiB
   replacing selinux-policy                                                                             noarch           41.37-1.fc41                                                                                            updates                                                                            31.6 KiB
 selinux-policy-devel                                                                                   noarch           41.38-1.fc41                                                                                            updates                                                                            22.9 MiB
   replacing selinux-policy-devel                                                                       noarch           41.37-1.fc41                                                                                            updates                                                                            22.9 MiB
 selinux-policy-targeted                                                                                noarch           41.38-1.fc41                                                                                            updates                                                                            18.9 MiB
   replacing selinux-policy-targeted                                                                    noarch           41.37-1.fc41                                                                                            updates                                                                            18.9 MiB
 swtpm                                                                                                  x86_64           0.9.0-8.fc41                                                                                            updates                                                                            54.2 KiB
   replacing swtpm                                                                                      x86_64           0.9.0-7.fc41                                                                                            updates                                                                            54.2 KiB
 swtpm-libs                                                                                             x86_64           0.9.0-8.fc41                                                                                            updates                                                                           103.0 KiB
   replacing swtpm-libs                                                                                 x86_64           0.9.0-7.fc41                                                                                            updates                                                                           103.0 KiB
 swtpm-selinux                                                                                          noarch           0.9.0-8.fc41                                                                                            updates                                                                           252.0 KiB
   replacing swtpm-selinux                                                                              noarch           0.9.0-7.fc41                                                                                            updates                                                                           252.0 KiB
 swtpm-tools                                                                                            x86_64           0.9.0-8.fc41                                                                                            updates                                                                           268.8 KiB
   replacing swtpm-tools                                                                                x86_64           0.9.0-7.fc41                                                                                            updates                                                                           268.9 KiB
 xorg-x11-drv-nvidia                                                                                    x86_64           3:570.144-1.fc41                                                                                        rpmfusion-nonfree-nvidia-driver                                                   190.6 MiB
   replacing xorg-x11-drv-nvidia                                                                        x86_64           3:570.133.07-1.fc41                                                                                     rpmfusion-nonfree-nvidia-driver                                                   190.5 MiB
 xorg-x11-drv-nvidia-cuda                                                                               x86_64           3:570.144-1.fc41                                                                                        rpmfusion-nonfree-nvidia-driver                                                     6.2 MiB
   replacing xorg-x11-drv-nvidia-cuda                                                                   x86_64           3:570.133.07-1.fc41                                                                                     rpmfusion-nonfree-nvidia-driver                                                     6.2 MiB
 xorg-x11-drv-nvidia-cuda-libs                                                                          x86_64           3:570.144-1.fc41                                                                                        rpmfusion-nonfree-nvidia-driver                                                   273.5 MiB
   replacing xorg-x11-drv-nvidia-cuda-libs                                                              x86_64           3:570.133.07-1.fc41                                                                                     rpmfusion-nonfree-nvidia-driver                                                   273.5 MiB
 xorg-x11-drv-nvidia-devel                                                                              x86_64           3:570.144-1.fc41                                                                                        rpmfusion-nonfree-nvidia-driver                                                     0.0   B
   replacing xorg-x11-drv-nvidia-devel                                                                  x86_64           3:570.133.07-1.fc41                                                                                     rpmfusion-nonfree-nvidia-driver                                                     0.0   B
 xorg-x11-drv-nvidia-kmodsrc                                                                            x86_64           3:570.144-1.fc41                                                                                        rpmfusion-nonfree-nvidia-driver                                                    75.7 MiB
   replacing xorg-x11-drv-nvidia-kmodsrc                                                                x86_64           3:570.133.07-1.fc41                                                                                     rpmfusion-nonfree-nvidia-driver                                                    76.0 MiB
 xorg-x11-drv-nvidia-libs                                                                               x86_64           3:570.144-1.fc41                                                                                        rpmfusion-nonfree-nvidia-driver                                                   361.9 MiB
   replacing xorg-x11-drv-nvidia-libs                                                                   x86_64           3:570.133.07-1.fc41                                                                                     rpmfusion-nonfree-nvidia-driver                                                   361.9 MiB
 xorg-x11-drv-nvidia-power                                                                              x86_64           3:570.144-1.fc41                                                                                        rpmfusion-nonfree-nvidia-driver                                                   233.7 KiB
   replacing xorg-x11-drv-nvidia-power                                                                  x86_64           3:570.133.07-1.fc41                                                                                     rpmfusion-nonfree-nvidia-driver                                                   233.7 KiB
 xorg-x11-drv-nvidia-xorg-libs                                                                          x86_64           3:570.144-1.fc41                                                                                        rpmfusion-nonfree-nvidia-driver                                                    19.4 MiB
   replacing xorg-x11-drv-nvidia-xorg-libs                                                              x86_64           3:570.133.07-1.fc41                                                                                     rpmfusion-nonfree-nvidia-driver                                                    19.4 MiB
Skipping packages with conflicts:
 kernel                                                                                                 x86_64           6.14.3-300.fc41                                                                                         localrepo-all                                                                       0.0   B

Transaction Summary:
 Upgrading:         31 packages
 Replacing:         31 packages
 Skipping:           1 package

Total size of inbound packages is 391 MiB. Need to download 391 MiB.
After this operation, 338 KiB will be freed (install 1 GiB, remove 1 GiB).
Is this ok [y/N]: 

which seems to be the correct behavior - dnf doesnt install any of the kernel packages for 6.14.3 but otherwise works normally.

Its worth noting that the 6.14.3 perf packages (that were not blocked by the new zfs-dkms.spec file dependencies) are still being pulled in. This is expected. Technically these are sort of kernel packages (they come from building kernel.src.rpm). Having these install early is probably harmless, but adding a few more "Conflicts" lines to block these is easy enough if desired.

[Chaots]

I just ran into this issue on a F41 machine too that was last update about 4 weeks ago, i did a dnf clean all && dnf -y update i now know i should not have done. This resulted in Fedora uninstalling/installing some needed dependencies relevant parts of the output i could recover from the window history:

[1002/1265] Removing kernel-modules-core-0:6.12.10-200.fc41.x86_64                                                                                                     100% |   2.7 KiB/s |   3.1 KiB |  00m01s
>>> Running pre-uninstall scriptlet: kernel-core-0:6.12.10-200.fc41.x86_64                                                                                                                                     
>>> Finished pre-uninstall scriptlet: kernel-core-0:6.12.10-200.fc41.x86_64                                                                                                                                    
>>> Scriptlet output:                                                                                                                                                                                          
>>> dkms: removing module zfs/2.3.1 for kernel 6.12.10-200.fc41.x86_64 (x86_64)                                                                                                                                
>>> Module zfs/2.3.1 for kernel 6.12.10-200.fc41.x86_64 (x86_64):                                                                                                                                              
>>> Before uninstall, this module version was ACTIVE on this kernel.                                                                                                                                           
>>> Deleting /lib/modules/6.12.10-200.fc41.x86_64/extra/zfs.ko.xz                                                                                                                                              
>>> Deleting /lib/modules/6.12.10-200.fc41.x86_64/extra/spl.ko.xz                                                                                                                                              
>>>                                                                                                                                                                                                            
>>> Running depmod... done.                                                                                                                                                                                    
>>>                                                          

and then at the end of the update run:

>>> Running post-transaction scriptlet: kernel-core-0:6.14.3-200.fc41.x86_64                                                                                                                                   
>>> Non-critical error in post-transaction scriptlet: kernel-core-0:6.14.3-200.fc41.x86_64                                                                                                                     
>>> Scriptlet output:                                                                                                                                                                                          
>>>                                                                                                                                                                                                            
>>> Error! zfs/2.2.3 is broken! Missing the source directory or the symbolic link pointing to it.                                                                                                              
>>> Manual intervention is required!                                                                                                                                                                           
>>>                                                                                                                                                                                                            
>>> Error! zfs/2.2.4 is broken! Missing the source directory or the symbolic link pointing to it.                                                                                                              
>>> Manual intervention is required!                                                                                                                                                                           
>>>                                                                                                                                                                                                            
>>> Error! zfs/2.2.5 is broken! Missing the source directory or the symbolic link pointing to it.                                                                                                              
>>> Manual intervention is required!                                                                                                                                                                           
>>>                                                                                                                                                                                                            
>>> Error! Your kernel headers for kernel 6.14.3-200.fc41.x86_64 cannot be found at /lib/modules/6.14.3-200.fc41.x86_64/build or /lib/modules/6.14.3-200.fc41.x86_64/source.                                   
>>> Please install the linux-headers-6.14.3-200.fc41.x86_64 package or use the --kernelsourcedir option to tell DKMS where it's located.                                                                       
>>> /usr/lib/kernel/install.d/40-dkms.install failed with exit status 21.                                                                                                                                      
>>>                                                                                                                                                                                                            
>>> [RPM] %posttrans(kernel-core-6.14.3-200.fc41.x86_64) scriptlet failed, exit status 21                                                                                                                      
Complete!

After this i tried to run another update to see what the issue for the previous run was.

sudo dnf update
Updating and loading repositories:
 Fedora 41 openh264 (From Cisco) - x86_64                                                                                                                              100% |   9.5 KiB/s |   6.0 KiB |  00m01s
 Fedora 41 - x86_64                                                                                                                                                    100% |  18.7 MiB/s |  35.3 MiB |  00m02s
 Fedora 41 - x86_64 - Updates                                                                                                                                          100% |  15.7 MiB/s |  12.6 MiB |  00m01s
 RPM Fusion for Fedora 41 - Free                                                                                                                                       100% | 224.4 KiB/s | 158.7 KiB |  00m01s
 RPM Fusion for Fedora 41 - Free - Updates                                                                                                                             100% | 279.3 KiB/s |  63.7 KiB |  00m00s
 RPM Fusion for Fedora 41 - Nonfree - Updates                                                                                                                          100% | 152.8 KiB/s |  34.7 KiB |  00m00s
 RPM Fusion for Fedora 41 - Nonfree                                                                                                                                    100% | 382.2 KiB/s |  86.8 KiB |  00m00s
 ZFS on Linux for Fedora 41                                                                                                                                            100% |  18.9 KiB/s |  20.2 KiB |  00m01s
Repositories loaded.
Problem 1: installed package zfs-dkms-2.3.1-1.fc41.noarch conflicts with kernel-devel > 6.13.999 provided by kernel-devel-6.14.3-200.fc41.x86_64 from updates
  - cannot install the best update candidate for package zfs-dkms-2.3.1-1.fc41.noarch
  - cannot install the best update candidate for package kernel-devel-6.12.10-200.fc41.x86_64
 Problem 2: installed package zfs-2.3.1-1.fc41.x86_64 requires zfs-kmod = 2.3.1, but none of the providers can be installed
  - installed package zfs-dkms-2.3.1-1.fc41.noarch conflicts with kernel-devel > 6.13.999 provided by kernel-devel-6.14.3-200.fc41.x86_64 from updates
  - package zfs-dkms-2.3.1-1.fc41.noarch from zfs conflicts with kernel-devel > 6.13.999 provided by kernel-devel-6.14.3-200.fc41.x86_64 from updates
  - cannot install the best update candidate for package zfs-2.3.1-1.fc41.x86_64
  - cannot install the best update candidate for package kernel-devel-6.12.11-200.fc41.x86_64
 Problem 3: installed package libvirt-daemon-driver-storage-zfs-10.6.0-7.fc41.x86_64 requires /sbin/zfs, but none of the providers can be installed
  - installed package libvirt-daemon-driver-storage-zfs-10.6.0-7.fc41.x86_64 requires /sbin/zpool, but none of the providers can be installed
  - package zfs-2.2.7-1.fc41.x86_64 from zfs requires libnvpair3(x86-64) = 2.2.7-1.fc41, but none of the providers can be installed
  - installed package zfs-2.3.1-1.fc41.x86_64 requires zfs-kmod = 2.3.1, but none of the providers can be installed
  - package zfs-2.3.1-1.fc41.x86_64 from zfs requires zfs-kmod = 2.3.1, but none of the providers can be installed
  - package zfs-2.3.0-1.fc41.x86_64 from zfs requires zfs-kmod = 2.3.0, but none of the providers can be installed
  - cannot install both libnvpair3-2.2.7-1.fc41.x86_64 from zfs and libnvpair3-2.3.1-1.fc41.x86_64 from @System
  - installed package zfs-dkms-2.3.1-1.fc41.noarch conflicts with kernel-devel > 6.13.999 provided by kernel-devel-6.14.3-200.fc41.x86_64 from updates
  - package zfs-dkms-2.3.1-1.fc41.noarch from zfs conflicts with kernel-devel > 6.13.999 provided by kernel-devel-6.14.3-200.fc41.x86_64 from updates
  - installed package kernel-devel-6.13.8-200.fc41.x86_64 conflicts with kernel-devel > 6.12.999 provided by zfs-dkms-2.3.0-1.fc41.noarch from zfs
  - cannot install the best update candidate for package libvirt-daemon-driver-storage-zfs-10.6.0-7.fc41.x86_64
  - cannot install the best update candidate for package libnvpair3-2.3.1-1.fc41.x86_64
  - cannot install the best update candidate for package kernel-devel-6.13.8-200.fc41.x86_64
  - problem with installed package
  - package zfs-fuse-0.7.2.2-31.fc41.x86_64 from fedora is filtered out by exclude filtering
 Problem 4: problem with installed package
  - installed package zfs-2.3.1-1.fc41.x86_64 requires zfs-kmod = 2.3.1, but none of the providers can be installed
  - package zfs-2.3.1-1.fc41.x86_64 from zfs requires zfs-kmod = 2.3.1, but none of the providers can be installed
  - package zfs-2.2.7-1.fc41.x86_64 from zfs requires libuutil3(x86-64) = 2.2.7-1.fc41, but none of the providers can be installed
  - package zfs-2.3.0-1.fc41.x86_64 from zfs requires libuutil3(x86-64) = 2.3.0-1.fc41, but none of the providers can be installed
  - installed package zfs-dkms-2.3.1-1.fc41.noarch conflicts with kernel-devel > 6.13.999 provided by kernel-devel-6.14.3-200.fc41.x86_64 from updates
  - package zfs-dkms-2.3.1-1.fc41.noarch from zfs conflicts with kernel-devel > 6.13.999 provided by kernel-devel-6.14.3-200.fc41.x86_64 from updates
  - cannot install both libuutil3-2.2.7-1.fc41.x86_64 from zfs and libuutil3-2.3.1-1.fc41.x86_64 from @System
  - cannot install both libuutil3-2.3.0-1.fc41.x86_64 from zfs and libuutil3-2.3.1-1.fc41.x86_64 from @System
  - package kernel-devel-matched-6.14.3-200.fc41.x86_64 from updates requires kernel-devel = 6.14.3-200.fc41, but none of the providers can be installed
  - cannot install the best update candidate for package libuutil3-2.3.1-1.fc41.x86_64
  - cannot install the best update candidate for package kernel-devel-matched-6.13.8-200.fc41.x86_64

Package                                                                Arch            Version                                                                Repository                                   Size
Skipping packages with conflicts:
 kernel-devel                                                          x86_64          6.14.3-200.fc41                                                        updates                                  78.3 MiB
 libnvpair3                                                            x86_64          2.2.7-1.fc41                                                           zfs                                     106.3 KiB
 libuutil3                                                             x86_64          2.2.7-1.fc41                                                           zfs                                      62.9 KiB
 libuutil3                                                             x86_64          2.3.0-1.fc41                                                           zfs                                      62.9 KiB
 zfs-dkms                                                              noarch          2.3.0-1.fc41                                                           zfs                                      57.4 MiB
Skipping packages with broken dependencies:
 kernel-devel-matched                                                  x86_64          6.14.3-200.fc41                                                        updates                                   0.0   B
 zfs                                                                   x86_64          2.2.7-1.fc41                                                           zfs                                       1.8 MiB
 zfs                                                                   x86_64          2.3.0-1.fc41                                                           zfs                                       1.9 MiB
 zfs                                                                   x86_64          2.3.1-1.fc41                                                           zfs                                       2.0 MiB

Nothing to do.

That made me find this report, given that it was mentioned it F41 would not be affected due to it's current kernel version, i made this post as it seems F41 now upgrades beyond the supported zfs packages.

I am not sure if the information is helpful, though as a mere mortal Linux user it caused me some trouble. I am sure I'll be able to get back to the previous state one way or another until the the 2.3.2 release from #17214 seeps through and likely fixes the problems "auto-magically"

Thanks for all the work all of you do!

[krestaino]

I also ran into this issue today. I blindly updated my Fedora Server 41 in Cockpit, which installed 6.14 and resulted in a kernel panic, preventing me from booting. I am now booting into 6.13 and manually blocking kernel updates.

$ cat /etc/dnf/dnf.conf

[main]
exclude=kernel* kernel-devel*

[tuxmaster5000]

@jkool702 I would expect 99.9% of our Fedora users are going to be running zfs-2.3.1. It has usually taken about a week for everyone to auto-update to the new ZFS release RPMs due to dnf query settings. I think Fedora 40-41 are still running 6.13 kernels or less, so updating to the upcoming zfs-2.3.2 (hopefully with #17271 included) should fix the issue going forward.

No, since this weekend Fedora-40 and newer will use 6.14.