zfs-2.1.6 breaks boot sequence on Debian bookworm/sid

[szubersk]

System information

Type	Version/Name
Distribution Name	Linux laptop.local 5.18.0-4-amd64 #1 SMP PREEMPT_DYNAMIC Debian 5.18.16-1 (2022-08-10) x86_64 GNU/Linux
Distribution Version	Linux laptop.local 5.18.0-4-amd64 #1 SMP PREEMPT_DYNAMIC Debian 5.18.16-1 (2022-08-10) x86_64 GNU/Linux
Kernel Version	Linux laptop.local 5.18.0-4-amd64 #1 SMP PREEMPT_DYNAMIC Debian 5.18.16-1 (2022-08-10) x86_64 GNU/Linux
Architecture	Linux laptop.local 5.18.0-4-amd64 #1 SMP PREEMPT_DYNAMIC Debian 5.18.16-1 (2022-08-10) x86_64 GNU/Linux
OpenZFS Version	zfs-2.1.6-1 zfs-kmod-2.1.6-1

Describe the problem you're observing

Oct 10 22:01:50 laptop.local systemd[1]: Hostname set to <laptop.local>.                                                           
Oct 10 22:01:50 laptop.local systemd[1]: sysinit.target: Found ordering cycle on local-fs.target/start                             
Oct 10 22:01:50 laptop.local systemd[1]: sysinit.target: Found dependency on zfs-mount.service/start                               
Oct 10 22:01:50 laptop.local systemd[1]: sysinit.target: Found dependency on zfs-load-key.service/start                            
Oct 10 22:01:50 laptop.local systemd[1]: sysinit.target: Found dependency on basic.target/start                                    
Oct 10 22:01:50 laptop.local systemd[1]: sysinit.target: Found dependency on sysinit.target/start                                  
Oct 10 22:01:50 laptop.local systemd[1]: sysinit.target: Job local-fs.target/start deleted to break ordering cycle starting with sysinit.target/start

Emergency workaround:
botch /etc/init.d/zfs-load-key and remove # X-Start-Before: zfs-mount line.

Describe how to reproduce the problem

Boot Debian bookworm/sid with zfs-2.1.6 installed.

Include any warning/errors/backtraces from the system logs

See above.

[gmelikov]

Don't know why it didn't blast before 2.1.6, but my thought is:

• zfs-load-key is generated from https://github.com/openzfs/zfs/blob/master/etc/init.d/zfs-load-key.in via systemd-sysv-generator
• by default systemd adds default.target to all services, if DefaultDependencies=no wasn't explicitly set
• default.target usually means all local FSes are already mounted -> waits for zfs-mount, boom, blast
• I don't see that systemd-sysv-generator supports for DefaultDependencies=no https://github.com/systemd/systemd/blob/main/src/sysv-generator/sysv-generator.c
• Maybe worse https://www.freedesktop.org/software/systemd/man/systemd-sysv-generator.html systemd does not support SysV scripts as part of early boot, so all wrapper units are ordered after basic.target.

I can reproduce it too on Debian testing aka Bookworm update from 2.1.5 to 2.1.6

[gmelikov]

Thankfully, real service file overrides sysv generator, so hotfix without breaking services start order is to copy-paste generator result and add needed line, or easier:

# cat > /lib/systemd/system/zfs-load-key.service
[Unit]
SourcePath=/etc/init.d/zfs-load-key
Description=Load ZFS keys for filesystems and volumes
Before=zfs-mount.service
After=zfs-import.service
DefaultDependencies=no

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
SuccessExitStatus=5 6
ExecStart=/etc/init.d/zfs-load-key start
ExecStop=/etc/init.d/zfs-load-key stop

[gmelikov]

I was sure zfs-load-key was included in 2.1.5, but it wasn't? https://salsa.debian.org/zfsonlinux-team/zfs/-/commit/1fc2ed09cb5036e2233219cd6c6385274085c8c3

[nabijaczleweli]

Why in the world is that even being included in the boot? It really shouldn't be.

[gmelikov]

@cdluminate looks like it's a regression in Debian downstream https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1017911

[nabijaczleweli]

Ha! It's because it's the only classic init script that doesn't have a basename service equivalent – having x.service suppresses generating for /etc/init.d/x.

@szubersk try ln -s /dev/null /etc/systemd/system/zfs-mount.service, worked for me in a dry testbed

This SysV init script should never (need to) be run under systemd, this domain is covered by zfs-mount-generator; all the other ones are suppressed implicitly due to above

[gmelikov]

@nabijaczleweli zfs-mount may be used, I think ln -s /dev/null /etc/systemd/system/zfs-load-key.service may be better as a one-line hotfix, or use #14010 (comment)

Not sure why it was included then, it really worked via zfs-mount-generator for me on previous 2.1* releases (but I thought that it worked via zfs-load-key :) ).

[nabijaczleweli]

I've updated my comment with notes for "why" – as for how we should fix it: idk, to be figured out.

For Debian, lumin@ can probably put out a -2 release with debian/zfsutils-linux.links:

/dev/null lib/systemd/system/zfs-load-key.service

(untested, should be equivalent to ln -s /dev/null /lib/systemd/system/zfs-load-key.service).

[ogelpre]

I was lazy to create a systemd unit file equivalent to zfs-load-key because I don't use systemd on my machines. I didn't expect to break downstream... So yes, maybe this is a downstream issue in the first place because Debian magically handles init files. Still, I think it would be a nice to provide a systemd unit file for loading ZFS keys on boot. I'll create a separte feature request.

[nabijaczleweli]

The only way to make systemd-sysv-generator skip the initscript is to make it Exist:
https://github.com/systemd/systemd/blob/8e7e4a730ba40bbc46c9d1e84207fd35781ca05a/src/sysv-generator/sysv-generator.c#L751-L762

Which we can easily do by masking it (i.e. symlinking to /dev/null). PR to be posted.

[gmelikov]

Just a note - upstream deb files won't install /etc/init.d/zfs-load-keys, at least now.