QUEST: Shutdown not working using PXE boot

[Joulinar]

Hi,

I'm still playing around trying to use my test systems using PXE boot. Booting themselves is not an issue and my RPi4B is starting using RootFS located on a NFS share 😃

root@DietPi4:~# df -h
Filesystem                                Size  Used Avail Use% Mounted on
192.168.0.10:/volume1/NFS/pxe/rpi4_armv6  2.7T  806G  1.9T  31% /
devtmpfs                                  3.9G     0  3.9G   0% /dev
tmpfs                                     3.9G     0  3.9G   0% /dev/shm
tmpfs                                     1.6G  8.6M  1.6G   1% /run
tmpfs                                     5.0M     0  5.0M   0% /run/lock
tmpfs                                     3.9G     0  3.9G   0% /tmp
tmpfs                                      50M  8.0K   50M   1% /var/log

Currently my challenge is to showdown/reboot the system. It's going to hang on bringing down the interfaces. I guess issue is somewhere in the order of actions. NFS share seems to disconnect before all scrips finished. Maybe you know what status the system is hanging on and if there is a way to shift thinks around and to have NFS share disconnected last? Probably a chicken/ egg challenge 🤔

[MichaIng]

Probably it's the other way round. ifup@eth0 does ifdown eth0 when stopping, hence deconfigures the network interface and would hence cut the NFS connection. Probably this is prevented somehow when this NFS mount serves the root filesystem.

Two ideas:

• We currently force ifup@.service to be a oneshot service type (instead of default "simple"), so that it needs to finish (instead of only just start) before other targets and services are reached/start which order themselves after it. This is why network-online.target now really means that the network is online, at least that the interfaces have been fully configured, before it was basically meaningless, only the loopback interface was up for sure. But when ifup@eth0 is stopped now, the effect is probably the same: The shutdown sequence does not continue until ifdown eth0 finished, which hangs. To test this:

  echo -e '[Service]\nType=simple' > /etc/systemd/system/ifup@.service.d/override.conf
  systemctl daemon-reload

• Since the above breaks the positive intention of using a oneshot interface, another idea is to simply stop the interface deconfiguration at shutdown. I'm not aware of any important effect of this when the system is powered of anyway. At least with PXE it makes sense that the NFS mount is active as long as possible for all important shutdown steps to finish:

  echo -e '[Service]\nExecStop=' > /etc/systemd/system/ifup@.service.d/override.conf
  systemctl daemon-reload

[Joulinar]

ok ifup seems to be fixed by setting ExecStop= to an empty value. However it's gonna stuck now on Raise network interfaces 🙃

[MichaIng]

So we gonna fix it the same way then:

mkdir -p /etc/systemd/system/networking.service.d
echo -e '[Service]\nExecStop=' > /etc/systemd/system/networking.service.d/override.conf
systemctl daemon-reload

Let's see, here it is actually strange since it does/should not deconfigure any interface:

ifdown -a --read-environment --exclude=lo

With the -a flag, all interfaces defined with auto <iface> in /etc/network/interfaces are affected, but we use allow-hotplug <iface>, which are deconfigured by ifup@<iface>.service only. And the loopback device is explicitly excluded here. However, let's see if it still helps.

[Joulinar]

Ok perfect that's working as well. System is now stopping and rebooting fine. This way I'm now able to switch images/installations on my test RPi between ARMv6/7/8 without swapping SD cards and/or doing restores.

[MichaIng]

Great. Good to know that it is not only an issue with our ifup@.service change then but on regular Debian ifupdown as well. I'm wondering how it is intended, if this is known or whether there is a superior way to solve this for PXE boot, e.g. some ifupdown hook package, a different network stack or so 🤔. Probably some web search reveals more info.

[Joulinar]

I guess I will flash a RPi OS to have it stored on NFS as well. Let's see how it behaves. At least on all the guides, even the Raspberry own one, I haven't seen such steps. Description was always in how to boot and not on how to shutdown 😂

[MichaIng]

Good idea. Raspberry Pi OS uses dhcpcd by default to configure the network, but networking.service is still present. If it works, then there must be a hook or so to change how the letter behaves.

[Joulinar]

ok i just copied the partitions from RPi OS image file to my NFS share and was able to boot RPi OS arm64 without issues. As well reboot is working without issues.

[MichaIng]

Can you check the following:

systemctl status networking
ls -l /etc/network/*.d/*

[Joulinar]

root@raspberrypi:~# systemctl status networking
● networking.service - Raise network interfaces
   Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
   Active: active (exited) since Mon 2021-10-18 00:08:33 BST; 20min ago
     Docs: man:interfaces(5)
  Process: 179 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=0/SUCCESS)
 Main PID: 179 (code=exited, status=0/SUCCESS)

Oct 18 00:08:31 raspberrypi systemd[1]: Starting Raise network interfaces...
Oct 18 00:08:33 raspberrypi systemd[1]: Started Raise network interfaces.

root@raspberrypi:~# ls -l /etc/network/*.d/*
-rwxr-xr-x 1 root root  301 Jul 25  2011 /etc/network/if-down.d/resolvconf
lrwxrwxrwx 1 root root   32 Apr 16  2021 /etc/network/if-down.d/wpasupplicant -> ../../wpa_supplicant/ifupdown.sh
-rwxr-xr-x 1 root root 1409 Jun  5  2018 /etc/network/if-post-down.d/wireless-tools
lrwxrwxrwx 1 root root   32 Apr 16  2021 /etc/network/if-post-down.d/wpasupplicant -> ../../wpa_supplicant/ifupdown.sh
-rwxr-xr-x 1 root root  344 Jun 30  2016 /etc/network/if-pre-up.d/ethtool
-rwxr-xr-x 1 root root 4191 Sep 15  2018 /etc/network/if-pre-up.d/wireless-tools
lrwxrwxrwx 1 root root   32 Apr 16  2021 /etc/network/if-pre-up.d/wpasupplicant -> ../../wpa_supplicant/ifupdown.sh
-rwxr-xr-x 1 root root  703 Jul 25  2011 /etc/network/if-up.d/000resolvconf
-rwxr-xr-x 1 root root 1685 Jun 30  2016 /etc/network/if-up.d/ethtool
lrwxrwxrwx 1 root root   32 Apr 16  2021 /etc/network/if-up.d/wpasupplicant -> ../../wpa_supplicant/ifupdown.sh
root@raspberrypi:~#

[MichaIng]

Strange, I cannot see any additional ifupdown configuration, so networking.service should behave exactly the same on both systems.

Probably on RPi OS there is nothing actually depending on the root filesystem when network gets deconfigured. On DietPi, RAMlog and PreBoot run Before=network-pre.target, hence on shutdown RAMlog's stop job, to store /var/log structure to disk runs after networking.service is stopped. Also it depends on the rootfs mount for that reason.

Probably we can further relax this: It doesn't hurt when network is up before RAMlog or PreBoot starts. We kept that mostly so that DietPi-Boot does not start to wait for network before even the loopback device is configured, but that has been removed now. Also DietPi-FirstBoot is an own service (those steps were in PreBoot and DietPi-Boot partly AFAIK, some time ago) which can have the Before=network-pre.target alone to only order the very first boot, where we want it: /etc/network/interfaces needs to be configured based on dietpi.txt selection before the networking/ifup services configure interfaces based on it.

I'll do a PR to remove this obsolete ordering.

[Joulinar]

Well it was not my intention to trigger some redesign 🤣

After some test I'm not really satisfied with the performance of the system once it has booted using PXE. It's quite slow on w/r operation. Especially on smaller files it's down to KB/s. Something I need to think on. Maybe I will try a different NFS server to have some comparison 🤔

[MichaIng]

The cleanup of boot service ordering after the recent changes just makes sense anyway, not only to potentially enable PXE shutdown 😄.