Should I use U-Boot?

[lilyball]

This project used to always use u-boot. As part of what looks like rpi 5 compatibility, a config option was added to control the use of uboot, and then more recently that option was defaulted to false. The README says nothing about this option and the sample setup does not enable u-boot, so I'm left wondering what I should do. Should I enable it because it used to always be enabled and so my rpi right now is presumably using u-boot? Should I leave it disabled because that's closer to the stock rpi configuration? Will I run into trouble migrating my system off of u-boot? What does u-boot even do for me?

[tstat]

I initially added the stock rpi method to boot from an nvme drive on a cm4, which I couldn't get working with uboot. I noticed that some other things started working properly, like the /dev/serial0 uart symlink. When the rpi 5 was released it worked without any modification with the stock method, whereas uboot lacked rpi5 support at the time. Even after uboot released rpi5 support, folks reported issues with it; so, I figured it was a better default for people just trying to get something working. FWIW, I use it as the default on my devices.

Uboot does support booting to a previous generation, and provides that interface over uart and hdmi. However, that has also been finicky for me. I don't think I ever got USB working in the uboot boot selector, so I can't select a different generation over hdmi/usb. Uart always worked fine.

I don't expect you will run into trouble migrating you system off of uboot. I have migrated a number of my rpi devices off of uboot by simply switching the option and using nixos-rebuild switch.

[lilyball]

I reverted my config back to enabling uboot, rebuilt the migration script, deleted the kernel.img file and re-ran the script, and this time it completed with no errors (and left about 5MB free on the partition). This let the rpi get past the rainbow screen. Unfortunately I didn't have a keyboard plugged in and couldn't stop it from auto-booting the last deploy, which promptly corrupted the firmware partition (not just ran out of space, I had to actually repair the partition afterwards).

However, after repairing and re-migrating the firmware and finding a working keyboard, I was finally able to boot back into the old system! So now I can deploy again with uboot enabled.

So, the takeaways:

• The firmware partition may not be large enough to switch from uboot to the linux kernel. The migration script should check to make sure the cp command actually works.
• If there isn't enough space, it needs to undo the previous changes. The kernel is migrated first, so even just detecting an error in copying the kernel means nothing has been changed yet, but it's possible there's space for the kernel but not space for other files. So perhaps the better approach is to calculate how much space is needed and check that there's enough space first. Or if it does get an error in the copy, maybe it could delete any files it's already installed and then re-run the migration script from the active system. Or even just copy the entire firmware partition into a safe location first so it can splat it back into place if something goes wrong.
• If possible, the migration script should resize the firmware partition to be large enough if it's too small. If it can't do that safely and automatically, then it would be really helpful if it could at least tell me how to do it manually.

[lilyball]

Ok, one more issue. Doing a new deploy works, but on reboot I get errors that look like

Moving Image from 0x80000 to 0x200000, end=2740000
ERROR: RD image overlaps OS image (OS=200000..2740000)

and it ends up booting my old configuration. Looking into this, it turns out that nixpkgs patches uboot to change the ramdisk address but the uboot in raspberry-pi-nix throws away that patch (it sets patches = []) and so my kernel is simply too big.

[tstat]

Oh dear, sorry I encouraged you to go down this path that turned into a pain.

The firmware partition size was increased from 30 to 128 in v0.1.0, so switching back and forth for images built on that release or later should be fine, but if your image was built before v0.1.0 then it isn't big enough to hold the kernel and firmware.

Thanks for writing up the problems you encountered and summarizing the takeaways; I'll have to make some corresponding issues later.

[lilyball]

Thinking about this again, I am curious why it’s doing the migration as a systemd unit instead of as an activation script. If it was the latter then it would be easier to handle failure, because that could fail the activation and tell the user to reactivate the current system (and deploy-rs would I believe automatically reactivate it). With that approach you could also do a preflight check to ensure the firmware partition has enough capacity before even starting activation.

It might also be simpler if this process built a derivation containing all the files that the firmware partition should have, it could then just rsync that into place (which would make it easy to delete leftover files too). And that makes it easy to measure how much space is needed.

[tstat]

Some of the motivation for switching was to have systemd ensure the firmware partition was mounted and provide the state directory to shove caching info into so that we can avoid touching the firmware partition when it isn't necessary. Activation scripts being ran on boot was also annoying: If you want to test something (e.g. putting a custom kernel in there), then the activation script runs on boot and you need to setup the test again if you want to reboot.