composefs-rs integration in bootc

[travier]

The composefs-rs project is a Rust implementation of composefs that is capable of generating composefs images from container images.

We should integrate it in bootc as an alternative to the ostree backend. This would help make progress on phasing out ostree, UKI support and unified storage:

• UKI/systemd-boot tracker #806
• Unified storage #20

To be able to do that, we need to make bootc capable of handling both repository formats and have it handle the transition from ostree to pure composefs.

A potential layout for this is discussed in containers/composefs-rs#38.

Here are suggested steps for creating a first proof of concept implementation:

• Add an option to bootc switch
  • bootc switch --composefs quay.io://foo:bar
  • bootc will import the container image using the composefs-rs library in a dedicated composefs repo
  • bootc will set up the repo as needed
• bootc will create a new "deployment" for this image
  • Do the three way merge for /etc, comparing previous image, new image, current changes
  • Or use overlayfs instead to do that for /etc
  • Setup /var so that it's shared with ostree deployments
• bootc will setup the new deployment for the next boot
  • UKI case:
    • GRUB: Generate GRUB config snippet to boot the UKIs in order
    • systemd-boot: Install the UKI in /boot/efi/EFI/Linux (order handling to be confirmed)
  • Non-UKI case:
    • GRUB: Install the kernel & initrd in /boot and setup the BLS config
    • systemd-boot: Install the kernel & initrd in /boot/efi and setup the BLS config

[cgwalters]

I added the area/composefs label to this, there's a few related things there, but especially #935 will step towards this from the other direction and I'd like to try to land that relatively early on.

progress on phasing out ostree

And yes while this is definitely a longer term plan there's A LOT of stuff there; for example, we document configuring ostree-prepare-root today, and so we'll need to continue honoring configs for /usr/lib/ostree/prepare-root.conf into the forseeable future.

https://bootc-dev.github.io/bootc/filesystem.html#enabling-state-overlays is another one.

[cgwalters]

BTW #935 is updated and working now at least.

#935 merged. Still TODO:

• Support for upgrades (we should check post fetching, before reboot, check if new image enables fsverity, if so do it before rebooting)
• bootc internals enable-fsverity or so

[cgwalters]

bootc switch --composefs quay.io://foo:bar

Hmm, that raises the bar quite high for initial experimentation here. I think it'd probably be a lot easier to start with bootc install actually because trying to make it part of switch implies that how etc and var work is fully compatible.

[cgwalters]

As far as how it's configured, we can definitely do something like bootc install --experimental-composefs-backend or so, but actually the other option is to just default to this for the "bare UKI" in the image (no bootloader); which rolls a bit into the question of configuring the bootloader. Maybe easiest for now is to automatically do this for images that have a UKI and don't include bootupd.

[cgwalters]

Sub-thread on the build side, we need to bikeshed how we expose this part:

containers/composefs-rs@dfdb1b1#diff-5731dcec97e8f5f570af734466fee700490739eda2a59b738e4417c7ee31c9ccR32

[travier]

As far as how it's configured, we can definitely do something like bootc install --experimental-composefs-backend

👍🏻

but actually the other option is to just default to this for the "bare UKI" in the image (no bootloader); which rolls a bit into the question of configuring the bootloader. Maybe easiest for now is to automatically do this for images that have a UKI and don't include bootupd.

I don't think we should have images without bootupd. Ideally we would have a single image with the UKI and degrade it into a kernel + initramfs + command line setup on demand in bootc/composefs-rs when requested for setups that don't support UKI or fs-verity. It's also likely that we will ship both the kernel & UKI in the image at the beginning.

[cgwalters]

Chatting with Pragyan, one option that may be simplest is to focus on the direct-UKI case w/o a bootloader. That would help prove out the installation path without involving the complexities of bootloaders. But I definitely see that most use cases would want one (especially one with boot counting).

[travier]

I don't think the use case without a bootloader is simpler or smaller in terms of code. No bootloader means either doing EFI variable management or a special ESP file handling logic. Neither of those will be reliable or easy to test compared to the current approach.

[cgwalters]

OK sorry, I don't have a really strong opinion and am fine with either path. Whatever is easier to demonstrate progress if again fine by me.

[cgwalters]

One tangential thing, I was rereading https://0pointer.net/blog/fitting-everything-together.html and thinking about the architecture here, and it does seem clear to me that in the general case we need to also encourage the dm-crypt+(dm-integrity|btrfs-with-full-checksum) for the stateful root. This definitely has implications on how we think of the installation path (which is necessarily distinct from the "dd or boot raw disk image" flow).