Requires running under rootful podman

[cgwalters]

A paper cut we hit today is that podman desktop defaults to rootless, and bib doesn't work with that because we need loopback. The core problem is we need to write Linux filesystems. The important Linux filesystems like XFS/ext4 in general really want to be only written by code from the Linux kernel.

Running the Linux kernel is either done by reusing the host kernel (privileged), or running a VM. But on the podman machine case we're already in a VM, which gets us into nested virt, and on Mac at least that's going to involve full emulation which usually mostly works but isn't considered a production scenario and definitely hits weird random bugs.

My inclination because we're already running this container with --privileged is just to behind the scenes reuse the fact that podman machine uses FCOS today and the core user has passwordless sudo enabled and basically reuse that to re-execute ourselves with real root privileges. Yes, this would not really be "rootless" but I personally don't care about that and I don't think users would really in general either.

[supakeen]

We're also investigating if we can do at least (some) of the filesystem work with libguestfs.

[cgwalters]

libguestfs is just a way to run VMs, so the nested virt concerns above apply.

[achilleas-k]

Right, so I was just reading about the internals and yeah libguestfs uses qemu to boot a kernel and sets up an "appliance" to talk to it. :|

[cgwalters]

The 3rd option (beyond host kernel and virt) is https://github.com/lkl/linux which is relatively new and specifically cptofs is about this problem but...I really don't think it's worth trying to scope this in right now.

[ondrejbudai]

libguestfs doesn't require KVM: https://libguestfs.org/guestfs-faq.1.html I guess it just falls back to emulation if there's no KVM. The question is how fast it is.

[achilleas-k]

Mounting directly uses FUSE and is pretty poor, but supposedly using the shell can be quite good. We can benchmark of course.

FTR, this works on rootless podman machine on macOS:
test.sh

#!/usr/bin/env bash

set -euo pipefail

fname="${1}"
truncate -s 100M "${fname}"

mkfs.ext4 "${fname}"

guestfish --rw -a "${fname}" << EOF
run
list-filesystems
mount /dev/sda /
copy-in test.sh /
cat /test.sh
quit
EOF

echo "DONE"
rm "${fname}"

Containerfile

FROM fedora:39

RUN dnf -y install libguestfs

ENV LIBGUESTFS_BACKEND=direct

COPY test.sh /test.sh
ENTRYPOINT ["/test.sh"]

[cgwalters]

Note that https://github.com/cgwalters/osbuildbootc/ doesn't use libguestfs, but it does use the underlying tool (supermin) to construct a VM root filesystem out of the container rootfs and works unprivileged today.

Honestly I think that code and approach there is much simpler than the "higher level" libguestfs approach because we have the ability to drive things at a low level.

So if we go down this path I think it'd make sense to look at merging that code.

(The other thing osbuildbootc does it defers all the heavy lifting to bootc install to-disk, which is #18 )

[cgwalters]

the underlying tool (supermin) to construct a VM root filesystem out of the container rootfs

That said what would make much more sense in a modern times is to use virtiofs as the root filesystem instead, it probably wouldn't be too hard. I just haven't dug into it.

[cgwalters]

Honestly I think that code and approach there is much simpler than the "higher level" libguestfs approach because we have the ability to drive things at a low level.

For example, forcing indirection through libguestfs's high level APIs reintroduce the same problems that osbuild creates today that motivates ostreedev/ostree#3094 - what we're doing often wants to do quite low level filesystem and block device things. libguestfs is just high level sugar for executing arbitrary code in a transient VM, and we can construct a transient VM without it.

[ondrejbudai]

I'm worried that doing the whole build under supermin might be extremely slow if KVM is not there. Whereas if we just offload the final copying part, it might be fine. I know that @achilleas-k is working on some benchmarks.

[ondrejbudai]

Also, full QEMU emulation isn't supported on RHEL. I wonder if guestfs has an exception....

[cgwalters]

libguestfs doesn't have an exception, its main use case is just targeted being used from Linux hosts.