osbuild: use bootc install to deploy the container by jbtrystram · Pull Request #4224 · coreos/coreos-assembler

jbtrystram · 2025-07-17T15:21:55Z

Instead of deploying the container to the tree then copy all the contents to the disk image, use bootc to directly manage the installation to the target filesystems.

Right now this requires to use the image as the buildroot so this requires python (for osbuild). This is tracked in [1].

[1] bootc-dev/bootc#1410 Requires osbuild/osbuild#2149

openshift-ci · 2025-07-17T15:21:59Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

gemini-code-assist

Code Review

The pull request introduces changes to use bootc install to deploy the container, which simplifies the image build process. There are a few critical issues in the YAML manifest related to copy-paste errors that lead to incorrect configurations for the 4k image builds and missing options for loopback devices. These issues need to be addressed.

src/osbuild-manifests/coreos.osbuild.x86_64.mpp.yaml

build.sh

dustymabe · 2025-07-17T20:35:37Z

I switched the CI on this to run against rawhide (contains python) so we could actually test the change.

dustymabe · 2025-07-19T21:31:47Z

A few diffs picked up by cosa diff --metal from #4226

cosa-diff-metal.txt

We should probably profile each diff (maybe in coreos/fedora-coreos-tracker#1827) and evaluate whether it's a change we want to make or not.

dustymabe · 2025-07-19T21:43:24Z

I can't get a built qemu image to boot. I suspect probably the root= and boot= UUIDs added on the kernel command line?

jbtrystram · 2025-07-21T13:22:50Z

I can't get a built qemu image to boot. I suspect probably the root= and boot= UUIDs added on the kernel command line?

do you mind sharing more logs ? What I am getting locally is ignition failing on coreos/fedora-coreos-tracker#1250

dustymabe · 2025-07-21T13:39:27Z

Ahh. I see that too now:

[    4.726843] ignition[875]: Ignition failed: failed to create users/groups: failed to configure users: failed to create user "core": exit status 10: Cmd: "useradd" "--root" "/sysroot" "--create-home" "--password" "*" "--comment" "CoreOS Admin" "--groups" "adm,sudo,systemd-journal,wheel" "core" Stdout: "" Stderr: "useradd: cannot lock /etc/group; try again later.\n"

jbtrystram · 2025-07-21T15:31:31Z

I can't get a built qemu image to boot. I suspect probably the root= and boot= UUIDs added on the kernel command line?

looks like removing those make the boot process go further (ignition completes), and out of the initramfs but fail to mount the boot partition.

jbtrystram · 2025-07-24T15:19:53Z

Blocked on bootc-dev/bootc#1441

jbtrystram · 2025-07-28T16:52:09Z

ok this works with the following PRs :

for the bootc PR, it can be built then added into the image through overrides/rootfs. Make sure to build rawhide.

src/osbuild-manifests/coreos.osbuild.x86_64.mpp.yaml

jbtrystram · 2025-07-29T12:44:45Z

follow-up : either find a way to get the boot components inside cosa, or change the bootc code to call bootupd from the deployed root . I think the latter is preferable.
I filed bootc-dev/bootc#1455

jbtrystram · 2025-07-29T20:29:22Z

follow-up : either find a way to get the boot components inside cosa, or change the bootc code to call bootupd from the deployed root . I think the latter is preferable. I filed bootc-dev/bootc#1455

Made bootc-dev/bootc#1460
With this, we no longer require to use the container as the buildroot, cosa works, so we could do that on all streams.

jbtrystram · 2025-07-30T07:44:35Z

Alright, marking this as ready for review as all the bits are in place.
I guess i need to update the osbuild manifest or the other arches as well, but I'll do that after a review to reduce the amount of back and forth.

This will need a release of bootc.

dustymabe

Some comments.

I think there are a few things we need to iron out before we can really move forward with this:

supporting both old and new paths at the same time

Do we need to? Usually when we make a change this large we roll it out slowly, which means we have to support both ways for some time.

This PR is ignoring that fact, but TBH looking at OSBuild configs that support both would be pretty intimidating, so I'm not excited about trying to do that either. I'd be interested in @jlebon or @travier's thoughts.

We need to make sure any/every diff that exists between images generated this way and the old way are considered and acknowleged as acceptable before we'd make this change.

src/cmd-osbuild

src/osbuild-manifests/coreos.osbuild.x86_64.mpp.yaml

jbtrystram · 2025-11-29T12:33:39Z

ext.config.boot.grub2-install is fixed with bootc-dev/bootc#1816 which basically replicates what we've been doing before in osbuild

this rpm provides locales under `/usr/share/locale`. grub2 install these files inside the /boot partition using what exist on the host. Mooving to bootc install results in an empty folder because those files don't exist in the buildroot, so add them. In the previous manifest we were using a chroot before calling bootupd which made sure bootupd (and by extension grub2) would pull content from the target root. It would be nicer to have bootc install also chroot before calling bootupd, it would have the added benefit to use the bootupd (and other) binaries that are shipped on the image rather than what is in the buildroot. Again, this is a complication coming from not having python in the image so we can't use ourselves as the buildroot, which would alleviate a lot of pain. See coreos#4224 (comment) And bootc-dev/bootc#1559

jbtrystram · 2025-12-03T17:21:39Z

Rebased to drop the osbuild patch since my changes have landed in a release

this rpm provides locales under `/usr/share/locale`. grub2 install these files inside the /boot partition using what exist on the host. Mooving to bootc install results in an empty folder because those files don't exist in the buildroot, so add them. In the previous manifest we were using a chroot before calling bootupd which made sure bootupd (and by extension grub2) would pull content from the target root. It would be nicer to have bootc install also chroot before calling bootupd, it would have the added benefit to use the bootupd (and other) binaries that are shipped on the image rather than what is in the buildroot. Again, this is a complication coming from not having python in the image so we can't use ourselves as the buildroot, which would alleviate a lot of pain. See coreos#4224 (comment) And bootc-dev/bootc#1559

src/osbuild-manifests/coreos.osbuild.x86_64.bootc.mpp.yaml

supakeen · 2026-01-22T17:42:05Z

Right now this requires to use the image as the buildroot so this requires python (for osbuild). This is tracked in [1].

Just got there through another related PR and didn't read the entire backlog here so if I get this wrong let me know but:

It's possible to use a separate container as the buildroot. You can generate a manifest with:

image-builder manifest \
  --bootc-ref quay.io/fedora/fedora-bootc:43 \
  --bootc-build-ref quay.io/fedora/fedora-bootc:43 \
  --bootc-default-fs ext4 \
  qcow2

to see what that would look like (example uses Fedora 43 as it has bootc 1.12.1 which fixes some bugs) but it's basically a org.osbuild.container-deploy in the build pipeline :)

I don't know how useful that is to you since I seem to remember that CoreOS doesn't use the buildroot from osbuild itself but uses the host (which is itself a container) as the buildroot?

jbtrystram · 2026-01-23T09:52:42Z

It's possible to use a separate container as the buildroot. You can generate a manifest with:

Thanks, that's a neat tip !

I don't know how useful that is to you since I seem to remember that CoreOS doesn't use the buildroot from osbuild itself but uses the host (which is itself a container) as the buildroot?

yes, we use coreos-assembler as the buildroot. This has several implications that still need to be solved in bootc :) One of them is that you need to read prepare-root.conf to set up the ostree repo, in which you will pull the image.

I don't know how useful that is to you since I seem to remember that CoreOS doesn't use the buildroot from osbuild itself but uses the host (which is itself a container) as the buildroot?

I'll give it a try :)

this rpm provides locales under `/usr/share/locale`. grub2 install these files inside the /boot partition using what exist on the host. Mooving to bootc install results in an empty folder because those files don't exist in the buildroot, so add them. In the previous manifest we were using a chroot before calling bootupd which made sure bootupd (and by extension grub2) would pull content from the target root. It would be nicer to have bootc install also chroot before calling bootupd, it would have the added benefit to use the bootupd (and other) binaries that are shipped on the image rather than what is in the buildroot. Again, this is a complication coming from not having python in the image so we can't use ourselves as the buildroot, which would alleviate a lot of pain. See coreos#4224 (comment) And bootc-dev/bootc#1559

dustymabe

Thanks for pushing this up hill for so long JB.

Since this PR has been open for a long time it might help to reset it by adding an initial commit that just copies the coreos.osbuild.x86_64.mpp.yaml to coreos.osbuild.x86_64.bootc.mpp.yaml and then apply your changes on top in later commits.

Eventually that initial commit would do the copy for all architectures, but for now just x86_64 would suffice.

dustymabe · 2026-02-17T20:27:47Z

src/cmdlib.sh

  (cd ${workdir}; bash ${tmp_builddir}/cmd.sh |& tee /dev/virtio-ports/cosa-cmdout) || rc=\$?
 else
-  (cd ${workdir}; bash)
+  (cd ${workdir}; RUNVM_SHELL=${RUNVM_SHELL:-} bash)


I'm on the fence about this suggestion but it may be a little more obvious what this var is used for if we came up with a new var for it:

Suggested change

(cd ${workdir}; RUNVM_SHELL=${RUNVM_SHELL:-} bash)

(cd ${workdir}; RUNVM_INTERACTIVE=${RUNVM_SHELL:-} bash)

src/cmdlib.sh

dustymabe · 2026-02-17T20:48:05Z

src/cmdlib.sh

+    # cosa initialized dir and use `src/config`.
+    # We redirect the output to /dev/null to avoid the noisy `dirs` output.
+    set +u
+    pushd "${configdir:-src/config}" > /dev/null


honestly, even with your comment above I'm not sure I understand why using $configdir is needed here. This was clearly working before (only callers being in cmd-fetch and cmd-build) and cmd-osbuild doesn't call prepare_build, which is where $configdir gets set, right?

because when called from the cmd-osbuild context, the current working directory is some temporary directory IIRC, so we need to move to $configdir to find the manifest.

dustymabe · 2026-02-17T20:50:32Z

src/cmdlib.sh

-    cat <<EOF >> "${vmpreparedir}/hostfiles"
-/usr/lib/osbuild/stages/org.osbuild.coreos.live-artifacts.mono
-/usr/lib/osbuild/stages/org.osbuild.coreos.live-artifacts.mono.meta.json
-EOF


this was added in aea526e and should have been cleaned up long ago. Can you put it in a separate commit?

dustymabe · 2026-02-17T20:51:29Z

src/cmdlib.sh

+# Include arbitrary files from the host  
+#     cat <<EOF >> "${vmpreparedir}/hostfiles"
+# /usr/lib/osbuild/stages/org.osbuild.ln
+# /usr/lib/osbuild/stages/org.osbuild.ln.meta.json


Can clean this up now the ln stage is in a release of osbuild and also included in the RPM.

I left it commented so it can be a reference to how someone can use this in the future

That was left before and helped me understand what i needed to do :)

dustymabe · 2026-02-17T20:54:18Z

src/osbuild-manifests/coreos.osbuild.x86_64.bootc.mpp.yaml

+            partition:
+              mpp-format-int: '{image.layout[''boot''].partnum}'
+            target: /boot
+      - type: org.osbuild.ln


To me something like this begs for a comment. i.e. "For backwards compatibility..."

Yes. That said it's worth discussing if we should drop that entirely. It's mostly used when debugging (and in some scripts, but checking for both is easy), so the name of the file doesn't matter too much

src/vmdeps.txt

src/osbuild-manifests/coreos.osbuild.x86_64.bootc.mpp.yaml

jbtrystram · 2026-02-24T16:02:20Z

osbuild and bootc released all the changes we need for this. I am waiting a few days to get the RPMs in fedora and will update this, hopefully in it's almost final form :D

@dustymabe

The log disk usage message comming every 10 seconds is quite noisy, hide it when we are in a shell in osbuild. I aslo added a couple of helpful tips in comments given by @dustymabe to work with osbuild.

These have been released into osbuild ages ago so there is no need to manually copy it into the supermin VM

Prep work to add a knob for using bootc install in osbuild. Refactor the override logic in a helper function so we can easily add those knobs down the line.

This is some scaffholding to be able to use another osbuild manifest. This will allow to keep the legacy build path with a new path that leverage bootc install to filesystem. Right now the manifest is a copy of the existing x86 manifest to make reviewing the following comments easier.

Instead of deploying the container to the tree then copy all the contents to the disk image, use bootc to directly manage the installation to the target filesystems. We can keep the manifest duplicated until we are confident to roll this to all streams or move to image-builder. Requires: bootc-dev/bootc#1460 bootc-dev/bootc#1451 osbuild/osbuild#2149 osbuild/osbuild#2152 bootc-dev/bootc#1978 bootc-dev/bootc#1909

Properly label the filesystem mount points before calling bootc, otherwise `/sysroot` and `/boot` end up being `unlabeled_t`

Create symlinks to the aleph file created by bootc so our tests and tooling find the aleph at the expected path. Note that when moving to image-build we will likely move this to an overlay in the config, that's way easier than having to wire up a blueprint option to allow creating arbitrary symlinks.

Bootc is looking for the prepare-root config file in the buildroot environnement because the main assumption is that it's run from the target container. However, in osbuild, it's run from te buildroot, because podman inside bwrap (inside supermin in our case) causes issues. It's fine for RHCOS and SCOS where we use the target container as the buildroot but we cannot do that for FCOS because we require python in the buildroot. For now, insert a prepare-root file in the supermin VM (use as the buildroot for osbuild) until either : - bootc learn to look into the container for it [1] - we ship python in our images and can use them as buildroot. Another approach would be to layer python and the osbuild dependencies on top of our image and use that as the buildroot, but that would create room for packages drift (what was in the repos at build time?). At least using COSA it's easier to keep track of versions. [1] bootc-dev/bootc#1410

Add a bootc install config file[1] to set ostree repo options so we inject the `grub_users` config on non-default entries. [1] https://bootc-dev.github.io/bootc/man/bootc-install-config.5.html#ostree See bootc-dev/bootc#1909

By default bootc calls bootups with the `--write-uuid` option that write a stamp file with the boot partition UUID in the UEFI parition. We want to restamp those UUID at first boot, so adding this option make sure bootc does not pass that flag to bootupd. See bootc-dev/bootc#1978

openshift-ci · 2026-03-02T16:50:08Z

@jbtrystram: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/rhcos	`c10da62`	link	true	`/test rhcos`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci bot added the do-not-merge/work-in-progress label Jul 17, 2025

gemini-code-assist bot reviewed Jul 17, 2025

View reviewed changes

dustymabe mentioned this pull request Jul 17, 2025

stages/ignition: parametrize the path to boot osbuild/osbuild#2149

Merged

openshift-merge-robot added the needs-rebase label Jul 18, 2025

dustymabe mentioned this pull request Jul 19, 2025

Use bootc install to-filesystem to build our bootimages coreos/fedora-coreos-tracker#1827

Open

This comment was marked as outdated.

Sign in to view

jbtrystram mentioned this pull request Jul 23, 2025

install to-filesystem: support opt-out for the root and boot mountspec kernel arguments bootc-dev/bootc#1441

Closed

jbtrystram commented Jul 28, 2025

View reviewed changes

src/osbuild-manifests/coreos.osbuild.x86_64.mpp.yaml Outdated Show resolved Hide resolved

jbtrystram force-pushed the osbuild-bootc-install-fs branch from 59f1061 to 254f877 Compare July 29, 2025 09:47

openshift-merge-robot removed the needs-rebase label Jul 29, 2025

jbtrystram force-pushed the osbuild-bootc-install-fs branch 4 times, most recently from bb4270f to 310bd60 Compare July 30, 2025 07:38

jbtrystram marked this pull request as ready for review July 30, 2025 07:44

openshift-ci bot removed the do-not-merge/work-in-progress label Jul 30, 2025

dustymabe reviewed Jul 31, 2025

View reviewed changes

jbtrystram force-pushed the osbuild-bootc-install-fs branch from 01b2a13 to 4a85c6c Compare December 2, 2025 12:13

jbtrystram force-pushed the osbuild-bootc-install-fs branch from 4a85c6c to d85142f Compare December 3, 2025 13:05

jbtrystram force-pushed the osbuild-bootc-install-fs branch from d85142f to b2bd8ef Compare January 14, 2026 14:21

joelcapitao reviewed Jan 14, 2026

View reviewed changes

src/osbuild-manifests/coreos.osbuild.x86_64.bootc.mpp.yaml Outdated Show resolved Hide resolved

jbtrystram force-pushed the osbuild-bootc-install-fs branch from b2bd8ef to 4a81e4e Compare February 16, 2026 14:07

dustymabe reviewed Feb 17, 2026

View reviewed changes

osbuild: make disk usage log conditional and add tips

2f3c56a

The log disk usage message comming every 10 seconds is quite noisy, hide it when we are in a shell in osbuild. I aslo added a couple of helpful tips in comments given by @dustymabe to work with osbuild.

jbtrystram force-pushed the osbuild-bootc-install-fs branch 2 times, most recently from 7169ace to 4b73ee0 Compare March 2, 2026 15:26

jbtrystram added 9 commits March 2, 2026 16:39

cmdlib/supermin-prepare: drop old osbuild stages

387f7c7

These have been released into osbuild ages ago so there is no need to manually copy it into the supermin VM

cmdlib: make a helper for manifest.metadata knobs

342b758

Prep work to add a knob for using bootc install in osbuild. Refactor the override logic in a helper function so we can easily add those knobs down the line.

osbuild/bootc: properly label the mount points

6408278

Properly label the filesystem mount points before calling bootc, otherwise `/sysroot` and `/boot` end up being `unlabeled_t`

osbuild/bootc: set ostree default BLS config

27460a0

Add a bootc install config file[1] to set ostree repo options so we inject the `grub_users` config on non-default entries. [1] https://bootc-dev.github.io/bootc/man/bootc-install-config.5.html#ostree See bootc-dev/bootc#1909

jbtrystram force-pushed the osbuild-bootc-install-fs branch from 4b73ee0 to dfe14ee Compare March 2, 2026 15:52

DNM: install bootc from testing

c10da62

	(cd ${workdir}; RUNVM_SHELL=${RUNVM_SHELL:-} bash)
	(cd ${workdir}; RUNVM_INTERACTIVE=${RUNVM_SHELL:-} bash)

Conversation

jbtrystram commented Jul 17, 2025

Uh oh!

openshift-ci bot commented Jul 17, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dustymabe commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dustymabe commented Jul 19, 2025

Uh oh!

dustymabe commented Jul 19, 2025

Uh oh!

jbtrystram commented Jul 21, 2025

Uh oh!

dustymabe commented Jul 21, 2025

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

jbtrystram commented Jul 21, 2025

Uh oh!

jbtrystram commented Jul 24, 2025

Uh oh!

jbtrystram commented Jul 28, 2025

Uh oh!

Uh oh!

jbtrystram commented Jul 29, 2025

Uh oh!

jbtrystram commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jbtrystram commented Jul 30, 2025

Uh oh!

dustymabe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jbtrystram commented Nov 29, 2025

Uh oh!

jbtrystram commented Dec 3, 2025

Uh oh!

Uh oh!

supakeen commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jbtrystram commented Jan 23, 2026

Uh oh!

dustymabe left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jbtrystram Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dustymabe commented Jul 17, 2025 •

edited

Loading

jbtrystram commented Jul 29, 2025 •

edited

Loading

supakeen commented Jan 22, 2026 •

edited

Loading

jbtrystram Mar 2, 2026 •

edited

Loading