centos 10 stream iso vm is stuck in starting

[chunfuwen]

1. Build out centos stream 10 iso using bib as below:

sudo podman run --rm -it --privileged --pull=newer --security-opt label=type:unconfined_t -v /var/lib/libvirt/images/output:/output -v /var/lib/libvirt/images/config.json:/config.json   -v /var/lib/libvirt/images/auth.json:/run/containers/0/auth.json  quay.io/centos-bootc/bootc-image-builder:latest  --type anaconda-iso --tls-verify=true  --config /config.json  quay.io/centos-bootc/centos-bootc:stream10  --chown 107:107  --rootfs ext4 '

2. use virt-install to install VM from generated iso with below command:

virt-install --name install_upstream_bib_bios_install-centos-bootc --disk path=/var/lib/libvirt/images/install_upstream_bib_bios_install-centos-bootc_bios.qcow2,bus=virtio,format=qcow2,size=12 --vcpus 3 --memory 3096 --osinfo detect=on,require=off --graphics vnc --video virtio --serial pty --wait 10 --cdrom /var/lib/libvirt/images/install_upstream_bib_bios_install-centos-bootc.iso --debug --machine q35 --noreboot

3. The VM is stuck in booting as attached screenshot
   

[mvo5]

I started looking into this in my morning and made some progress:

• with the following diff centos10-iso.diff.txt
• it boots to
  

If I manually press "text" install it will finish and reboot and our smoke test works. So it's "just" a matter now of figuring out why it claims Wayland was unable to start on your machine and how ti fix that.

[edit: the behavior is actually not consistent, on my next run no crash was reported but the install also did not continue (tmux reported the pane was death) so more investigation is needed and the wayland crash could be a red herring]

[edit2: after running it yet again there is still no wayland issue, however it does (as before) not continue and report Kickstart file /run/install/ks.cfg is missing. This is curious was we specify via the kernel comamndline: inst.ks=hd:LABEEL=Container-Installer-x86_64:/osbuild.ks so maybe another red-herring. the dract log shows that it does fetch osbuild.ks from /dev/sr0 but I cannot see where the file ends up in]

[mvo5]

Some more data, it seems anaconda is crashing when it tries to read osbuild.ks (we pass that via the kernel commandline as inst.ks=.... The error is that it cannot load the ca-certificates bundle while importing the python3-requests package (which it does not really need for getting the file locally fwiw).

The package is installed in the boot system but it seems it is not part of the initramfs which is (AIUI) when dracut-initqueue is run.

I'm a bit puzzled that only bib is affected by this, I would assume anything that passes a custom inst.ks= would see the same issue.

[edit: the relevant upstream bug is psf/requests#6749 and it seems to got introduced in https://github.com/psf/requests/pull/6667/files#diff-a4598bf8444e0feca489e60704c809c0dbfc47b90c4364216c931b497b3312d7R75]
[edit2: the above is a red-herring for this particular issue]

[mvo5]

Thanks to @thozza this should be fixed now via osbuild/osbuild#1846 and #625 so once a new osbuild with this fix is available on the image this should work.

[mvo5]

This seems indeed to be working now, see https://github.com/osbuild/bootc-image-builder/actions/runs/10617103004/job/29429091199?pr=621

test/test_build.py::test_iso_installs[quay.io/centos-bootc/centos-bootc:stream10,anaconda-iso] PASSED [ 13%]