Arch Linux template can't be built on 4.1 seemingly due to chroot failure to create etc

[GaelicGabe]

How to file a helpful issue

Qubes OS release

4.1

Brief summary

I am having issues building the arch linux template on the 4.1 release. I'm having various issues on step 5 in this guide.

Steps to reproduce

Follow the guide down to step 5 and run make vmm-xen-vm or make qubes-vm

Expected behavior

It is expected to complete the compilation.

Actual behavior

make vmm-xen-vm errors with the output:

Currently installed dependencies:
createrepo_c-0.17.5-1.fc34.x86_64
debootstrap-1.0.126-1.nmu1.fc34.noarch
devscripts-2.21.3-1.fc34.x86_64
dpkg-dev-1.21.1-1.fc34.noarch
git-2.34.1-1.fc34.x86_64
perl-Digest-MD5-2.58-2.fc34.x86_64
perl-Digest-SHA-6.02-459.fc34.x86_64
python3-pyyaml-5.4.1-2.fc34.x86_64
python3-sh-1.14.1-2.fc34.noarch
rpm-build-4.16.1.3-1.fc34.x86_64
rpmdevtools-9.6-1.fc34.noarch
systemd-container-248.10-1.fc34.x86_64
wget-1.21.3-1.fc34.x86_64
--> Archlinux dist-prepare-chroot (makefile):
-> Building vmm-xen (archlinux) for archlinux vm (logfile: build-logs/vmm-xen-vm-archlinux.log)
--> build failed!
--> Archlinux dist-build-dep (makefile)
  --> Generate locales...
tee: /home/user/qubes-builder/chroot-vm-archlinux/etc/locale.gen: No such file or directory
en_US.UTF-8 UTF-8
make[2]: *** [/home/user/qubes-builder/qubes-src/builder-archlinux/Makefile.archlinux:101: dist-build-dep] Error 1
make[1]: *** [Makefile.generic:197: packages] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:265: vmm-xen-vm] Error 1

The log contains:

--> Archlinux dist-build-dep (makefile)
  --> Generate locales...
tee: /home/user/qubes-builder/chroot-vm-archlinux/etc/locale.gen: No such file or directory
en_US.UTF-8 UTF-8
make[2]: *** [/home/user/qubes-builder/qubes-src/builder-archlinux/Makefile.archlinux:101: dist-build-dep] Error 1

Running make qubes-vm errors with the output:

Currently installed dependencies:
createrepo_c-0.17.5-1.fc34.x86_64
debootstrap-1.0.126-1.nmu1.fc34.noarch
devscripts-2.21.3-1.fc34.x86_64
dpkg-dev-1.21.1-1.fc34.noarch
git-2.34.1-1.fc34.x86_64
perl-Digest-MD5-2.58-2.fc34.x86_64
perl-Digest-SHA-6.02-459.fc34.x86_64
python3-pyyaml-5.4.1-2.fc34.x86_64
python3-sh-1.14.1-2.fc34.noarch
rpm-build-4.16.1.3-1.fc34.x86_64
rpmdevtools-9.6-1.fc34.noarch
systemd-container-248.10-1.fc34.x86_64
wget-1.21.3-1.fc34.x86_64
mknod: ./test-dev-null: File exists
mknod: ./test-dev-null: File exists
mknod: ./test-dev-null: File exists
--> Archlinux dist-prepare-chroot (makefile):
*******************************************************************************
***                               ERROR                                      ***
*** Cannot create chroot because the current filesystem is mounted as nodev. ***
*** Build Qubes on a different filesystem, or run 'make remount' to remount  ***
*** /home with dev option.
***                                                                          ***
*******************************************************************************
make[1]: *** [Makefile.generic:159: generic-prepare-chroot] Error 1
make: *** [Makefile:265: core-qubesdb-vm] Error 1
make: *** Waiting for unfinished jobs....
*******************************************************************************
***                               ERROR                                      ***
*** Cannot create chroot because the current filesystem is mounted as nodev. ***
*** Build Qubes on a different filesystem, or run 'make remount' to remount  ***
*** /home with dev option.
***                                                                          ***
*******************************************************************************
*******************************************************************************
***                               ERROR                                      ***
*** Cannot create chroot because the current filesystem is mounted as nodev. ***
*** Build Qubes on a different filesystem, or run 'make remount' to remount  ***
*** /home with dev option.
***                                                                          ***
*******************************************************************************
make[1]: *** [Makefile.generic:159: generic-prepare-chroot] Error 1
--> Archlinux dist-prepare-chroot (makefile):
make: *** [Makefile:265: core-qrexec-vm] Error 1
make[1]: *** [Makefile.generic:159: generic-prepare-chroot] Error 1
--> Archlinux dist-prepare-chroot (makefile):
make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:265: linux-utils-vm] Error 1
-> Building core-vchan-xen (archlinux) for archlinux vm (logfile: build-logs/core-vchan-xen-vm-archlinux.log)
-> Building vmm-xen (archlinux) for archlinux vm (logfile: build-logs/vmm-xen-vm-archlinux.log)
--> build failed!
--> build failed!
--> Archlinux dist-build-dep (makefile)
  --> Generate locales...
tee: /home/user/qubes-builder/chroot-vm-archlinux/etc/locale.gen: No such file or directory
en_US.UTF-8 UTF-8
make[2]: *** [/home/user/qubes-builder/qubes-src/builder-archlinux/Makefile.archlinux:101: dist-build-dep] Error 1
make[1]: *** [Makefile.generic:197: packages] Error 1
make[1]: *** [Makefile.generic:197: packages] Error 1
make: *** [Makefile:265: core-vchan-xen-vm] Error 1
make: *** [Makefile:265: vmm-xen-vm] Error 1

The log contains:

--> Archlinux dist-build-dep (makefile)
  --> Generate locales...
tee: /home/user/qubes-builder/chroot-vm-archlinux/etc/locale.gen: No such file or directory
en_US.UTF-8 UTF-8
make[2]: *** [/home/user/qubes-builder/qubes-src/builder-archlinux/Makefile.archlinux:101: dist-build-dep] Error 1

The directory chroot-vm-archlinux only contains home, tmp and var.

etc does not seem to have been created correctly.

[icequbes1]

Have you run make remount as instructed, before building VM packages?

Try removing the chroot (sudo rm -rf chroot-vm-archlinux) and retrying all of Step 5 in the guide.

[GaelicGabe]

Have you run make remount as instructed, before building VM packages?

Try removing the chroot (sudo rm -rf chroot-vm-archlinux) and retrying all of Step 5 in the guide.

I have run make remount once and also several times (once before install-deps and once right before make qubes-vm).
Neither of the time it changes the outcome.
I've also removed chroot-vm-archlinux and retried, but with the same result.

[unman]

I regularly rebuild the archlinux packages and templates on 4.1 using both a Fedora-34 and debian-11 qube for building. I don't think (I haven't got your original post to hand) that you say what build machine you are using, or which of the routes outlined on the somewhat outdated guide you have followed. It would be helpful if you could do this. Use a stock template cloned, and then install the dependencies as needed. Use setup to select the template to build. That works. Try it with and without `USE_DIST_BUILD_TOOLS=1` in the builder.conf file.

[JamesClarke7283]

It does not work for me. even with USE_DIST_BUILD_TOOLS=1 set.

I am using fedora-34 to build, and building the standard archlinux template.

Also that outdated guide is the official one, if its the guide thats the problem, could you upload an updated version?

Log of errors is here:
error.log

[lubellier]

I see a lot of "Permission denied", could you check that sudo works?
See the qubes-builder : "One additional useful requirement is that ‘sudo root’ must work without any prompt, which is default on most distros (e.g. ‘sudo bash’ brings you the root shell without asking for any password). This is important as the builder needs to switch to root and then back to user several times during the build process."

Also, please give us the git describe output.

[JamesClarke7283]

I see a lot of "Permission denied", could you check that sudo works? See the qubes-builder : "One additional useful requirement is that ‘sudo root’ must work without any prompt, which is default on most distros (e.g. ‘sudo bash’ brings you the root shell without asking for any password). This is important as the builder needs to switch to root and then back to user several times during the build process."

Also, please give us the git describe output.

In the VM i am using to build, sudo itself is working without a prompt.

Here is proof:

[user@build-archlinux qubes-builder]$ sudo whoami
root

Sudo does not work within the chroot, before ive had to install and configure sudo manually(edit sudoers, with nopasswd added), the build still failed that time.

About the git describe output, i assume you mean in step 3.

[user@build-archlinux qubes-builder]$ git tag -v $(git describe)
object 1fb9d931e2644a4e8819f486db9c08140bd8e1c7
type commit
tag mm_1fb9d931
tagger Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> 1646914540 +0100

Tag for commit 1fb9d931e2644a4e8819f486db9c08140bd8e1c7
gpg: Signature made Thu Mar 10 12:15:41 2022 GMT
gpg:                using RSA key 0064428F455451B3EBE78A7F063938BA42CFA724
gpg: Good signature from "Marek Marczykowski-Górecki (Qubes OS signing key) <marmarek@invisiblethingslab.com>" [full]

And here is the output of git describe on its own:

[user@build-archlinux qubes-builder]$ git describe
mm_1fb9d931

[JamesClarke7283]

Just to give a bigger picture, here are logs for all steps in step 5. i start btw with a fresh appVM so its not polluted.

Part 1 Command:

make remount

Log here for command output:

step5-1.log

Part 2 Command:

make install-deps

Log here for command output:

step5-2.log

Part 3 Command:

make get-sources

Log here for command output:

step5-3.log

Part 4 Command:

make qubes-vm

Log here for command output:

step5-4.log

Right after running those commands i run Part 4 command again make qubes-vm
Here is the output of that:
step5-4-again.log

[gio256]

Try without the /etc/profile.d/99-makeflags.sh file and using make qubes-vm. There may be some implicit dependencies between targets.

[unman]

On Mon, Apr 04, 2022 at 05:35:40AM -0700, JamesClarke7283 wrote: It does not work for me. even with USE_DIST_BUILD_TOOLS=1 set. Also that outdated guide is the official one, if its the guide thats the problem, could you upload an updated version?

That guide is *not* official. It is in the External documentation section, marked as "unoffial, third-party documentation"

[JamesClarke7283]

Try without the /etc/profile.d/99-makeflags.sh file and using make qubes-vm. There may be some implicit dependencies between targets.

Ah it now works, it builds and i have installed it, after commenting out those lines in that file.
Thanks.

I also got that xenstore-read: xs_open: permission denied error, so i did the command to fix that:
Remedy Is Here

[unman]

If I read this correctly, the 5-4.log shows an error asking for 'make remount', which seems at odds with your step 1. Just to be sure I tested with a clean build machine, NOT following those instructions, but following my nose (and the official docs). Git clone qubes-builder Run setup and select "archlinux-minimal", and follow prompts. make get-sources make qubes-vm Errors - follow advice to 'make remount' make qubes-vm works for me. If it matters, that's a stock Fedora-34 template, and the builder has memory 800, maxmem 8000, 4vcpus. Can you test that?

[JamesClarke7283]

If I read this correctly, the 5-4.log shows an error asking for 'make remount', which seems at odds with your step 1. Just to be sure I tested with a clean build machine, NOT following those instructions, but following my nose (and the official docs). Git clone qubes-builder Run setup and select "archlinux-minimal", and follow prompts. make get-sources make qubes-vm Errors - follow advice to 'make remount' make qubes-vm works for me. If it matters, that's a stock Fedora-34 template, and the builder has memory 800, maxmem 8000, 4vcpus. Can you test that?

Do you have this file: /etc/profile.d/99-makeflags.sh

If you don't, thats probably why it works.

As commented above, mine now builds after i commented out lines from that file.

[GaelicGabe]

Try without the /etc/profile.d/99-makeflags.sh file and using make qubes-vm. There may be some implicit dependencies between targets.

I've followed this advice by not creating the 99-makeflags.sh file in the build template and I've also found a bug in the make get-sources script where it uses outdated git links that result in: The unauthenticated git protocol on port 9418 is no longer supported. I used this SO post to solve it by running git config --global url."https://github.com/".insteadOf git://github.com/ before I ran make get-sources

It all works up until I run make template which for some reason does not produce the install-template.sh script used to install the arch template.
I'm wondering if I can go ahead and transfer the arch template .rpm file and install it manually in dom0 via dnf or if I need the install-template.sh script to do this.

From what I can tell someone discussed this as being a known bug but I can't see it mentioned in the PR.

[gio256]

It's likely you ran into an error in the template build process, and that's why install-templates.sh wasn't generated. Look through your qubes-builder/build-logs/template-archlinux.log file for indications of errors.

In particular, you can grep for xenstore-read: xs_open: Permission denied and see the fix for this issue here.

[GaelicGabe]

It's likely you ran into an error in the template build process, and that's why install-templates.sh wasn't generated. Look through your qubes-builder/build-logs/template-archlinux.log file for indications of errors.

In particular, you can grep for xenstore-read: xs_open: Permission denied and see the fix for this issue here.

This worked, I had to make sure to run the sudo chgrp qubes /dev/xen/* several times during make template so I recommend people use something like this in a parallel terminal.
while [ 0 ]; do echo "sudoing"; sudo chgrp qubes /dev/xen/*; sleep 5; done

However I'm running into the issue of dom0 saying sudo yum: command not found when I'm trying to run the install-templates.sh script. Is this normal and should I just install yum in dom0 or is this an outdated part?

[gio256]

See #7305. I believe yum used to be installed in dom0 by default but no longer is, so you'll have to install it yourself.

[GaelicGabe]

See #7305. I believe yum used to be installed in dom0 by default but no longer is, so you'll have to install it yourself.

Alright, this seems to be solved for me. I have successfully built the minimal archlinux template and it seems to be able to install things via pacman. Thank you for your help and thanks to everyone else.
Unless anyone else wants me to keep the issue open I will close it later tonight.