Build ISO minikube image for ARM (aarch64)

[afbjorklund]

If we want to run any hypervisor driver on arm64, we need a new "minikube.iso" that works on the architecture.

Currently the image is full of amd64 binaries that we download from other places, so it does not build from source.

Buildroot does support the ARM architectures (armv7, arm64).

For instance the Raspberry Pi OS still uses 32-bit by default...

[afbjorklund]

Needed for #6159 and #9224

[afbjorklund]

Looked into this a bit, and there is no syslinux support on the arm64 platform - so it is unlikely to be an ISO-9660.

BR2_TARGET_ROOTFS_ISO9660=y
BR2_TARGET_SYSLINUX=y

It would work the same way though, but more like a "minikube.img". The nearest config is qemu_aarch64_virt_defconfig

BR2_TARGET_ROOTFS_EXT2=y
BR2_TARGET_ROOTFS_EXT2_4=y

So far I have used Raspberry Pi, and it has a custom bootloader and a custom config.

The other real hardware is using "Das U-Boot", but we only need something for a VM...

BR2_TARGET_ROOTFS_CPIO=y
BR2_TARGET_ROOTFS_CPIO_GZIP=y

[afbjorklund]

Here is how it starts QEMU by default:

amd64

make qemu_x86_64_defconfig world
qemu-system-x86_64 -M pc -kernel output/images/bzImage -drive file=output/images/rootfs.ext2,if=virtio,format=raw -append "rootwait root=/dev/vda console=tty1 console=ttyS0" -serial stdio -net nic,model=virtio -net user

• output/images/bzImage
• output/images/rootfs.ext2

arm64

make qemu_aarch64_virt_defconfig world
qemu-system-aarch64 -M virt -cpu cortex-a53 -nographic -smp 1 -kernel output/images/Image -append "rootwait root=/dev/vda console=ttyAMA0" -netdev user,id=eth0 -device virtio-net-device,netdev=eth0 -drive file=output/images/rootfs.ext4,if=none,format=raw,id=hd0 -device virtio-blk-device,drive=hd0

• output/images/Image
• output/images/rootfs.ext4

---

So what is needed is a nice way to bundle the kernel (Image) and the rootfs (initrd.cpio.gz) into one disk image.

The buildroot "genimage" script could help with this, perhaps. Then we just need some simple bootloader for it...

[afbjorklund]

Using grub2 with efi seems to be the simplest, since it has built-in support (unlike syslinux or gummiboot).

boot/grub2/readme.txt

amd64

board/pc
configs/pc_x86_64_efi_defconfig

qemu-system-x86_64 \
	-M pc \
	-bios </path/to/OVMF_CODE.fd> \
	-drive file=output/images/disk.img,if=virtio,format=raw \
	-net nic,model=virtio \
	-net user

ovmf: /usr/share/OVMF/OVMF_CODE.fd

arm64

board/aarch64-efi/
configs/aarch64_efi_defconfig

qemu-system-aarch64 \
	-M virt \
	-cpu cortex-a57 \
	-m 512 \
	-nographic \
	-bios </path/to/QEMU_EFI.fd> \
	-drive file=output/images/disk.img,if=none,format=raw,id=hd0 \
	-device virtio-blk-device,drive=hd0 \
	-netdev user,id=eth0 \
	-device virtio-net-device,netdev=eth0

qemu-efi-aarch64: /usr/share/qemu-efi-aarch64/QEMU_EFI.fd

For now we will continue without a root partition, since minikube assumes that it will be running from tmpfs.

[afbjorklund]

Here are the reference board sizes:

4.9M	output/images/bzImage
88M	output/images/rootfs.ext2
93M	output/images/disk-amd64.img

20M	output/images/Image
42M	output/images/rootfs.ext4
63M	output/images/disk-arm64.img

Will put them up as a separate project.

See: https://github.com/afbjorklund/minimal-buildroot

---

PS. They are both using ext4, just have different filenames when generating (and then a symlink between them)
Comes in handy when having both in the same file structure, just as with the compressed/uncompressed kernel...

For the real minikube OS, we have the files in an initrd so it will go on a "boot" partition instead of the "root" partition.
Then the init will create a tmpfs (needed for containers), and copy everything from the rootfs and do a switch_root.

Device                        Start    End Sectors  Size Type
output/images/disk-amd64.img1    64  32831   32768   16M EFI System
output/images/disk-amd64.img2 32832 278591  245760  120M Linux root (x86)

Device                        Boot Start    End Sectors  Size Id Type
output/images/disk-arm64.img1          1  65536   65536   32M ef EFI (FAT-12/16/32)
output/images/disk-arm64.img2      65537 475136  409600  200M 83 Linux

[bluestealth]

@afbjorklund It is possible to create an iso image to boot on arm64 and (hybrid bios/uefi) amd64 if you wanted to maintain backwards compatibility, and not require UEFI firmware on Intel. I have this working in a branch.
I don't have any access to arm64 boards that support kvm, so I had to test this in qemu so far. https://github.com/kubernetes/minikube/compare/master...bluestealth:arm64?expand=1

[afbjorklund]

@bluestealth : nice, seems like you have already gotten started on it. I saw that debian had a livecd also for arm64, so it should be possible. that will probably make it easier to interface with libmachine, but it seems like your kvm2 driver needed some hacks in it...
will take a look at your branch later, so far it seems that replacing isolinux with grub is pretty much all that is needed for booting. I'm just using Raspberry Pi at the moment, but trying to get something different together for the arm64 hackathon later this week.

[bluestealth]

@afbjorklund Yes, I used the debian documentation to get it working, which is really good. Some of my KVM hacks were to get it working with Qemu, I have even more hacks in another branch to allow minikube to work as a client to foreign architectures, which is kind a mess.