ERROR: qemu-system-aarch64: Addressing limited to 32 bits

[mprimeaux]

What Happened?

When starting minikube v1.26.0-beta.1 with the following configuration, it fails to start.

- cpus: 8
- driver: qemu2
- memory: 32768

Log file is attached though the error indicates a 32-bit memory address limitation. My host configuration is...

• OS: macOS Monterey (12.4)
• CPU: Apple M1 Ultra
• Memory: 128 GB

NOTE: I wasn't able to select qemu2 as the driver below on this form.

Attach the log file

logs.txt

Operating System

macOS (Default)

Driver

No response

[afbjorklund]

Currently it is disabled in the driver (highmem=off) due to some bugs, which might be fixed in macOS 12.4

https://patchwork.kernel.org/project/qemu-devel/patch/20201126215017.41156-9-agraf@csgraf.de/#23800615

https://gitlab.com/qemu-project/qemu/-/issues/903#note_911000975 says that it should have been fixed in beta

But no idea about M1 Ultra and 32 GiB, the issues were for Pro/Max and 8 GiB. Currently, machine is hard-coded.

[afbjorklund]

It should check the macOS version, and only apply the workaround if < 12.4

                qemuMachine = "virt"
                qemuCPU = "cortex-a72"
                // highmem=off needed, see https://patchwork.kernel.org/project/qemu-devel/patch/20201126215017.41156-9-agraf@csgraf.de/#23800615 for details
                if runtime.GOOS == "darwin" {
                        qemuMachine = "virt,highmem=off"
                }

Also the virt basic machine, should be separated out from the other options.

LIke so: qemuMachine += ",highmem=off"

[mprimeaux]

@afbjorklund As always, thanks for your guidance here and the above links. I'll review #14146 to see where I can help.

[sharifelgamal]

So I don't think it has anything to do with the version of macOS, we have an M1 machine with Monterey and 16 GB of RAM and it still fails. I don't really have a good way to test with machines with more RAM than that.

[mprimeaux]

@sharifelgamal I have several M1 machines with greater than 16 GB of RAM and am happy to help if you point me in the right direction with respect to branch to pull down, etc.

[sharifelgamal]

if you're comfortable building your own minikube binary, then all you need to do is comment out https://github.com/kubernetes/minikube/blob/master/pkg/minikube/registry/drvs/qemu2/qemu2.go#L125 and build minikube. Otherwise, I can supply an arm64 minikube binary for you to test with.

[mprimeaux]

I’m very comfortable building my own minukube binary. Let me give it a go tomorrow first thing. Appreciate the link.

[mprimeaux]

Looks like there's still a failure.

😄  minikube v1.26.0-beta.1 on Darwin 12.4 (arm64)
✨  Using the qemu2 (experimental) driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating qemu2 VM (CPUs=8, Memory=32768MB, Disk=20000MB) ...- OUTPUT: 
ERROR: qemu-system-aarch64: VCPU supports less PA bits (36) than requested by the memory map (40)
                                                                                                                                                      
🔥  Deleting "minikube" in qemu2 ...
🤦  StartHost failed, but will try again: creating host: create: creating: exit status 1

I'll dig in a bit more.

I also see more supported accelerators supported by the QEMU binary than reported here.

❯ qemu-system-aarch64 -accel help                                                                                                                    ─╯
Accelerators supported in QEMU binary:
hvf
tcg

❯ minikube config view                                                                                                                                     ─╯
- cpus: 8
- driver: qemu2
- memory: 32768

[mprimeaux]

Setting himem=off and qemuCPU = "host" from cortex-a72 did allow minikube to start.

❯ out/minikube start                                                                                                                                 ─╯
😄  minikube v1.26.0-beta.1 on Darwin 12.4 (arm64)
✨  Using the qemu2 (experimental) driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating qemu2 VM (CPUs=8, Memory=32768MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.23.6 on Docker 20.10.16 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

However, it seems setting CPUType: "host" is not advisable, unless I'm misinterpreting the context.

UPDATE: Actually, if we aren't using the "live migration" feature then setting qemuCPU = "host" seems fine. @sharifelgamal and @afbjorklund, what are your thoughts? Apologies that I don't have the historical context yet to make an informed decision.

[afbjorklund]

The "tcg" is the non-accelerated fallback
The "cortex-a72" is from the raspberry pi 4

[mprimeaux]

Is there a reason that for runtime.GOOS == "darwin" we couldn't use "host" for CPUType?

[sharifelgamal]

Looks like host is a reasonable choice for CPU for darwin. I think we can change that from cortex-a72 and only set highmem=off when available RAM is under 32 GB.

[mprimeaux]

@sharifelgamal Wouldn't we need highmem=off when available RAM is under 4 GB? Here's an excerpt from the QEMU docs:

Some 32-bit guests on the "virt" machine may need "-machine highmem=off" if they are unable to support LPAE and are confused by the PCI ECAM now being above the 4GB mark in the address space.

[sharifelgamal]

@mprimeaux Ideally yes, but I just tried it with 8 GB of memory and it was still failing. If you want to try with 16GB of RAM and see if that works I'd be happy to lower the threshold.

[mprimeaux]

Ideally yes, but I just tried it with 8 GB of memory and it was still failing. If you want to try with 16GB of RAM and see if that works I'd be happy to lower the threshold.

Yes, likewise, those values failed for me. I'm trying 4 GB now.

[mprimeaux]

Setting highmem=off failed for me at 3.5 GiB and 4 GiB but passed for me at 3GiB. So it seems that 3 GiB is the limit, at least on macOS.

In short, all minikube config set memory values greater than 3072 fail for me when highmem=off.

[mprimeaux]

I was able to go up to 63 GiB of memory with minikube using the qemu2 driver but not 64 GiB.

❯ out/minikube start                                                                                                                                 ─╯
😄  minikube v1.26.0-beta.1 on Darwin 12.4 (arm64)
✨  Using the qemu2 (experimental) driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating qemu2 VM (CPUs=8, Memory=65536MB, Disk=20000MB) ...- OUTPUT: 
ERROR: qemu-system-aarch64: Addressing limited to 36 bits, but memory exceeds it by 1073741824 bytes

So, it seems there's a lower memory threshold of 3072 MB for highmem=on and a maximum memory limit of 64,512 MB for QEMU.

[sharifelgamal]

ok, this is a version of qemu problem actually. qemu 6.2.0 (which is currently the default qemu version in brew) requires highmem=off no matter what. qemu 7.0.0 only needs it for the threshold you found. I can amend my PR (#14291) to make that work.

[mprimeaux]

Sounds good. I am using qemu 7.0.0.

❯ qemu-system-aarch64 --version                                                                                                                      ─╯
QEMU emulator version 7.0.0
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers

FWIW, my playing around today resulted in these changes (rows 124 through 127).