Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mac M1 , Qemu2 Driver Not found #14176

Closed
dineshsnkumar opened this issue May 16, 2022 · 10 comments · Fixed by #14182
Closed

Mac M1 , Qemu2 Driver Not found #14176

dineshsnkumar opened this issue May 16, 2022 · 10 comments · Fixed by #14182
Labels
co/qemu-driver QEMU related issues kind/bug Categorizes issue or PR as related to a bug.

Comments

@dineshsnkumar
Copy link

dineshsnkumar commented May 16, 2022

What Happened?

I ran minikube using qemu2 driver but I am getting a PROVIDER_QEMU2_NOT_FOUND error

minikube start --driver=qemu2 --memory=26.5g --disk-size=270g --cpus=8 \
 --addons="[dashboard,default-storageclass,ingress,ingress-dns,metrics-server,registry,storage-provisioner]" \
 --kubernetes-version=1.23.6 --wait=all --dns-proxy=false --host-dns-resolver=true \
 --insecure-registry="10.0.0.0/16" --insecure-registry="192.168.49.0/24"
😄  minikube v1.26.0-beta.0 on Darwin 12.0.1 (arm64)
✨  Using the qemu2 (experimental) driver based on user configuration

🤷  Exiting due to PROVIDER_QEMU2_NOT_FOUND: The 'qemu2' provider was not found: exec: "qemu-system-aarch64": executable file not found in $PATH
💡  Suggestion: Install qemu-system
📘  Documentation: https://minikube.sigs.k8s.io/docs/reference/drivers/qemu2/

So I installed qemu using homebrew and now I am getting the following error

minikube start --driver=qemu2 --memory=26.5g --disk-size=270g --cpus=8 \
 --addons="[dashboard,default-storageclass,ingress,ingress-dns,metrics-server,registry,storage-provisioner]" \
 --kubernetes-version=1.23.6 --wait=all --dns-proxy=false --host-dns-resolver=true \
 --insecure-registry="10.0.0.0/16" --insecure-registry="192.168.49.0/24"
😄  minikube v1.26.0-beta.0 on Darwin 12.0.1 (arm64)
✨  Using the qemu2 (experimental) driver based on user configuration

🤷  Exiting due to PROVIDER_QEMU2_NOT_FOUND: The 'qemu2' provider was not found: stat /opt/homebrew/Cellar/qemu/6.2.0_1/share/qemu/edk2-aarch64-code.fd: no such file or directory
💡  Suggestion: Install uefi firmware
📘  Documentation: https://minikube.sigs.k8s.io/docs/reference/drivers/qemu2/

Attach the log file

Binary: Built with gc go1.18.1 for darwin/arm64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0516 14:24:13.513114   28241 out.go:296] Setting OutFile to fd 1 ...
I0516 14:24:13.513240   28241 out.go:348] isatty.IsTerminal(1) = true
I0516 14:24:13.513242   28241 out.go:309] Setting ErrFile to fd 2...
I0516 14:24:13.513244   28241 out.go:348] isatty.IsTerminal(2) = true
I0516 14:24:13.513294   28241 root.go:322] Updating PATH: /Users/dineshsankuri/.minikube/bin
I0516 14:24:13.513506   28241 out.go:303] Setting JSON to false
I0516 14:24:13.535424   28241 start.go:115] hostinfo: {"hostname":"Emplifis-MacBook-Pro.local","uptime":15888,"bootTime":1652709565,"procs":468,"os":"darwin","platform":"darwin","platformFamily":"Standalone Workstation","platformVersion":"12.0.1","kernelVersion":"21.1.0","kernelArch":"arm64","virtualizationSystem":"","virtualizationRole":"","hostId":"283ed594-67ba-5a7a-afbd-6537a2c07502"}
W0516 14:24:13.535530   28241 start.go:123] gopshost.Virtualization returned error: not implemented yet
I0516 14:24:13.556486   28241 out.go:177] 😄  minikube v1.26.0-beta.0 on Darwin 12.0.1 (arm64)
I0516 14:24:13.595928   28241 notify.go:193] Checking for updates...
I0516 14:24:13.596059   28241 driver.go:358] Setting default libvirt URI to qemu:///system
I0516 14:24:13.616017   28241 out.go:177] ✨  Using the qemu2 (experimental) driver based on user configuration
I0516 14:24:13.652781   28241 start.go:284] selected driver: qemu2
I0516 14:24:13.652787   28241 start.go:806] validating driver "qemu2" against <nil>
I0516 14:24:13.652801   28241 start.go:817] status for qemu2: {Installed:false Healthy:false Running:false NeedsImprovement:false Error:stat /opt/homebrew/Cellar/qemu/6.2.0_1/share/qemu/edk2-aarch64-code.fd: no such file or directory Reason: Fix:Install uefi firmware Doc:https://minikube.sigs.k8s.io/docs/reference/drivers/qemu2/ Version:}
I0516 14:24:13.671828   28241 out.go:177]
W0516 14:24:13.691417   28241 out.go:239] 🤷  Exiting due to PROVIDER_QEMU2_NOT_FOUND: The 'qemu2' provider was not found: stat /opt/homebrew/Cellar/qemu/6.2.0_1/share/qemu/edk2-aarch64-code.fd: no such file or directory
W0516 14:24:13.691501   28241 out.go:239] 💡  Suggestion: Install uefi firmware
W0516 14:24:13.691528   28241 out.go:239] 📘  Documentation: https://minikube.sigs.k8s.io/docs/reference/drivers/qemu2/
I0516 14:24:13.728811   28241 out.go:177]

Operating System

macOS (Default)

Driver

N/A

@afbjorklund
Copy link
Collaborator

Please use text for the logs

Looks like an old qemu version was hardcoded, that will need to use the current version (7.0 ?)

@afbjorklund afbjorklund added qemu-driver kind/bug Categorizes issue or PR as related to a bug. labels May 16, 2022
@dineshsnkumar
Copy link
Author

Sorry about using the image. I updated the post.

In homebrew the latest version of qemu is 6.2.0 so I tried to build the current version 7.0.0 from source code but I am having issues

In file included from /Users/dineshsankuri/Software/qemu-7.0.0/include/qemu/cpuid.h:14:
/Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/include/cpuid.h:14:2: error: this header is for x86 only
#error this header is for x86 only
 ^
/Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/include/cpuid.h:286:5: error: invalid output constraint '=a' in asm
    __cpuid(__leaf, __eax, __ebx, __ecx, __edx);
    ^
/Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/include/cpuid.h:251:11: note: expanded from macro '__cpuid'
        : "=a"(__eax), "=r" (__ebx), "=c"(__ecx), "=d"(__edx) \
          ^
/Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/include/cpuid.h:301:5: error: invalid output constraint '=a' in asm
    __cpuid(__leaf, *__eax, *__ebx, *__ecx, *__edx);
    ^
/Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/include/cpuid.h:251:11: note: expanded from macro '__cpuid'
        : "=a"(__eax), "=r" (__ebx), "=c"(__ecx), "=d"(__edx) \
          ^
/Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/include/cpuid.h:315:5: error: invalid output constraint '=a' in asm
    __cpuid_count(__leaf, __subleaf, *__eax, *__ebx, *__ecx, *__edx);
    ^
/Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/include/cpuid.h:258:11: note: expanded from macro '__cpuid_count'
        : "=a"(__eax), "=r" (__ebx), "=c"(__ecx), "=d"(__edx) \
          ^
In file included from ../tcg/tcg.c:431:
/Users/dineshsankuri/Software/qemu-7.0.0/tcg/i386/tcg-target.c.inc:1041:30: warning: cast to 'const void *' from smaller integer type 'tcg_target_long' (aka 'int') [-Wint-to-void-pointer-cast]
    diff = tcg_pcrel_diff(s, (const void *)arg) - 7;
                             ^~~~~~~~~~~~~~~~~
/Users/dineshsankuri/Software/qemu-7.0.0/tcg/i386/tcg-target.c.inc:4088:9: error: invalid output constraint '=a' in asm
        __cpuid_count(7, 0, a, b7, c7, d);
        ^
/Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/include/cpuid.h:258:11: note: expanded from macro '__cpuid_count'
        : "=a"(__eax), "=r" (__ebx), "=c"(__ecx), "=d"(__edx) \
          ^
In file included from ../tcg/tcg.c:431:
/Users/dineshsankuri/Software/qemu-7.0.0/tcg/i386/tcg-target.c.inc:4094:9: error: invalid output constraint '=a' in asm
        __cpuid(1, a, b, c, d);
        ^
/Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/include/cpuid.h:251:11: note: expanded from macro '__cpuid'
        : "=a"(__eax), "=r" (__ebx), "=c"(__ecx), "=d"(__edx) \
          ^
In file included from ../tcg/tcg.c:431:
/Users/dineshsankuri/Software/qemu-7.0.0/tcg/i386/tcg-target.c.inc:4114:44: error: invalid output constraint '=a' in asm
            asm(".byte 0x0f, 0x01, 0xd0" : "=a" (xcrl), "=d" (xcrh) : "c" (0));
                                           ^
/Users/dineshsankuri/Software/qemu-7.0.0/tcg/i386/tcg-target.c.inc:4140:9: error: invalid output constraint '=a' in asm
        __cpuid(0x80000001, a, b, c, d);
        ^
/Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/include/cpuid.h:251:11: note: expanded from macro '__cpuid'
        : "=a"(__eax), "=r" (__ebx), "=c"(__ecx), "=d"(__edx) \
          ^
1 warning and 8 errors generated.
ninja: build stopped: subcommand failed.
make[1]: *** [run-ninja] Error 1
make: *** [all] Error 2

@sharifelgamal
Copy link
Collaborator

6.2.0 is the correct version of qemu to install, it could be that the installation went to an unexpected location. I'm working on making firmware locations a bit more flexible.

@sharifelgamal
Copy link
Collaborator

What does ls /opt/homebrew/Cellar/qemu return for you?

@dineshsnkumar
Copy link
Author

It lists the version of qemu

6.2.0

@sharifelgamal
Copy link
Collaborator

ok, so the source of the issue is in fact the hardcoded path, i'll get that fixed asap and it'll be in the next beta.

@sharifelgamal
Copy link
Collaborator

@afbjorklund
Copy link
Collaborator

Actually the 7.0 was not released yet, the problem here was that brew was outdated (missing some patches, in 6.2.0_1)

The new more flexible location should be able to cope with both or all three, so that is a good fix to have either way.

@dineshsnkumar dineshsnkumar changed the title Mac M1 , Quemu2 Driver Not found Mac M1 , Qemu2 Driver Not found May 17, 2022
@dineshsnkumar
Copy link
Author

dineshsnkumar commented May 17, 2022

I was able to run using qemu driver but I am having issues enabling registry and ingress addons

minikube start --driver=qemu2 --memory=26.5g --disk-size=270g --cpus=8 \
 --addons="[dashboard,default-storageclass,ingress,ingress-dns,metrics-server,registry,storage-provisioner]" \
 --kubernetes-version=1.23.6 --wait=all --dns-proxy=false --host-dns-resolver=true \
 --insecure-registry="10.0.0.0/16" --insecure-registry="192.168.49.0/24"
😄  minikube v1.26.0-beta.0 on Darwin 12.0.1 (arm64)
✨  Using the qemu2 (experimental) driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating qemu2 VM (CPUs=8, Memory=27136MB, Disk=276480MB) ...
❗  This VM is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.23.6 on Docker 20.10.14 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
🔎  Verifying registry addon...
    ▪ Using image k8s.gcr.io/metrics-server/metrics-server:v0.6.1
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.2.0
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
🔎  Verifying ingress addon...
❗  Enabling 'registry' returned an error: running callbacks: [waiting for kubernetes.io/minikube-addons=registry pods: timed out waiting for the condition]
❗  Enabling 'ingress' returned an error: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]
🌟  Enabled addons: storage-provisioner, ingress-dns, default-storageclass, metrics-server
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Also, I am getting error configuring env to use docker deamon

~ minikube docker-env

❌  Exiting due to GUEST_STATUS: state: EOF

╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                             │
│    😿  If the above advice does not help, please let us know:                                                               │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                                                             │
│                                                                                                                             │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.                                      │
│    Please also attach the following file to the GitHub issue:                                                               │
│    - /var/folders/4n/4xjs2vc17bl528n2_76727xh0000gp/T/minikube_docker-env_cc2624d33c129ea5cb114f87ecb54f6350dfdddc_0.log    │
│                                                                                                                             │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

I attached the log file below

log.txt

@sharifelgamal
Copy link
Collaborator

This looks to be several different issues, docker-env and ingress both work on my machine, so it might have something to do with the insecure registry parameters. This driver is still experimental so we don't expect everything to work right away. The registry addon just doesn't work on arm64 at all right now, due to the underlying image used not being multiarch.

The initial issue posted here will be fixed when I merge the associated PR. The rest of the known issues for this driver are being tracked in #14146

@spowelljr spowelljr added co/qemu-driver QEMU related issues and removed qemu-driver labels Sep 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
co/qemu-driver QEMU related issues kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants