minikube (qemu2) failes to start with "failed to get IP address" #15960

denisko · 2023-03-03T09:09:04Z

What Happened?

Minikube fails to start:
Exiting due to GUEST_PROVISION: Failed to start host: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: could not find an IP address for xx:xx:xx:xx:xx:xx

Attach the log file

minikube-logs.txt
minikube_logs_e4032c57df89461a76cf8ea248c98ea6558dccb1_0.log

Operating System

macOS (Default)

Driver

QEMU

The text was updated successfully, but these errors were encountered:

afbjorklund · 2023-03-05T17:17:00Z

Some kind of compatibility issue with socket_vmnet, it only lists one DHCP entry coming from "colima"

I0303 10:05:06.492396   25101 main.go:141] libmachine: Searching for 2:a8:48:27:20:f2 in /var/db/dhcpd_leases ...
I0303 10:05:06.492450   25101 main.go:141] libmachine: Found 1 entries in /var/db/dhcpd_leases!
I0303 10:05:06.492474   25101 main.go:141] libmachine: dhcp entry: {Name:colima IPAddress:192.168.106.2 HWAddress:52:55:55:da:34:59 ID:1,52:55:55:da:34:59 Lease:0x63eb6dd7}

mprimeaux · 2023-04-20T00:50:14Z

I'm not sure what is going on yet but I am also experiencing this issue, which seems to be a regression as compared to versions before minikube 1.30.x but I do need to verify.

I'll post logs but, in summary, I'm unable to successfully start a minikube cluster when using socket_vmnet (via brew or direct source compilation) in NAT or bridged mode. This used to work but does not even after following steps outlined in Known Issues.

Here are my machine details:

❯ macchina                                                                                                                                                 ─╯

                  ,MMMM.           Host        -  <REMOVED>
                .MMMMMM            Machine     -  Mac13,2
                MMMMM,             Kernel      -  22.4.0
      .;MMMMM:' MMMMMMMMMM;.       OS          -  macOS 13.3.1 Ventura
    MMMMMMMMMMMMNWMMMMMMMMMMM:     DE          -  Aqua
  .MMMMMMMMMMMMMMMMMMMMMMMMWM.     WM          -  Quartz Compositor
  MMMMMMMMMMMMMMMMMMMMMMMMM.       Packages    -  263 (Homebrew)
 ;MMMMMMMMMMMMMMMMMMMMMMMM:        Terminal    -  iTerm2 (Version 3.4.19)
 :MMMMMMMMMMMMMMMMMMMMMMMM:        Shell       -  zsh
 .MMMMMMMMMMMMMMMMMMMMMMMMM.       Uptime      -  3d 5h 49m
  MMMMMMMMMMMMMMMMMMMMMMMMMMM.     CPU         -  Apple M1 Ultra (20)
   .MMMMMMMMMMMMMMMMMMMMMMMMMM.    Resolution  -  5120x2880@60fps (as 2560x1440)
     MMMMMMMMMMMMMMMMMMMMMMMM                     5120x2880@60fps (as 2560x1440)
      ;MMMMMMMMMMMMMMMMMMMM.       Brightness  -  100%
        .MMMM,.    .MMMM,.         CPU Load    -  18%
                                   Memory      -  60.2 GB/134.2 GB

Minikube version is:

minikube version: v1.30.1
commit: 08896fd1dc362c097c925146c4a0d0dac715ace0

I will do more testing tomorrow.

mprimeaux · 2023-04-23T22:33:14Z

FWIW, here are my logs from today's latest attempt to start minikube with socket_vmnet.

minikube_logs_246a8c5e046f473e09b808593858b4737e8d9296_0.log

❯ minikube config view
- container-runtime: containerd
- cpus: 6
- driver: qemu2
- memory: 16384

❯ minikube start --network socket_vmnet
😄  minikube v1.30.1 on Darwin 13.3.1 (arm64)
✨  Using the qemu2 driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
💾  Downloading Kubernetes v1.26.3 preload ...
    > preloaded-images-k8s-v18-v1...:  358.98 MiB / 358.98 MiB  100.00% 44.94 M
🔥  Creating qemu2 VM (CPUs=6, Memory=16384MB, Disk=20000MB) ...
🔥  Deleting "minikube" in qemu2 ...
🤦  StartHost failed, but will try again: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: could not find an IP address for 5e:40:c0:5c:ab:86
🔥  Creating qemu2 VM (CPUs=6, Memory=16384MB, Disk=20000MB) ...
😿  Failed to start qemu2 VM. Running "minikube delete" may fix it: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: could not find an IP address for 42:74:86:6:b8:92

❌  Exiting due to GUEST_PROVISION: error provisioning guest: Failed to start host: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: could not find an IP address for 42:74:86:6:b8:92

╭───────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                           │
│    😿  If the above advice does not help, please let us know:                             │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                           │
│                                                                                           │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────╯

I'm using socket_vmnet via brew but not in bridged mode.

❯ sudo brew services info socket_vmnet 
socket_vmnet (homebrew.mxcl.socket_vmnet)
Running: ✔
Loaded: ✔
Schedulable: ✘
User: root
PID: 743

I've also tried different values of container-runtime to no avail. I'll find more time tomorrow to debug a bit deeper.

torenware · 2023-07-18T21:20:58Z

Just tried this with newly released minikube v1.31.0. Still busted on amd64 (might be working as 1.30.x on arm64!)

mprimeaux · 2023-07-18T22:45:23Z

Still busted on arm64, also. As several folks have mentioned, the fix for me on both amd64 and arm64 is to restart socket_vmnet via brew services and then to run the following:

minikube delete
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd

This workaround works for me 100% of the time.

spowelljr · 2023-07-18T22:50:34Z

minikube v1.31.0 should unblock the firewall if it detects it's being blocked.

$ $ ./out/minikube start --driver qemu
😄  minikube v1.31.0 on Darwin 13.4.1 (arm64)
✨  Using the qemu2 driver based on user configuration
🌐  Automatically selected the socket_vmnet network
🔑  Your firewall is blocking bootpd which is required for socket_vmnet. The following commands will be executed to unblock bootpd:

    $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd 
    $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd 


Password: 
👍  Starting control plane node minikube in cluster minikube
💾  Downloading Kubernetes v1.27.3 preload ...
    > preloaded-images-k8s-v18-v1...:  327.72 MiB / 327.72 MiB  100.00% 50.68 M
🔥  Creating qemu2 VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
❗  This VM is having trouble accessing https://registry.k8s.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.27.3 on Docker 24.0.4 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

But restarting socket_vmnet via brew services is also required to get it working for you @mprimeaux?

torenware · 2023-07-19T01:04:37Z

Still busted on arm64, also. As several folks have mentioned, the fix for me on both amd64 and arm64 is to restart socket_vmnet via brew services and then to run the following:
minikube delete
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd
This workaround works for me 100% of the time.

I can't get it to run on amd64, even with the ownership on /usr/local/var/run/socket_vmnet set to root:admin (which makes it possible to run minikube start as non-root). Here's what I see on my Intel Macbook Pro, immediately after giving the user password for sudo:

Robs-MBP:images myuser$ minikube -p vmnet delete
🔥  Deleting "vmnet" in qemu2 ...
💀  Removed all traces of the "vmnet" cluster.
Robs-MBP:images myuser$ minikube -p vmnet start --driver qemu
😄  [vmnet] minikube v1.31.0 on Darwin 11.7.8
✨  Using the qemu2 driver based on user configuration
🌐  Automatically selected the socket_vmnet network
👍  Starting control plane node vmnet in cluster vmnet
🔥  Creating qemu2 VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
🔑  Your firewall is blocking bootpd which is required for socket_vmnet. The following commands will be executed to unblock bootpd:

    $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
    $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd


🔄  Successfully unblocked bootpd process from firewall, retrying
🔥  Deleting "vmnet" in qemu2 ...
🤦  StartHost failed, but will try again: creating host: create: creating: ip not found: failed to get IP address: could not find an IP address for 32:82:30:a9:ad:e1
🔥  Creating qemu2 VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
🔑  Your firewall is blocking bootpd which is required for socket_vmnet. The following commands will be executed to unblock bootpd:

    $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
    $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd


🔄  Successfully unblocked bootpd process from firewall, retrying
😿  Failed to start qemu2 VM. Running "minikube delete -p vmnet" may fix it: creating host: create: creating: ip not found: failed to get IP address: could not find an IP address for b2:f9:59:88:37:d6

❌  Exiting due to GUEST_PROVISION: error provisioning guest: Failed to start host: creating host: create: creating: ip not found: failed to get IP address: could not find an IP address for b2:f9:59:88:37:d6

╭───────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                           │
│    😿  If the above advice does not help, please let us know:                             │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                           │
│                                                                                           │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────╯

mprimeaux · 2023-07-19T01:09:49Z

Did you run sudo brew services restart socket_vmnet prior to the above?

torenware · 2023-07-19T01:13:19Z

@mprimeaux Yep. Here's the status:

Robs-MBP:images myuser$ sudo brew services info socket_vmnet
socket_vmnet (homebrew.mxcl.socket_vmnet)
Running: ✔
Loaded: ✔
Schedulable: ✘
User: root
PID: 94341

mprimeaux · 2023-07-19T01:15:18Z

@torenware There are days I wish I were on a beach :)

Interestingly enough, I had this very scenario happen when I was in an airport last week. Turns our, the captured network was preventing my VM from obtaining a DHCP address. By chance, are you on a captured network?

torenware · 2023-07-19T01:18:09Z

File:

@torenware There are days I wish I were on a beach :)

Interestingly enough, I had this very scenario happen when I was in an airport last week. Turns our, the captured network was preventing my VM from obtaining a DHCP address. By chance, are you on a captured network?

Nope. Plain vanilla wifi, no fancy stuff going on.

torenware · 2023-07-19T01:18:46Z

logs.txt

haroldship · 2023-08-30T08:44:02Z

I also have this issue, Macbook Pro 16" 2019 Intel running Ventura 13.5.
logs.txt
minikube_logs_0720879eb88b105f38a2418cc93cdf68b441630a_0.log

richpryce · 2023-09-27T11:55:53Z

I am having this exact same issue as well. Has a fix been found?

blind3dd · 2023-11-12T00:23:35Z

I think I have found a workaround + there is a code change needed I've just created.

First you start colima, make sure it works, then socket_vmnet.

Also make sure to define proper firewall rules for bridge created.
When colima starts it defines the port, make sure your lease is something with 127.0.0.1 in it matching the mac address found through the leases from /var/db/dhcpd_leases.

What is more, you're going to need to edit the ~/.minikube/machines/<your_machine>/config.json to put there SSHPort and SSHKeyPath of lima ($HOME/.lima/_config/user), I have gone with ssh-add because colima ~/.colima/ssh_config has ForwardAgent yes, anyway the Port there is generated different every time, so probably this PR below should be updated to discover it automatically and read from ssh_config of colima.

After editing a profile for colima/preparing it (containerd, sshfs):

colima start --cpu 4 --memory 8 --runtime containerd --vm-type qemu --ssh-agent --very-verbose --arch aarch64

minikube start --driver qemu2 --network socket_vmnet --cpus 4 --memory 8192 --disk-size 60G --container-runtime containerd --namespace default -p kubernetes-colima --network socket_vmnet --alsologtostderr -v=7

I have also made sure I've added proper anchor and pfctl rules for the bridge created as mentioned above and made sure other firewalls are not blocking bootpd (system and apps).

Basically this needs to appear (in my case) - username cannot be overriden hence the initial PR.
ssh -p 49590 usualsuspectx@127.0.0.1 Welcome to Alpine! The Alpine Wiki contains a large amount of how-to guides and general information about administrating Alpine systems. See <https://wiki.alpinelinux.org/>. You can setup the system with the command: setup-alpine You may change this message by editing /etc/motd. colima:~$

λ /Users/ colima list PROFILE STATUS ARCH CPUS MEMORY DISK RUNTIME ADDRESS default Running aarch64 2 4GiB 60GiB containerd

Initial PR: 7e9f3a6

PR has been updated to cover the SSHPort discovery of colima machine and its ssh_config.

afbjorklund · 2023-11-12T12:07:22Z

It was not intended to start two VMs, so it seems strange to have both colima and minikube qemu2 running.

Overriding the user name is something that could be considered for all drivers, but it is not needed for the kic image.

For colima, it normally starts k3s when you run it with --kubernetes.

colima start --kubernetes

For lima, it simply runs kubeadm on the node instead of using minikube.

limactl start template://k8s

blind3dd · 2023-11-14T16:37:46Z

Agreed that it is not common, I had to run vmnet separately having those 2 machines and I do not run k3s there with colima, however I was able to obtain the minikube address with the usage of localhost and MAC address in this setup. I am generally thinking about moving this towards lima driver (or dig into qemu2 even deeper) and it's just how I started: see the ip below:

|      Profile      | VM Driver |  Runtime   |    IP     | Port | Version | Status  | Nodes | Active |
|-------------------|-----------|------------|-----------|------|---------|---------|-------|--------|
| kubernetes-colima | qemu2     | containerd | 10.0.2.15 | 8443 | v1.27.4 | Stopped |     1 |        |
|-------------------|-----------|------------|-----------|------|---------|---------|-------|--------|```

blind3dd · 2023-12-16T02:33:35Z

Agreed that 2 VMs are not intended to start, I changed the course and worked with dhcp leases a little bit, created proper bridge with vmnet and used lima without colima, now it looks correct, getting myself to speed with the project, can't wait to add more nodes

λ ~/ minikube profile list
|-------------------|-----------|------------|-------------|------|---------|---------|-------|--------|
| Profile | VM Driver | Runtime | IP | Port | Version | Status | Nodes | Active |
|-------------------|-----------|------------|-------------|------|---------|---------|-------|--------|
| kubernetes-colima | qemu2 | containerd | 192.168.5.4 | 8443 | v1.28.4 | Running | 1 | |
|-------------------|-----------|------------|-------------|------|---------|---------|-------|--------|

λ ~/ kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-5dd5756b68-hkgbw 1/1 Running 0 41s
kube-system etcd-kubernetes-colima 1/1 Running 0 54s
kube-system kube-apiserver-kubernetes-colima 1/1 Running 0 54s
kube-system kube-controller-manager-kubernetes-colima 1/1 Running 0 54s
kube-system kube-proxy-q4k9b 1/1 Running 0 42s
kube-system kube-scheduler-kubernetes-colima 1/1 Running 0 54s
kube-system storage-provisioner 1/1 Running 0 54s

λ ~/ k get no
NAME STATUS ROLES AGE VERSION
kubernetes-colima Ready control-plane 4m1s v1.28.4`

If I am able to find some free time I will see if this issue can be somehow addressed - need to make sure I have full understanding of the setup. Starts getting proper shapes.

Started with:
minikube start --driver qemu2 --cpus 4 --memory 8192 --disk-size 60G --container-runtime containerd --namespace default -p kubernetes-colima --alsologtostderr -v=7 --cni=bridge

k8s-triage-robot · 2024-03-15T03:07:55Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2024-04-14T03:20:13Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle rotten
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

obnoxxx · 2024-06-05T10:30:41Z

@mprimeaux wrote:

Still busted on arm64, also. As several folks have mentioned, the fix for me on both amd64 and arm64 is to restart socket_vmnet via brew services and then to run the following:
minikube delete
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd
This workaround works for me 100% of the time.

This worked for me once yesterday, and I was so happy, but today it fails again:

GUEST_PROVISION: error provisioning guest: Failed to start host: creating host: create: creating: ip not found: failed to get IP address: could not find an IP address for 66:17:20:72:c8:65

In my case, the firewall does not seem to be the problem because bootpdis listed there. But my suspicion is that the problem is that bootpdis not even running. How can I get it to run?

obnoxxx · 2024-06-05T13:54:28Z

I managed to get bootpd running by doing sudo /usr/libexec/bootpd but it doesnt help minikube starting. Still failing with failed to get IP address

afbjorklund added co/qemu-driver QEMU related issues os/macos kind/bug Categorizes issue or PR as related to a bug. labels Mar 5, 2023

afbjorklund mentioned this issue Nov 12, 2023

qemu2 default ssh user - point to host one to be able to ssh to colima without docker desktop #17602

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 15, 2024

obnoxxx mentioned this issue Jun 4, 2024

Can not start minikube on MacOS #19014

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

minikube (qemu2) failes to start with "failed to get IP address" #15960

minikube (qemu2) failes to start with "failed to get IP address" #15960

denisko commented Mar 3, 2023 •

edited

Loading

afbjorklund commented Mar 5, 2023

mprimeaux commented Apr 20, 2023

mprimeaux commented Apr 23, 2023

torenware commented Jul 18, 2023

mprimeaux commented Jul 18, 2023 •

edited

Loading

spowelljr commented Jul 18, 2023 •

edited

Loading

torenware commented Jul 19, 2023

mprimeaux commented Jul 19, 2023

torenware commented Jul 19, 2023

mprimeaux commented Jul 19, 2023 •

edited

Loading

torenware commented Jul 19, 2023

torenware commented Jul 19, 2023

haroldship commented Aug 30, 2023

richpryce commented Sep 27, 2023

blind3dd commented Nov 12, 2023 •

edited

Loading

afbjorklund commented Nov 12, 2023 •

edited

Loading

blind3dd commented Nov 14, 2023 •

edited

Loading

blind3dd commented Dec 16, 2023 •

edited

Loading

k8s-triage-robot commented Mar 15, 2024

k8s-triage-robot commented Apr 14, 2024

obnoxxx commented Jun 5, 2024

obnoxxx commented Jun 5, 2024

minikube (qemu2) failes to start with "failed to get IP address" #15960

minikube (qemu2) failes to start with "failed to get IP address" #15960

Comments

denisko commented Mar 3, 2023 • edited Loading

What Happened?

Attach the log file

Operating System

Driver

afbjorklund commented Mar 5, 2023

mprimeaux commented Apr 20, 2023

mprimeaux commented Apr 23, 2023

torenware commented Jul 18, 2023

mprimeaux commented Jul 18, 2023 • edited Loading

spowelljr commented Jul 18, 2023 • edited Loading

torenware commented Jul 19, 2023

mprimeaux commented Jul 19, 2023

torenware commented Jul 19, 2023

mprimeaux commented Jul 19, 2023 • edited Loading

torenware commented Jul 19, 2023

torenware commented Jul 19, 2023

haroldship commented Aug 30, 2023

richpryce commented Sep 27, 2023

blind3dd commented Nov 12, 2023 • edited Loading

afbjorklund commented Nov 12, 2023 • edited Loading

blind3dd commented Nov 14, 2023 • edited Loading

blind3dd commented Dec 16, 2023 • edited Loading

k8s-triage-robot commented Mar 15, 2024

k8s-triage-robot commented Apr 14, 2024

obnoxxx commented Jun 5, 2024

obnoxxx commented Jun 5, 2024

denisko commented Mar 3, 2023 •

edited

Loading

mprimeaux commented Jul 18, 2023 •

edited

Loading

spowelljr commented Jul 18, 2023 •

edited

Loading

mprimeaux commented Jul 19, 2023 •

edited

Loading

blind3dd commented Nov 12, 2023 •

edited

Loading

afbjorklund commented Nov 12, 2023 •

edited

Loading

blind3dd commented Nov 14, 2023 •

edited

Loading

blind3dd commented Dec 16, 2023 •

edited

Loading