hyperkit start: Could not find an IP address for 6a:c7:25:8a:7f:2b

[zhaytee]

Is this a BUG REPORT or FEATURE REQUEST?: Bug

Please provide the following details:

Environment:

Minikube version: 0.22.0

• OS: macOS 10.12.6
• VM Driver: hyperkit
• ISO version: minikube-v0.23.3.iso
• Install tools:
• Others:

What happened:
Cluster did not start:

$ minikube start --vm-driver hyperkit -v 10
Starting local Kubernetes v1.7.5 cluster...
Starting VM...
Found binary path at /usr/local/bin/docker-machine-driver-hyperkit
Launching plugin server for driver hyperkit
Plugin server listening at address 127.0.0.1:58797
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetState
(minikube) Calling .Start
(minikube) Generated UUID 23bf5b8b-93fe-11e7-b603-784f43517719
(minikube) Generated MAC 6a:c7:25:8a:7f:2b
(minikube) Starting with cmdline: loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes base host=minikube
E0907 11:56:39.075265   54829 start.go:143] Error starting host: Error starting stopped host: IP address never found in dhcp leases file Temporary Error: Could not find an IP address for 6a:c7:25:8a:7f:2b
Temporary Error: Could not find an IP address for 6a:c7:25:8a:7f:2b
[... snip a ton of these errors...]

What you expected to happen:
The cluster starts normally.

How to reproduce it (as minimally and precisely as possible):
See above command

Output of minikube logs (if applicable):

Anything else do we need to know:
I built docker-machine-driver-hyperkit myself, but got the minikube binary from the latest official release here on Github. If there's a more official place to get docker-machine-driver-hyperkit, I'll try using that one.

[r2d4]

Can you see if there are any entries in the "/var/db/dhcpd_leases file?

[zhaytee]

@r2d4 Yep, looks like there are two entries. I've since gone back to running minikube using virtualbox, which may be relevant to its contents:

$ cat /var/db/dhcpd_leases
{
	name=minikube
	ip_address=192.168.64.3
	hw_address=1,ce:8d:37:74:2a:93
	identifier=1,ce:8d:37:74:2a:93
	lease=0x59b2e6b9
}
{
	name=minikube
	ip_address=192.168.64.2
	hw_address=1,96:fe:4d:30:dd:21
	identifier=1,96:fe:4d:30:dd:21
	lease=0x599d2141
}

[zhaytee]

Hey folks. I rebuilt docker-machine-driver-hyperkit from the git repo at tag v0.22.0, deleted ~/.minikube, deleted /var/db/dhcpd_leases, and tried to create a fresh cluster using minikube 0.22.0: minikube start --vm-driver hyperkit -v 10

First attempt, I get this curious error (curious because my laptop has over 50GB free; I suspect the error is referring to something else):

$ minikube start --vm-driver hyperkit -v 10
Starting local Kubernetes v1.7.5 cluster...
Starting VM...
Found binary path at ~/.local/bin/docker-machine-driver-hyperkit
Launching plugin server for driver hyperkit
Plugin server listening at address 127.0.0.1:50171
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetMachineName
(minikube) Calling .DriverName
(minikube) Calling .DriverName
Creating CA: ~/.minikube/certs/ca.pem
Creating client certificate: ~/.minikube/certs/cert.pem
Running pre-create checks...
(minikube) Calling .PreCreateCheck
(minikube) Calling .GetConfigRaw
Creating machine...
(minikube) Calling .Create
(minikube) Downloading ~/.minikube/cache/boot2docker.iso from file://~/.minikube/cache/iso/minikube-v0.23.3.iso...
(minikube) DBG | Writing magic tar header
(minikube) DBG | Writing SSH key tar header
E0913 00:06:02.067017   10983 start.go:143] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: truncate ~/.minikube/machines/minikube/minikube.rawdisk: no space left on device.

So I try the same command again, and it gets past that point. But now I get a different error, a permissions issue with getting ssh going (it can't read id_rsa because it's owned by root?):

$ minikube start --vm-driver hyperkit -v 10
Starting local Kubernetes v1.7.5 cluster...
Starting VM...
Found binary path at ~/.local/bin/docker-machine-driver-hyperkit
Launching plugin server for driver hyperkit
Plugin server listening at address 127.0.0.1:50177
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetState
(minikube) Error reading pid file: open ~/.minikube/machines/minikube/hyperkit.json: no such file or directory
(minikube) Calling .Start
(minikube) Generated UUID 07a53fc3-9852-11e7-8a4e-acde48001122
(minikube) Generated MAC 6e:fd:b9:e9:1a:eb
(minikube) Starting with cmdline: loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes base host=minikube
(minikube) Calling .GetConfigRaw
(minikube) Calling .DriverName
Waiting for SSH to be available...
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) Calling .GetSSHPort
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHUsername
Using SSH client type: native
<nil>
Error getting ssh command 'exit 0' : Error getting config for native Go SSH: open ~/.minikube/machines/minikube/id_rsa: permission denied

Here are the contents of ~/.minikube/machines/minikube:

$ ls -l ~/.minikube/machines/minikube
total 434776
-rw-------  1 root        staff  111521792 Sep 13 00:05 boot2docker.iso
-rw-r--r--  1 root        staff    7187728 Sep 13 00:05 bzimage
-rw-------  1 zhaytee  staff       2545 Sep 13 00:06 config.json
-rw-r--r--  1 root        staff      65536 Sep 13 00:06 console-ring
-rw-r--r--  1 root        staff       2254 Sep 13 00:06 hyperkit.json
-rw-r--r--  1 root        staff          5 Sep 13 00:06 hyperkit.pid
-rw-------  1 root        staff       1679 Sep 13 00:05 id_rsa
-rw-------  1 root        staff        381 Sep 13 00:05 id_rsa.pub
-rw-r--r--  1 root        staff  103790296 Sep 13 00:06 initrd
-rw-r--r--  1 root        staff        206 Sep 13 00:06 isolinux.cfg
-rw-r--r--  1 root        staff       4608 Sep 13 00:06 minikube.rawdisk
lrwxr-xr-x  1 root        staff         12 Sep 13 00:06 tty@ -> /dev/ttys031

I had installed docker-machine-driver-hyperkit using the Makefile entry install-hyperkit-driver, which sets the owner to root:wheel and sets the setuid bit via chmod u+s.

Is there an obvious path to victory here? Wondering what I'm missing.

Thank you!

[antoineco]

edit: can't reproduce anymore after wiping ~/.minikubeand /var/db/dhcpd_leases.

--

I have the same error message as the OP as soon as I try to restart a stopped minikube VM after a macOS reboot (and only in that case):

Temporary Error: Could not find an IP address for <mac_addr>.

It should be mentioned that <mac_addr> is different after every invocation of minikube start.

minikube v0.23.0
docker-machine-driver-hyperkit @10f2bb57

[kenden]

I can only get it to work using sudo.
Else I get:

virtio_net: Could not create vmnet interface, permission denied or no entitlement?
start.go:150] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: vmnet: error from vmnet.framework: -1

Repro steps:

# current last commit: bc20281f Update hyperkit go driver.
$ make out/docker-machine-driver-hyperkit  # build docker-machine-driver-hyperkit
$ make  # build minikube
export PATH=$PATH:/Users/myuser/dev/go/src/k8s.io/minikube/out/  # add hyperkit driver to path

$ ./out/minikube start --vm-driver hyperkit -v 7
Starting local Kubernetes v1.8.0 cluster...
Starting VM...
Found binary path at /Users/myuser/devgo/src/k8s.io/minikube/out/docker-machine-driver-hyperkit
Launching plugin server for driver hyperkit
Plugin server listening at address 127.0.0.1:57472
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetMachineName
(minikube) Calling .DriverName
(minikube) Calling .DriverName
Running pre-create checks...
(minikube) Calling .PreCreateCheck
(minikube) Calling .GetConfigRaw
Creating machine...
(minikube) Calling .Create
(minikube) Downloading /Users/myuser/.minikube/cache/boot2docker.iso from file:///Users/myuser/.minikube/cache/iso/minikube-v0.23.5.iso...
(minikube) DBG | 2017/11/21 11:16:50 [INFO] Creating ssh key...
(minikube) DBG | 2017/11/21 11:16:50 [INFO] Creating raw disk image...
(minikube) DBG | Writing magic tar header
(minikube) DBG | Writing SSH key tar header
(minikube) Generated UUID 171ba2a0-cea5-11e7-9b4d-3c15c2ce618a
(minikube) DBG | virtio_net: Could not create vmnet interface, permission denied or no entitlement?
E1121 11:16:53.072514   39579 start.go:150] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: vmnet: error from vmnet.framework: -1
.

# ----> virtio_net: Could not create vmnet interface, permission denied or no entitlement?
# ----> let's try with sudo

$ ./out/minikube delete
$ sudo ./out/minikube start --vm-driver hyperkit
Starting local Kubernetes v1.8.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.

$ kubectl get nodes
NAME       STATUS    ROLES     AGE       VERSION
minikube   Ready     <none>    12s       v1.8.0

Version:
macOS 10.13.1 (High Sierra)

$ uname -a
Darwin apollo 17.2.0 Darwin Kernel Version 17.2.0: Fri Sep 29 18:27:05 PDT 2017; root:xnu-4570.20.62~3/RELEASE_X86_64 x86_64 i386 MacBookPro11,3 Darwin

[antoineco]

Did you forget to set the setuid flag on the docker-machine-driver-hyperkit binary cher Monsieur Nerden? (sudo chmod u+s out/docker-machine-driver-hyperkit)

[kenden]

@antoineco Indeed, merci gamin. When replacing my steps:

$ make out/docker-machine-driver-hyperkit  # build docker-machine-driver-hyperkit
$ export PATH=$PATH:/Users/myuser/dev/go/src/k8s.io/minikube/out/

by:

$ make install-hyperkit-driver

then it works fine, and without sudo.

It also works fine using the official version v0.23.0.

[dlorenc]

Thanks for the update! We should add an error that helps track down when you forget to set setuid.

[dlorenc]

This should be fixed with #2215

[antoineco]

@dlorenc the original issue was not about permissions, rather about this error:

Temporary Error: Could not find an IP address for <mac_addr>

Can we please reopen?