Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

macOS: Can't create minikube with xhyve #708

Closed
pquentin opened this issue Oct 18, 2016 · 31 comments
Closed

macOS: Can't create minikube with xhyve #708

pquentin opened this issue Oct 18, 2016 · 31 comments
Labels
co/xhyve kind/bug Categorizes issue or PR as related to a bug. os/macos

Comments

@pquentin
Copy link

pquentin commented Oct 18, 2016

Minikube version (use minikube version): 0.11.0

Environment:

  • OS (e.g. from /etc/os-release): macOS Sierra 10.12
  • VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): xhyve
  • Docker version (e.g. docker -v): Docker version 1.12.2, build bb80604 installed through homebrew
  • Install tools: Installed minikube using homebrew cask
  • Others: Docker for Mac is not installed. Docker Toolbox is not installed.

What happened:

My team uses Kubernetes in production and we recently decided to switch local development from docker-compose to minikube. Their migration went well, but not mine. I tried to create a minikube cluster like this:

$ docker -v
Docker version 1.12.2, build bb80604
$ minikube delete
Deleting local Kubernetes cluster...
Machine deleted.
$ minikube start --vm-driver=xhyve --show-libmachine-logs --v=10 --alsologtostderr
I1018 14:54:32.935093   73832 notify.go:111] Checking for updates...
Starting local Kubernetes cluster...
Found binary path at /usr/local/bin/docker-machine-driver-xhyve
Launching plugin server for driver xhyve
Plugin server listening at address 127.0.0.1:61922
() DBG | operation not supported by device
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetMachineName
(minikube) Calling .DriverName
Running pre-create checks...
(minikube) Calling .PreCreateCheck
(minikube) DBG | ===== Docker Machine xhyve Driver Version 0.2.3 (Homebrew) =====
(minikube) DBG |
(minikube) DBG | executing: /usr/local/bin/VBoxManage -v
(minikube) DBG | STDOUT: 5.0.4r102546
(minikube) DBG |
(minikube) DBG | STDERR:
(minikube) Calling .GetConfigRaw
Creating machine...
(minikube) Calling .Create
(minikube) Downloading /Users/quentin/.minikube/cache/boot2docker.iso from file:///Users/quentin/.minikube/cache/iso/minikube-0.7.iso...
(minikube) Creating VM...
(minikube) Extracting vmlinuz64 and initrd.img from boot2docker.iso...
(minikube) DBG | Mounting boot2docker.iso
(minikube) DBG | executing: &{/usr/bin/hdiutil [hdiutil attach /Users/quentin/.minikube/machines/minikube/boot2docker.iso -mountpoint /Users/quentin/.minikube/machines/minikube/b2d-image] []  <nil> 0xc42002e010 0xc42002e018 [] <nil> <nil> <nil> <nil> <nil> false [] [] [] [] <nil> <nil>} attach /Users/quentin/.minikube/machines/minikube/boot2docker.iso -mountpoint /Users/quentin/.minikube/machines/minikube/b2d-image
(minikube) /dev/disk4                                           /Users/quentin/.minikube/machines/minikube/b2d-image
(minikube) DBG | Extracting vmlinuz64 into /Users/quentin/.minikube/machines/minikube
(minikube) DBG | Extracting initrd.img into /Users/quentin/.minikube/machines/minikube
(minikube) DBG | Unmounting boot2docker.iso
(minikube) DBG | executing: &{/usr/bin/hdiutil [hdiutil detach /Users/quentin/.minikube/machines/minikube/b2d-image] []  <nil> 0xc42002e010 0xc42002e018 [] <nil> <nil> <nil> <nil> <nil> false [] [] [] [] <nil> <nil>} detach /Users/quentin/.minikube/machines/minikube/b2d-image
(minikube) "disk4" unmounted.
(minikube) "disk4" ejected.
(minikube) Generating 20000MB disk image...
(minikube) DBG | executing: &{/usr/bin/hdiutil [hdiutil create -megabytes 20000 -type SPARSEBUNDLE /Users/quentin/.minikube/machines/minikube/root-volume] []  <nil> 0xc42002e010 0xc42002e018 [] <nil> <nil> <nil> <nil> <nil> false [] [] [] [] <nil> <nil>} create -megabytes 20000 -type SPARSEBUNDLE /Users/quentin/.minikube/machines/minikube/root-volume
(minikube) created: /Users/quentin/.minikube/machines/minikube/root-volume.sparsebundle
(minikube) Creating SSH key...
(minikube) Fix file permission...
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/boot2docker.iso
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/config.json
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/id_rsa
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/id_rsa.pub
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/initrd.img
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/root-volume.sparsebundle
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/vmlinuz64
(minikube) Generate UUID...
(minikube) DBG | Generated UUID: 99C7DD29-AAD5-4633-9E15-4DE3F8BBCC97
(minikube) Convert UUID to MAC address...
(minikube) DBG | Converted MAC address: 4a:86:6b:3c:1a:09
(minikube) Starting minikube...
(minikube) DBG | [xhyve -A -U 99C7DD29-AAD5-4633-9E15-4DE3F8BBCC97 -c 1 -m 1024M -l com1,autopty -s 0:0,hostbridge -s 31,lpc -s 2:0,virtio-net -s 3,ahci-cd,/Users/quentin/.minikube/machines/minikube/boot2docker.iso -s 4:0,ahci-hd,/dev/rdisk4 -f kexec,/Users/quentin/.minikube/machines/minikube/vmlinuz64,/Users/quentin/.minikube/machines/minikube/initrd.img,loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 base host=minikube -F /Users/quentin/.minikube/machines/minikube/minikube.pid -s 5,virtio-9p,host=/Users]
(minikube) Waiting for VM to come online...
(minikube) DBG |
(minikube) DBG | Not there yet 1/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG | operation not supported by device
(minikube) Waiting on a pseudo-terminal to be ready... done
(minikube) Hook up your terminal emulator to /dev/ttys003 in order to connect to your VM
(minikube) DBG |
(minikube) DBG | Not there yet 2/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 3/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 4/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG | rdmsr to register 0x34 on vcpu 0
(minikube) DBG |
(minikube) DBG | Not there yet 5/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 6/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 7/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 8/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 9/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 10/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
...
...
...
(minikube) DBG |
(minikube) DBG | Not there yet 57/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 58/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 59/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 60/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
E1018 14:56:42.882671   73832 start.go:87] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: Machine didn't return an IP after 120 seconds, aborting. Retrying.
I1018 14:56:42.883546   73832 cluster.go:75] Machine exists!
Found binary path at /usr/local/bin/docker-machine-driver-xhyve
Launching plugin server for driver xhyve
Plugin server listening at address 127.0.0.1:61953
() DBG | operation not supported by device
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetState
I1018 14:56:42.898091   73832 cluster.go:82] Machine state:  Running
Waiting for SSH to be available...
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG |
Error getting ssh command 'exit 0' : IP not found for MAC  in DHCP leases
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG |
Error getting ssh command 'exit 0' : IP not found for MAC  in DHCP leases
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG |
Error getting ssh command 'exit 0' : IP not found for MAC  in DHCP leases
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG |
Error getting ssh command 'exit 0' : IP not found for MAC  in DHCP leases
...
...
...
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG |
Error getting ssh command 'exit 0' : IP not found for MAC  in DHCP leases
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG |
Error getting ssh command 'exit 0' : IP not found for MAC  in DHCP leases
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG |
Error getting ssh command 'exit 0' : IP not found for MAC  in DHCP leases
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG |
Error getting ssh command 'exit 0' : IP not found for MAC  in DHCP leases
E1018 14:59:42.986279   73832 start.go:87] Error starting host: Error configuring auth on host: %s: Too many retries waiting for SSH to be available.  Last error: Maximum number of retries (60) exceeded. Retrying.
I1018 14:59:42.986522   73832 cluster.go:75] Machine exists!
Found binary path at /usr/local/bin/docker-machine-driver-xhyve
Launching plugin server for driver xhyve
Plugin server listening at address 127.0.0.1:61978
() DBG | operation not supported by device
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetState
I1018 14:59:43.006596   73832 cluster.go:82] Machine state:  Running
Waiting for SSH to be available...
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG |
Error getting ssh command 'exit 0' : IP not found for MAC  in DHCP leases
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG |
Error getting ssh command 'exit 0' : IP not found for MAC  in DHCP leases
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG |
Error getting ssh command 'exit 0' : IP not found for MAC  in DHCP leases
...
...
...

As you can see from the shortened logs, this does not work.

What you expected to happen:

I expected cluster creation to work. What's strange is that it does work after installing Docker for Mac using brew cask install docker (and not brew install docker). But it's the same Docker version!

$ docker -v
Docker version 1.12.2, build bb80604
$ minikube delete
Deleting local Kubernetes cluster...
Machine deleted.
$ minikube start --vm-driver=xhyve --show-libmachine-logs --v=10 --alsologtostderr
I1018 15:02:49.748996   74634 notify.go:111] Checking for updates...
Starting local Kubernetes cluster...
Found binary path at /usr/local/bin/docker-machine-driver-xhyve
Launching plugin server for driver xhyve
Plugin server listening at address 127.0.0.1:62023
() DBG | operation not supported by device
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetMachineName
(minikube) Calling .DriverName
Running pre-create checks...
(minikube) Calling .PreCreateCheck
(minikube) DBG | ===== Docker Machine xhyve Driver Version 0.2.3 (Homebrew) =====
(minikube) DBG |
(minikube) DBG | executing: /usr/local/bin/VBoxManage -v
(minikube) DBG | STDOUT: 5.0.4r102546
(minikube) DBG |
(minikube) DBG | STDERR:
(minikube) Calling .GetConfigRaw
Creating machine...
(minikube) Calling .Create
(minikube) Downloading /Users/quentin/.minikube/cache/boot2docker.iso from file:///Users/quentin/.minikube/cache/iso/minikube-0.7.iso...
(minikube) Creating VM...
(minikube) Extracting vmlinuz64 and initrd.img from boot2docker.iso...
(minikube) DBG | Mounting boot2docker.iso
(minikube) DBG | executing: &{/usr/bin/hdiutil [hdiutil attach /Users/quentin/.minikube/machines/minikube/boot2docker.iso -mountpoint /Users/quentin/.minikube/machines/minikube/b2d-image] []  <nil> 0xc42008e008 0xc42008e010 [] <nil> <nil> <nil> <nil> <nil> false [] [] [] [] <nil> <nil>} attach /Users/quentin/.minikube/machines/minikube/boot2docker.iso -mountpoint /Users/quentin/.minikube/machines/minikube/b2d-image
(minikube) /dev/disk5                                           /Users/quentin/.minikube/machines/minikube/b2d-image
(minikube) DBG | Extracting vmlinuz64 into /Users/quentin/.minikube/machines/minikube
(minikube) DBG | Extracting initrd.img into /Users/quentin/.minikube/machines/minikube
(minikube) DBG | Unmounting boot2docker.iso
(minikube) DBG | executing: &{/usr/bin/hdiutil [hdiutil detach /Users/quentin/.minikube/machines/minikube/b2d-image] []  <nil> 0xc42008e008 0xc42008e010 [] <nil> <nil> <nil> <nil> <nil> false [] [] [] [] <nil> <nil>} detach /Users/quentin/.minikube/machines/minikube/b2d-image
(minikube) "disk5" unmounted.
(minikube) "disk5" ejected.
(minikube) Generating 20000MB disk image...
(minikube) DBG | executing: &{/usr/bin/hdiutil [hdiutil create -megabytes 20000 -type SPARSEBUNDLE /Users/quentin/.minikube/machines/minikube/root-volume] []  <nil> 0xc42008e008 0xc42008e010 [] <nil> <nil> <nil> <nil> <nil> false [] [] [] [] <nil> <nil>} create -megabytes 20000 -type SPARSEBUNDLE /Users/quentin/.minikube/machines/minikube/root-volume
(minikube) created: /Users/quentin/.minikube/machines/minikube/root-volume.sparsebundle
(minikube) Creating SSH key...
(minikube) Fix file permission...
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/boot2docker.iso
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/config.json
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/id_rsa
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/id_rsa.pub
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/initrd.img
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/root-volume.sparsebundle
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/vmlinuz64
(minikube) Generate UUID...
(minikube) DBG | Generated UUID: 33748FBA-E4D9-48FA-91AC-8623EC32479B
(minikube) Convert UUID to MAC address...
(minikube) DBG | Converted MAC address: 12:69:c2:1a:41:8b
(minikube) Starting minikube...
(minikube) DBG | [xhyve -A -U 33748FBA-E4D9-48FA-91AC-8623EC32479B -c 1 -m 1024M -l com1,autopty -s 0:0,hostbridge -s 31,lpc -s 2:0,virtio-net -s 3,ahci-cd,/Users/quentin/.minikube/machines/minikube/boot2docker.iso -s 4:0,ahci-hd,/dev/rdisk5 -f kexec,/Users/quentin/.minikube/machines/minikube/vmlinuz64,/Users/quentin/.minikube/machines/minikube/initrd.img,loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 base host=minikube -F /Users/quentin/.minikube/machines/minikube/minikube.pid -s 5,virtio-9p,host=/Users]
(minikube) Waiting for VM to come online...
(minikube) DBG |
(minikube) DBG | Not there yet 1/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG | operation not supported by device
(minikube) Waiting on a pseudo-terminal to be ready... done
(minikube) Hook up your terminal emulator to /dev/ttys003 in order to connect to your VM
(minikube) DBG |
(minikube) DBG | Not there yet 2/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 3/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 4/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG | rdmsr to register 0x34 on vcpu 0
(minikube) DBG |
(minikube) DBG | Not there yet 5/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 6/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 7/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 8/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 9/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 10/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 11/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 12/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 13/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 14/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG | 192.168.64.8
(minikube) DBG | IP found in DHCP lease table: 192.168.64.8
(minikube) DBG | Got an ip: 192.168.64.8
(minikube) DBG | Getting to WaitForSSH function...
(minikube) DBG | Using SSH client type: external
(minikube) DBG | {[-o BatchMode=yes -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@192.168.64.8 -o IdentitiesOnly=yes -i /Users/quentin/.minikube/machines/minikube/id_rsa -p 22] /usr/bin/ssh}
(minikube) DBG | About to run SSH command:
(minikube) DBG | exit 0
(minikube) DBG | SSH cmd err, output: <nil>:
(minikube) DBG | Using SSH client type: external
(minikube) DBG | {[-o BatchMode=yes -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@192.168.64.8 -o IdentitiesOnly=yes -i /Users/quentin/.minikube/machines/minikube/id_rsa -p 22] /usr/bin/ssh}
(minikube) DBG | About to run SSH command:
(minikube) DBG | echo -e "#/bin/bash\nsudo mkdir -p /Users\nsudo mount -t 9p -o version=9p2000 -o trans=virtio -o uname=quentin -o dfltuid=1000 -o dfltgid=50 -o access=any host /Users" | sudo tee /var/lib/boot2docker/bootlocal.sh && sudo chmod +x /var/lib/boot2docker/bootlocal.sh && /var/lib/boot2docker/bootlocal.sh
(minikube) DBG | SSH cmd err, output: <nil>: #/bin/bash
(minikube) DBG | sudo mkdir -p /Users
(minikube) DBG | sudo mount -t 9p -o version=9p2000 -o trans=virtio -o uname=quentin -o dfltuid=1000 -o dfltgid=50 -o access=any host /Users
(minikube) DBG |
(minikube) Calling .GetConfigRaw
(minikube) Calling .DriverName
(minikube) Calling .DriverName
Waiting for machine to be running, this may take a few minutes...
(minikube) Calling .GetState
Detecting operating system of created instance...
Waiting for SSH to be available...
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) Calling .GetSSHPort
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/quentin/.minikube/machines/minikube/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@192.168.64.8 -o IdentitiesOnly=yes -i /Users/quentin/.minikube/machines/minikube/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
exit 0
SSH cmd err, output: <nil>:
Detecting the provisioner...
(minikube) Calling .GetSSHHostname
(minikube) Calling .GetSSHPort
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/quentin/.minikube/machines/minikube/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@192.168.64.8 -o IdentitiesOnly=yes -i /Users/quentin/.minikube/machines/minikube/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
cat /etc/os-release
SSH cmd err, output: <nil>: NAME=Boot2Docker
VERSION=1.11.1
ID=boot2docker
ID_LIKE=tcl
VERSION_ID=1.11.1
PRETTY_NAME="Boot2Docker 1.11.1 (TCL 7.1); master : 901340f - Fri Jul  1 22:52:19 UTC 2016"
ANSI_COLOR="1;34"
HOME_URL="http://boot2docker.io"
SUPPORT_URL="https://github.com/boot2docker/boot2docker"
BUG_REPORT_URL="https://github.com/boot2docker/boot2docker/issues"

found compatible host: boot2docker
Provisioning with boot2docker...

...
...
...

(I think it would have worked with Docker Toolbox too.)

How to reproduce it (as minimally and precisely as possible):

Update: While the steps were completely reproducible for me, they no longer are, see #708 (comment)

  1. brew cask uninstall docker
  2. brew install docker
  3. minikube delete
  4. minikube start --vm-driver=xhyve --show-libmachine-logs --v=10 --alsologtostderr

Anything else do we need to know:

  • With docker, I can create a minikube but only with the virtualbox driver.
  • With xhyve + docker for mac, I can ping 8.8.8.8 and google.com successfully, but I can't curl google.com: it timeouts (separate issue but I thought I should mention it).
  • With virtualbox + docker, I can ping, and curl google.com, but minikube service is too slow to be used at shell startup: each invocation takes 1.5s while my coworkers report 0.07s on the same hardware.
@r2d4 r2d4 added kind/bug Categorizes issue or PR as related to a bug. os/macos labels Oct 18, 2016
@mboersma
Copy link

mboersma commented Oct 18, 2016

I'm having the same problem described above, as I've noted in #646. But I've tried both brew install docker and brew cask install docker and neither one works well with minikube + xhyve on my Mac.

@dlorenc
Copy link
Contributor

dlorenc commented Oct 18, 2016

@zchee do you think this is the same UUID issue you've mentioned a few other times?

@0ff
Copy link

0ff commented Oct 19, 2016

With xhyve + docker for mac, I can ping 8.8.8.8 and google.com successfully, but I can't curl google.com: it timeouts (separate issue but I thought I should mention it).

@pquentin The exact same thing happens for me - for some reason, minikube decides to use the host as the DNS Server in this setup. See the /etc/resolv.conf file in your minikube instance (i.e. minikube ssh); if you change this (or run dnsmasq on your host and bind it to your minikube's ip) it'll work just fine.

@pquentin
Copy link
Author

@0ff Changed it to 8.8.8.8 but there's no change. I don't think it was using the host ip on my network though. What's surprising too is that I can ping google.com, and this requires DNS resolution, right?

Hmm, so I just managed to create a minikube with xhyve + docker (not for mac). I think the only thing I did differently was to remove ~/.minikube. This appears to confirm what @mboersma said, Docker for Mac does not seem to actually change things here, so I'll remove that from the title.

@pquentin pquentin changed the title macOS: Can't create minikube with Docker: Docker for Mac is needed macOS: Can't create minikube with xhyve Oct 19, 2016
@zchee
Copy link
Contributor

zchee commented Oct 20, 2016

@dlorenc maybe yes.
I'll detail commnet later.

but short answer:

  1. vmnet write some macOS internal files and convert hardcoded UUID
  2. if kill -9 to xhyve-driver(just example), but not cleanup internal files
  3. After that, vmnet does not convert (or use) same UUID because exists junk(before) UUID data in macOS internal files.

@jimmidyson
Copy link
Member

The reason for the hardcoded UUID was to try to get a consistent IP across restarts. Do you think that is still going to be possible?

@zchee
Copy link
Contributor

zchee commented Oct 20, 2016

@jimmidyson Yeah, I know. But vmnet interface is not simply.
actually, I had same issue at previous. So now test and use disabled use hardcoded UUID version.(means, self build minikube binary)
maybe, it solved remove /var/db/dhcpd_leases and /Library/Preferences/SystemConfiguration/com.apple.vmnet.plist, but very risky for end-users.

Now I consider and research solution...

@zchee
Copy link
Contributor

zchee commented Oct 20, 2016

And just question.
How did you decide UUID? use uuidgen?

@jimmidyson
Copy link
Member

I didn't... @chirino did iirc.

@zchee
Copy link
Contributor

zchee commented Oct 20, 2016

@jimmidyson @dlorenc It might not relate, but macOS have original(?) UUID rules. (I think and maybe)


When I use satori/go.uuid(or etc. google/uuid is same) for generate UUID, sometimes test is fail.
It seems that according to the RFC 4122, but vmnet.framework has recognized it as invalid.

This is why I have to generate a uuid using cgo.
https://github.com/zchee/docker-machine-driver-xhyve/blob/master/xhyve/uuid.go

I think control of the UUID is should carefully.
But, as https://github.com/zchee/docker-machine-driver-xhyve#does-not-clean-up-the-vmnet-when-remove-a-vm, I don't yet understand vmnet.framework internal.

If we want to the same IP with no bugs, needs cleanup /var/db/dhcpd_leases and /Library/Preferences/SystemConfiguration/com.apple.vmnet.plist(or etc). and should be check valid(duplicate?) UUID process.


Edit: uuidgen command line source Sierra version:
http://opensource.apple.com/source/xnu/xnu-3248.60.10/libkern/uuid/uuid.c

@jimmidyson
Copy link
Member

@zchee You're far more knowledgeable than me on this! I'm not even a mac user... but I'm really grateful for your investigation as it's affecting a number of users. I'm sure we can find quite a few testers of any fixes you propose.

@r2d4
Copy link
Contributor

r2d4 commented Oct 20, 2016

@zchee the static UUID in minikube was created using uuidgen on OS X

@chirino
Copy link

chirino commented Oct 20, 2016

Note that in the original issue reported is not using the static UUID. Logging is different when the static UUID is used.

(minikube) Generate UUID...
(minikube) DBG | Generated UUID: 99C7DD29-AAD5-4633-9E15-4DE3F8BBCC97

when the static UUID is used, you will see this in the logs instead.

(minikube) Using Supplied UUID: 57FD2012-FA4A-4FF7-AEFF-26E1A1D76847

@zchee
Copy link
Contributor

zchee commented Oct 20, 2016

@jimmidyson oh, thanks :) I'll do my best.

@chirino Yes, and other issue thread have same bug.
Now I research it.

I'll create repository that sample and simple UUID convert script written Go later.

@zchee
Copy link
Contributor

zchee commented Oct 20, 2016

@r2d4 Got it. thanks.

maybe caused by vmnet internal process or conver(search) IP from dhcpd_leases.
I'll debug it.

@chirino
Copy link

chirino commented Oct 20, 2016

FYI i've run into a similar issue in the past. What I did wrong was trying to build docker-machine-driver-xhyve using plain go get. Builds fine but it does not work. You HAVE to build with make.

@zchee
Copy link
Contributor

zchee commented Oct 20, 2016

@chirino Ah, good information. I'll also check the go get plain build binary behavior.

@zchee
Copy link
Contributor

zchee commented Oct 20, 2016

@pquentin and small reply

With xhyve + docker for mac, I can ping 8.8.8.8 and google.com successfully, but I can't curl google.com: it timeouts (separate issue but I thought I should mention it).

It means use Docker for Mac only? If so, It might solve with

System Preferences -> Network -> Location -> Edit Location... -> +(create clean setting)

It might is associated as well with this problem.

@pquentin
Copy link
Author

pquentin commented Oct 21, 2016

@zchee Thanks! It does not appear to help though. I hope I did that correctly.

As I was saying, it turns out that there's no longer a difference between Docker for Mac and Docker. Maybe because I now remove ~/.minikube each time. (I also hope that it was really Docker and Docker for Mac above, because docker -v now returns something different for the two versions. I can produce the logs again if that would help.)

The minikube creation itself is no longer a problem for me: I would be happy to close this issue, but it looks like it's now bigger than me. :)

@mboersma
Copy link

mboersma commented Oct 24, 2016

You HAVE to build [docker-machine-driver-xhyve] with make

I've been doing this as @zchee suggested.

I would be happy to close this issue, but it looks like it's now bigger than me.

Unfortunately, yes, at least for me and a couple coworkers. Using xhyve and minikube v0.12.0 ends up here:

$ minikube start --logtostderr --show-libmachine-logs --vm-driver=xhyve
I1024 14:19:39.583187   22679 notify.go:111] Checking for updates...
Starting local Kubernetes cluster...
I1024 14:19:39.816177   22679 cluster.go:79] Machine exists!
I1024 14:19:39.836294   22679 cluster.go:86] Machine state:  Running
Waiting for SSH to be available...
E1024 14:22:39.968309   22679 start.go:90] Error starting host: Error configuring auth on host: %s: Temporary Error: Too many retries waiting for SSH to be available.  Last error: Maximum number of retries (60) exceeded. Retrying.
E1024 14:22:39.968418   22679 start.go:96] Error starting host:  Error configuring auth on host: %s: Temporary Error: Too many retries waiting for SSH to be available.  Last error: Maximum number of retries (60) exceeded

I've also tried with and without the "regenerate UUID" patch to the xhyve driver, but it doesn't seem to matter.

@zchee
Copy link
Contributor

zchee commented Oct 25, 2016

@mboersma Sorry for the delay.

I was some debug and research vmnet.framework internal. I understand some of the internal implementation. but still not understand causes.

However, found a good debugging way.
AFAIK, If you already disable SIP macOS feature(because not handled some error if enable), you can get a useful debug log in it.

open -a Console.app

and re-try minikube start .... and visual-grep(check with your eyes) to Console.app log...

If you are confused by many of the logs, add to search dialog.

subsystem: com.apple.network

Please try it and tell me the error log.

In my case, I was tested case of have not sudo privilege with enabling virtio-net. (use original xhyve command, not use docker-machine-driver-xhyve and minikube)

Ultimately, I found that internet sharing was processed in the

xhyve -A -U 87ADA66A-775D-4070-8DEA-B83AB3A47C60 -l com1,stdio -s 0:0,hostbridge -s 31,lpc -s 2:0,virtio-net -f 'kexec,/path/to/vmlinuz,/path/to/initrd,"earlyprintk=serial console=ttyS0 bootmem_debug debug loglevel=7'
            |
/System/Library/Frameworks/vmnet.framework
            | 
/System/Library/PrivateFrameworks/Netrb.framework

@bhundven
Copy link

I have the same output as @mboersma.

@adamreese
Copy link

@zchee
Copy link
Contributor

zchee commented Nov 5, 2016

@adamreese Thanks for detail log!
It seems failed nw_interface_create_with_name in the libsystem_network.dylib.
I'll debug it and read https://opensource.apple.com/.

@puja108
Copy link
Member

puja108 commented Feb 6, 2017

Like I already mentioned in #646 this is now solved in my system with some latest updates. Not sure which one actually fixed it, but my current versions are:

minikube version: v0.16.0
docker-machine version 0.9.0, build 15fd4c7
docker-machine-driver-xhyve: stable 0.3.1 (bottled), HEAD
xhyve: stable 0.2.0 (bottled), HEAD```

@andyp1per
Copy link

It's not working for me. In Console.app I see:

docker-machine-driver-xhyve[982] invalid client
error: aborting XPC connection to daemon
interface 0x7000600, netrb client create failed

in the minikube log:

(minikube) Convert UUID to MAC address...
E0515 14:12:41.673413 1746 start.go:119] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: Could not convert the UUID to MAC address: exit status 1.

@maduramarathe9
Copy link

Hi,

I followed these steps:

  1. brew install kubectl
  2. brew cask install minikube
  3. brew install docker-engine-driver=xhyve
  4. minikube start --vm-driver=xhyve

minikube failed to start

"
$ minikube start --vm-driver=xhyve
Starting local Kubernetes v1.7.0 cluster...
Starting VM...
Downloading Minikube ISO
97.80 MB / 97.80 MB [==============================================] 100.00% 0s
E0822 23:28:00.726776 57157 start.go:129] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: Could not convert the UUID to MAC address: exit status 1.

Retrying.
E0822 23:28:00.727478 57157 start.go:135] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: Could not convert the UUID to MAC address: exit status 1

An error has occurred. Would you like to opt in to sending anonymized crash
information to minikube to help prevent future errors?
To opt out of these messages, run the command:
minikube config set WantReportErrorPrompt false

Please enter your response [Y/n]:
y

"

Please let me know where I am going wrong.

@maduramarathe9
Copy link

rm -rf ~/.minikube
sudo chown root:wheel $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
sudo chmod u+s $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve

minikube start --vm-driver=xhyve

This helped resolve my issue:

$ minikube start --vm-driver=xhyve
Starting local Kubernetes v1.7.0 cluster...
Starting VM...
Downloading Minikube ISO
97.80 MB / 97.80 MB [==============================================] 100.00% 0s
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Starting cluster components...
Connecting to cluster...
Setting up kubeconfig...
Kubectl is now configured to use the cluster.

@pkbelidha
Copy link

$kubectl run hello-minikube --image=grc.io/google_container/echoserver:1.4 --port=8080

Error: failed to discover supported resources: Get https://192.168.99.100:8443/api: dial tcp 192.168.99.100:8443: i/o timeout

Can you please help me how to resolve this ?

@ankurkumarz
Copy link

Yes it worked for me as well.

Please follow the same order.

sudo rm -rf ~/.minikube
sudo chown root:wheel $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
sudo chmod u+s $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
minikube start --vm-driver=xhyve

@joeljohnston
Copy link

I'm having this same problem with both minikube and docker-machine using the docker-machine-driver-xhyve 0.33 release. The problem began happening after i upgraded from minikube 0.25.0 to minikube 0.25.2. The solution above does not resolve the issue. I've emptied the dhcpd_leases file and ensured permissions are set appropriately to no avail. Please help. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
co/xhyve kind/bug Categorizes issue or PR as related to a bug. os/macos
Projects
None yet
Development

No branches or pull requests