SSH fails infinitely with minikube start --vm-driver=xhyve on macOS Sierra 10.12.6

[ligerzero459]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Minikube version (use minikube version):
minikube version: v0.21.0

Environment:

• OS (e.g. from /etc/os-release): macOS Sierra 10.12.6
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName):
  "DriverName": "xhyve"
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION):
  "Boot2DockerURL": "file:///Users/(username)/.minikube/cache/iso/minikube-v0.23.0.iso"
• Install tools:

1. Homebrew 1.3.1

• Others:

What happened:
When starting the VM for the first time after installing Minikube, the VM builds, boots and then repeats the following infinitely after running minikube start --vm-driver xhyve --v=10

(minikube) DBG | Getting to WaitForSSH function...
(minikube) DBG | Using SSH client type: external
(minikube) DBG | Using SSH private key: /Users/(username)/.minikube/machines/minikube/id_rsa (-rw-------)
(minikube) DBG | &{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@192.168.64.13 -o IdentitiesOnly=yes -i /Users/(username)/.minikube/machines/minikube/id_rsa -p 22] /usr/bin/ssh <nil>}
(minikube) DBG | About to run SSH command:
(minikube) DBG | exit 0
(minikube) DBG | SSH cmd err, output: exit status 255: 
(minikube) DBG | Error getting ssh command 'exit 0' : Something went wrong running an SSH command!
(minikube) DBG | command : exit 0
(minikube) DBG | err     : exit status 255
(minikube) DBG | output  : 
(minikube) DBG | 

How to reproduce it (as minimally and precisely as possible):
Install xhyve using the following:

$ brew install docker-machine-driver-xhyve
$ sudo chown root:wheel $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
$ sudo chmod u+s $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve

Install minikube from scratch for the first time using the guide at https://github.com/kubernetes/minikube/releases/tag/v0.21.0

Output of minikube logs (if applicable):

Starting local Kubernetes v1.7.0 cluster...
Starting VM...
Found binary path at /usr/local/bin/docker-machine-driver-xhyve
Launching plugin server for driver xhyve
Plugin server listening at address 127.0.0.1:50654
() DBG | operation not supported by device
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetMachineName
(minikube) Calling .DriverName
(minikube) Calling .DriverName
Running pre-create checks...
(minikube) Calling .PreCreateCheck
(minikube) DBG | ===== Docker Machine xhyve Driver Version 0.3.3 (Homebrew7d92f74) =====
(minikube) DBG | 
(minikube) Calling .GetConfigRaw
Creating machine...
(minikube) Calling .Create
(minikube) Downloading /Users/(username)/.minikube/cache/boot2docker.iso from file:///Users/(username)/.minikube/cache/iso/minikube-v0.23.0.iso...
(minikube) Creating VM...
(minikube) DBG | Mounting boot2docker.iso
(minikube) DBG | executing: &{/usr/bin/hdiutil [hdiutil attach /Users/(username)/.minikube/machines/minikube/boot2docker.iso -mountpoint /Users/(username)/.minikube/machines/minikube/b2d-image] []  <nil> 0xc420086008 0xc420086010 [] <nil> <nil> <nil> <nil> <nil> false [] [] [] [] <nil> <nil>} attach /Users/(username)/.minikube/machines/minikube/boot2docker.iso -mountpoint /Users/(username)/.minikube/machines/minikube/b2d-image
(minikube) /dev/disk13         	                               	/Users/(username)/.minikube/machines/minikube/b2d-image
(minikube) DBG | Extracting Kernel Options...
(minikube) DBG | Extracted Options "loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes base host=minikube"
(minikube) DBG | Extracting /Users/(username)/.minikube/machines/minikube/b2d-image/boot/bzImage into /Users/(username)/.minikube/machines/minikube/bzImage
(minikube) DBG | Extracting /Users/(username)/.minikube/machines/minikube/b2d-image/boot/initrd into /Users/(username)/.minikube/machines/minikube/initrd
(minikube) DBG | Unmounting boot2docker.iso
(minikube) DBG | executing: &{/usr/bin/hdiutil [hdiutil detach /Users/(username)/.minikube/machines/minikube/b2d-image] []  <nil> 0xc420086008 0xc420086010 [] <nil> <nil> <nil> <nil> <nil> false [] [] [] [] <nil> <nil>} detach /Users/(username)/.minikube/machines/minikube/b2d-image
(minikube) "disk13" unmounted.
(minikube) "disk13" ejected.
(minikube) Generating 20000MB disk image...
(minikube) DBG | executing: &{/usr/bin/hdiutil [hdiutil create -megabytes 20000 -type SPARSEBUNDLE /Users/(username)/.minikube/machines/minikube/root-volume] []  <nil> 0xc420086008 0xc420086010 [] <nil> <nil> <nil> <nil> <nil> false [] [] [] [] <nil> <nil>} create -megabytes 20000 -type SPARSEBUNDLE /Users/(username)/.minikube/machines/minikube/root-volume
(minikube) created: /Users/(username)/.minikube/machines/minikube/root-volume.sparsebundle
(minikube) Creating SSH key...
(minikube) Fix file permission...
(minikube) DBG | /Users/(username)/.minikube/machines/minikube/boot2docker.iso
(minikube) DBG | /Users/(username)/.minikube/machines/minikube/bzImage
(minikube) DBG | /Users/(username)/.minikube/machines/minikube/config.json
(minikube) DBG | /Users/(username)/.minikube/machines/minikube/id_rsa
(minikube) DBG | /Users/(username)/.minikube/machines/minikube/id_rsa.pub
(minikube) DBG | /Users/(username)/.minikube/machines/minikube/initrd
(minikube) DBG | /Users/(username)/.minikube/machines/minikube/root-volume.sparsebundle
(minikube) Generate UUID...
(minikube) DBG | Generated UUID: 48CEAC0E-92FB-403B-8C21-5B5FE8C52573
(minikube) Convert UUID to MAC address...
(minikube) DBG | Running command: /usr/local/bin/docker-machine-driver-xhyve [xhyve -A -U 48CEAC0E-92FB-403B-8C21-5B5FE8C52573 -c 2 -m 2048M -l com1,autopty -s 0:0,hostbridge -s 31,lpc -s 2:0,virtio-net -s 3:0,ahci-cd,/Users/(username)/.minikube/machines/minikube/boot2docker.iso -s 4:0,ahci-hd,/dev/rdisk13 -f kexec,/Users/(username)/.minikube/machines/minikube/bzImage,/Users/(username)/.minikube/machines/minikube/initrd,loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes base host=minikube -M]
(minikube) DBG | Converted MAC address: ba:54:cf:89:aa:ee
(minikube) Starting minikube...
(minikube) DBG | [xhyve -A -U 48CEAC0E-92FB-403B-8C21-5B5FE8C52573 -c 2 -m 2048M -l com1,autopty -s 0:0,hostbridge -s 31,lpc -s 2:0,virtio-net -s 3:0,ahci-cd,/Users/(username)/.minikube/machines/minikube/boot2docker.iso -s 4:0,ahci-hd,/dev/rdisk13 -f kexec,/Users/(username)/.minikube/machines/minikube/bzImage,/Users/(username)/.minikube/machines/minikube/initrd,loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes base host=minikube -F /Users/(username)/.minikube/machines/minikube/minikube.pid -s 5,virtio-9p,host=/Users]
(minikube) Waiting for VM to come online...
(minikube) DBG | 
(minikube) DBG | Not there yet 1/60, error: IP not found for MAC ba:54:cf:89:aa:ee in DHCP leases
(minikube) DBG | operation not supported by device
(minikube) Waiting on a pseudo-terminal to be ready... done
(minikube) Hook up your terminal emulator to /dev/ttys006 in order to connect to your VM
(minikube) DBG | 
(minikube) DBG | Not there yet 2/60, error: IP not found for MAC ba:54:cf:89:aa:ee in DHCP leases
(minikube) DBG | 
(minikube) DBG | Not there yet 3/60, error: IP not found for MAC ba:54:cf:89:aa:ee in DHCP leases
(minikube) DBG | 
(minikube) DBG | Not there yet 4/60, error: IP not found for MAC ba:54:cf:89:aa:ee in DHCP leases
(minikube) DBG | 
(minikube) DBG | Not there yet 5/60, error: IP not found for MAC ba:54:cf:89:aa:ee in DHCP leases
(minikube) DBG | rdmsr to register 0x64e on vcpu 1
(minikube) DBG | rdmsr to register 0x34 on vcpu 1
(minikube) DBG | 
(minikube) DBG | Not there yet 6/60, error: IP not found for MAC ba:54:cf:89:aa:ee in DHCP leases
(minikube) DBG | 192.168.64.13
(minikube) DBG | IP found in DHCP lease table: 192.168.64.13
(minikube) DBG | Got an ip: 192.168.64.13
(minikube) DBG | Getting to WaitForSSH function...
(minikube) DBG | Using SSH client type: external
(minikube) DBG | Using SSH private key: /Users/(username)/.minikube/machines/minikube/id_rsa (-rw-------)
(minikube) DBG | &{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@192.168.64.13 -o IdentitiesOnly=yes -i /Users/(username)/.minikube/machines/minikube/id_rsa -p 22] /usr/bin/ssh <nil>}
(minikube) DBG | About to run SSH command:
(minikube) DBG | exit 0
(minikube) DBG | SSH cmd err, output: exit status 255: 
(minikube) DBG | Error getting ssh command 'exit 0' : Something went wrong running an SSH command!
(minikube) DBG | command : exit 0
(minikube) DBG | err     : exit status 255
(minikube) DBG | output  : 
(minikube) DBG | 
(Repeat Infinitely)

Anything else do we need to know:
xhyve Driver v.0.3.3
Homebrew 1.3.1
docker-machine version 0.12.0, build 45c69ad
minikube version: v0.21.0

[r2d4]

This should be fixed at HEAD, where we are defaulting to the native golang SSH implementation. Do you have a different ssh implementation on your path?

[ligerzero459]

Apologies, I thought I'd replied when I first saw this.

My current PATH is this: /usr/local/bin/google-cloud-sdk/bin /usr/local/bin /usr/bin /bin /usr/sbin /sbin /opt/X11/bin /usr/local/go/bin /Library/Frameworks/Mono.framework/Versions/Current/Commands

According to what I can tell, I should be using the default ssh from /usr/bin

[Maixy]

Still seeing this issue w/ the v0.23.0 release - is this something that can be circumvented manually w/ a default ssh path reassignment?

[hgl]

Same with 0.23.0, keep getting this error

(minikube) DBG | Getting to WaitForSSH function...
(minikube) DBG | Using SSH client type: external
(minikube) DBG | Using SSH private key: /Users/name/.minikube/machines/minikube/id_rsa (-rw-------)
(minikube) DBG | &{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@192.168.64.7 -o IdentitiesOnly=yes -i /Users/name/.minikube/machines/minikube/id_rsa -p 22] /usr/bin/ssh <nil>}
(minikube) DBG | About to run SSH command:
(minikube) DBG | exit 0
(minikube) DBG | SSH cmd err, output: exit status 255: 
(minikube) DBG | Error getting ssh command 'exit 0' : Something went wrong running an SSH command!
(minikube) DBG | command : exit 0
(minikube) DBG | err     : exit status 255
(minikube) DBG | output  : 

macOS High Sierra

[hgl]

It seems minikube is unusable on macOS right now? Is there any workaround?

[peanutgyz]

same error

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close