Incompatible hyperkit driver causes "Waiting for SSH access ..." hang after macOS restart

[erichartline]

After rebooting my computer, any attempts to restart my Minikube cluster do not work. I get the following output:

😄  minikube v0.35.0 on darwin (amd64)
💡  Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
🔄  Restarting existing hyperkit VM for "minikube" ...
⌛  Waiting for SSH access ...

And it just sits there, never even timing out no matter how long I wait.

How to replicate:

minikube start --cpus=4 --memory=4096 --disk-size=30g --vm-driver=hyperkit --kubernetes-version=v1.10.12 --extra-config=apiserver.authorization-mode=RBAC

I am using macOS Mojave Version 10.14.3. Here's my output from minikube logs:

💣  command runner: getting ssh client for bootstrapper: Error dialing tcp via ssh client: dial tcp 192.168.64.11:22: connect: operation timed out

😿  Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉  https://github.com/kubernetes/minikube/issues/new

[tstromberg]

Thank you for the bug report! Sorry that minikube didn't work for you. I tested your command-line on macOS 10.4.3, but wasn't able to reproduce it. This could be a few different things:

• The ISO not coming up at the IP that hyperkit thinks it should be at
• Hyperkit failing to route packets to the VM
• A firewall blocking packets to the VM

Do you mind supplying a few commands so that we can help figure out what's going on?

• Does starting a new minikube profile work, as in:

minikube start --cpus=4 --memory=4096 --disk-size=30g --vm-driver=hyperkit --kubernetes-version=v1.10.12 --extra-config=apiserver.authorization-mode=RBAC -p new

If so, I'm curious what version of the ISO your existing build has. You can find out by running:

grep MinikubeISO ~/.minikube/profiles/minikube/config.json

If neither work, do you mind running the following commands:

ping -v <IP reported by minikube>
route get <IP reported by minikube>
ls -la /usr/local/bin/docker-machine-driver-hyperkit
md5 /usr/local/bin/docker-machine-driver-hyperkit

Thanks!

[erichartline]

Hi, thanks for the quick response!

Starting a new minikube profile works fine. The only issue is when I shut off my computer, then attempt to restart an existing cluster next time.

The output from the grep command is

"MinikubeISO": "https://storage.googleapis.com/minikube/iso/minikube-v0.35.0.iso",`

[tstromberg]

OK. That sounds like a bug with the hyperkit driver. Do you mind running this for me?

% ls -lad $(type -a docker-machine-driver-hyperkit)

Assuming it doesn't say 27779836 bytes long, I'm curious if upgrading to the latest driver version fixes the situation:

% curl -LO https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-hyperkit \
&& sudo install -o root -g wheel -m 4755 docker-machine-driver-hyperkit /usr/local/bin/

If this doesn't fix it, do you mind adding the output of minikube start --alsologtostderr -v=8 after a reboot?

We have a long-standing issue where we don't have control over which hyperkit driver is installed on machines, and I'm hoping this is the evidence we need to finally fix the issue.

[erichartline]

Yes - upgrading to the latest driver version seems to have solved my problem! I'll test it out again on my next reboot and report back.

[erichartline]

Still working after my latest reboot. Looks like that did the trick. Thanks for your help!

[tstromberg]

Thanks! I'm going to keep this open for others who run into this issue, at least until we have a way to always guarantee the appropriate hyperkit version.

[shishir139]

I ran into the same issue and this can be solved by a simple command below:

minikube delete && minikube start

Hope that solves problems of many people!!

[erichartline]

I ran into the same issue and this can be solved by a simple command below:

minikube delete && minikube start

Hope that solves problems of many people!!

That works, but it's not really a viable solution when you have a lot of things in your cluster. For us, it took a long time to reinstall everything in the cluster so having to delete it every time was a huge problem.

[mightyguava]

I think I'm running into the same issue, and it's the same as #3542. I haven't been able to solve it with minikube delete && minikube start. minikube is just hanging completely.

[repetty]

I've run into this problem, too.

MacOS 10.14.4 (up-to-date, as of today)
Virtualbox 6.0.4 r128413 (up-to-date, as of today)

minikube delete && minikube start   didn't fix the problem.

curl -LO https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-hyperkit
&& sudo install -o root -g wheel -m 4755 docker-machine-driver-hyperkit /usr/local/bin/   didn't work although perhaps there's some maneuver that I should employ which I haven't.

[prasadram]

Even i am facing similar problem , but it didn't resolved with any of above solutions. I have tried by installing Virtual box with that also it didn't worked. below is the response when i triggered with Virtualbox

 minikube v0.34.1 on darwin (amd64)

🔥 Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
💣 Unable to start VM: create: creating: write |1: broken pipe

😿 Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉 https://github.com/kubernetes/minikube/issues/new

Basically i have tried with xhvye, hyperkit, VirtualBox but none of these didn't worked
please let me know if any one knows how to fix this.
Mac OS High Sierra : 10.13.6

[lazeyliu]

i just do this, and it works well

rm -rf ~/.minikube/machines/minikube/hyperkit.pid

minikube start

[tstromberg]

Marking as fixed in v1.0, so long as the appropriately up to date hyperkit driver has been installed:

curl -LO https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-hyperkit && sudo install -o root -g wheel -m 4755 docker-machine-driver-hyperkit /usr/local/bin/

[monomasmalo]

hi all, just a quick follow-up from my experience that points to this not being fully resolved in v1.0.

i'm running:
minikube v1.1.0
hyperkit v0.20180403-17-g3e954c (latest available via homebrew as of 23 May 2019)

when minikube is shut down cleanly (i.e. minikube stop) it restarts fine. whenever it is not shut down cleanly (i.e. shutting down terminal for a forced reboot), minikube start hangs at the Waiting for SSH access ... step. performing the manual cleanup step suggested by @lazeyliu resolves the issue.

perhaps some cleanup checks need to be added a start up.

[jyobeer]

I am facing the same issue. can anyone help please. I have already tried the above solution for installing the latest hyperkit but still the issue persists.

minikube version - v1.9.2
OS version - Ubuntu 16.04