Incompatible hyperkit driver causes "Waiting for SSH access ..." hang after macOS restart

[erichartline]

After rebooting my computer, any attempts to restart my Minikube cluster do not work. I get the following output:

😄  minikube v0.35.0 on darwin (amd64)
💡  Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
🔄  Restarting existing hyperkit VM for "minikube" ...
⌛  Waiting for SSH access ...

And it just sits there, never even timing out no matter how long I wait.

How to replicate:

minikube start --cpus=4 --memory=4096 --disk-size=30g --vm-driver=hyperkit --kubernetes-version=v1.10.12 --extra-config=apiserver.authorization-mode=RBAC

I am using macOS Mojave Version 10.14.3. Here's my output from minikube logs:

💣  command runner: getting ssh client for bootstrapper: Error dialing tcp via ssh client: dial tcp 192.168.64.11:22: connect: operation timed out

😿  Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉  https://github.com/kubernetes/minikube/issues/new

[tstromberg]

Thank you for the bug report! Sorry that minikube didn't work for you. I tested your command-line on macOS 10.4.3, but wasn't able to reproduce it. This could be a few different things:

• The ISO not coming up at the IP that hyperkit thinks it should be at
• Hyperkit failing to route packets to the VM
• A firewall blocking packets to the VM

Do you mind supplying a few commands so that we can help figure out what's going on?

• Does starting a new minikube profile work, as in:

minikube start --cpus=4 --memory=4096 --disk-size=30g --vm-driver=hyperkit --kubernetes-version=v1.10.12 --extra-config=apiserver.authorization-mode=RBAC -p new

If so, I'm curious what version of the ISO your existing build has. You can find out by running:

grep MinikubeISO ~/.minikube/profiles/minikube/config.json

If neither work, do you mind running the following commands:

ping -v <IP reported by minikube>
route get <IP reported by minikube>
ls -la /usr/local/bin/docker-machine-driver-hyperkit
md5 /usr/local/bin/docker-machine-driver-hyperkit

Thanks!

[erichartline]

Hi, thanks for the quick response!

Starting a new minikube profile works fine. The only issue is when I shut off my computer, then attempt to restart an existing cluster next time.

The output from the grep command is

"MinikubeISO": "https://storage.googleapis.com/minikube/iso/minikube-v0.35.0.iso",`

[tstromberg]

OK. That sounds like a bug with the hyperkit driver. Do you mind running this for me?

% ls -lad $(type -a docker-machine-driver-hyperkit)

Assuming it doesn't say 27779836 bytes long, I'm curious if upgrading to the latest driver version fixes the situation:

% curl -LO https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-hyperkit \
&& sudo install -o root -g wheel -m 4755 docker-machine-driver-hyperkit /usr/local/bin/

If this doesn't fix it, do you mind adding the output of minikube start --alsologtostderr -v=8 after a reboot?

We have a long-standing issue where we don't have control over which hyperkit driver is installed on machines, and I'm hoping this is the evidence we need to finally fix the issue.

[erichartline]

Yes - upgrading to the latest driver version seems to have solved my problem! I'll test it out again on my next reboot and report back.

[erichartline]

Still working after my latest reboot. Looks like that did the trick. Thanks for your help!

[tstromberg]

Thanks! I'm going to keep this open for others who run into this issue, at least until we have a way to always guarantee the appropriate hyperkit version.

[shishir139]

I ran into the same issue and this can be solved by a simple command below:

minikube delete && minikube start

Hope that solves problems of many people!!

[erichartline]

I ran into the same issue and this can be solved by a simple command below:

minikube delete && minikube start

Hope that solves problems of many people!!

That works, but it's not really a viable solution when you have a lot of things in your cluster. For us, it took a long time to reinstall everything in the cluster so having to delete it every time was a huge problem.