Minikube tunnel hangs on Ubuntu 20 in Windows 10

[johnnybigert]

Steps to reproduce the issue:

1. minikube start
2. kubectl apply -f manifests/storage-deployment.yaml
3. kubectl apply -f manifests/storage-service.yaml
4. minikube tunnel

Full output of minikube logs command:
out.txt

Full output of failed command:
Observed behavior: The minikube tunnel command provides no output and appears to be frozen. (However, using htop I can see that the tunnel command is using cpu and memory. Also, if I do minikube stop in another terminal, the tunnel command starts outputting error messages.)
Expected behavior: Output of service IP addresses etc. like the output in this example.

Note that the Ubuntu 20.04 is running in Windows 10 (WSL2).

[spowelljr]

Hi @johnnybigert, thanks for reporting your issue with minikube!

There's a large discussion about running minikube on WSL2 on this issue #7879.

There are multiple comments discussing the tunnel command, I'd recommend reading through them and seeing if any of them are applicable to you and maybe solve your issue, thanks!

[johnnybigert]

@spowelljr Thanks for responding. I read #7879, nothing helpful there what I could see, just @ashishsecdev having the same issue as me.

[spowelljr]

@johnnybigert Thanks for checking out that issue, I'll see if I have time to check it out today. If you could though, try running minikube tunnel --alsologtostderr that should give you a more verbose output and possibly see where the problem lies.

[johnnybigert]

Nice @spowelljr, got some more info:

$ minikube tunnel --alsologtostderr I0810 16:20:47.832318 42964 out.go:286] Setting OutFile to fd 1 ... I0810 16:20:47.833475 42964 out.go:338] isatty.IsTerminal(1) = true I0810 16:20:47.833541 42964 out.go:299] Setting ErrFile to fd 2... I0810 16:20:47.834013 42964 out.go:338] isatty.IsTerminal(2) = true I0810 16:20:47.834427 42964 root.go:312] Updating PATH: /home/johnny/.minikube/bin I0810 16:20:47.836683 42964 mustload.go:65] Loading cluster: minikube I0810 16:20:47.838820 42964 cli_runner.go:115] Run: docker container inspect minikube --format={{.State.Status}} I0810 16:20:48.061876 42964 host.go:66] Checking if "minikube" exists ... I0810 16:20:48.062366 42964 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "8443/tcp") 0).HostPort}}'" minikube I0810 16:20:48.250879 42964 api_server.go:164] Checking apiserver status ... I0810 16:20:48.251030 42964 ssh_runner.go:149] Run: sudo pgrep -xnf kube-apiserver.*minikube.* I0810 16:20:48.251212 42964 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube I0810 16:20:48.443480 42964 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:65391 SSHKeyPath:/home/johnny/.minikube/machines/minikube/id_rsa Username:docker} I0810 16:20:48.604169 42964 ssh_runner.go:149] Run: sudo egrep ^[0-9]+:freezer: /proc/2229/cgroup I0810 16:20:48.616791 42964 api_server.go:180] apiserver freezer: "7:freezer:/docker/504ab5c5ef196d5a3224239f6d9c3fcd9260f9d769071c0ef37fc7d95b970bc7/kubepods/burstable/podcefbe66f503bf010430ec3521cf31be8/4aa3af29e94614945e7ae4185c8028c567c6e5561d7042dd0100083b2b1cf6b4" I0810 16:20:48.616910 42964 ssh_runner.go:149] Run: sudo cat /sys/fs/cgroup/freezer/docker/504ab5c5ef196d5a3224239f6d9c3fcd9260f9d769071c0ef37fc7d95b970bc7/kubepods/burstable/podcefbe66f503bf010430ec3521cf31be8/4aa3af29e94614945e7ae4185c8028c567c6e5561d7042dd0100083b2b1cf6b4/freezer.state I0810 16:20:48.628164 42964 api_server.go:202] freezer state: "THAWED" I0810 16:20:48.628233 42964 api_server.go:239] Checking apiserver healthz at https://127.0.0.1:65390/healthz ... I0810 16:20:48.638548 42964 api_server.go:265] https://127.0.0.1:65390/healthz returned 200: ok I0810 16:20:48.638635 42964 tunnel.go:57] Checking for tunnels to cleanup... I0810 16:20:48.648585 42964 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube

...and this is where it freezes. (Not sure why the newlines gets removed, sorry about the poor readability.)

[pfeigl]

I'm having the very same problem. Here's my log if it helps, but it looks very similar to that of @johnnybigert

I0831 18:22:04.551095    3328 out.go:286] Setting OutFile to fd 88 ...
I0831 18:22:04.575864    3328 out.go:333] TERM=,COLORTERM=, which probably does not support color
I0831 18:22:04.575864    3328 out.go:299] Setting ErrFile to fd 92...
I0831 18:22:04.575864    3328 out.go:333] TERM=,COLORTERM=, which probably does not support color
I0831 18:22:04.591706    3328 mustload.go:65] Loading cluster: minikube
I0831 18:22:04.623706    3328 cli_runner.go:115] Run: docker container inspect minikube --format={{.State.Status}}
I0831 18:22:05.787724    3328 cli_runner.go:168] Completed: docker container inspect minikube --format={{.State.Status}}: (1.1639982s)
I0831 18:22:05.787724    3328 host.go:66] Checking if "minikube" exists ...
I0831 18:22:05.803656    3328 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "8443/tcp") 0).HostPort}}'" minikube
I0831 18:22:06.142604    3328 api_server.go:164] Checking apiserver status ...
I0831 18:22:06.169184    3328 ssh_runner.go:149] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0831 18:22:06.183977    3328 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0831 18:22:06.525544    3328 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:52996 SSHKeyPath:C:\Users\philipp.feigl\.minikube\machines\minikube\id_rsa Username:docker}
I0831 18:22:06.688352    3328 ssh_runner.go:149] Run: sudo egrep ^[0-9]+:freezer: /proc/2083/cgroup
I0831 18:22:06.699709    3328 api_server.go:180] apiserver freezer: "20:freezer:/docker/563f5ed6dc71139f783d154e8b6d83c6d3e88b4cdf35c12882ab51dee0ab7373/kubepods/burstable/podcefbe66f503bf010430ec3521cf31be8/26a2826f49d8b2eeda603e849abc41548dc757aab1fd6a03c9d2087f6118c318"
I0831 18:22:06.727813    3328 ssh_runner.go:149] Run: sudo cat /sys/fs/cgroup/freezer/docker/563f5ed6dc71139f783d154e8b6d83c6d3e88b4cdf35c12882ab51dee0ab7373/kubepods/burstable/podcefbe66f503bf010430ec3521cf31be8/26a2826f49d8b2eeda603e849abc41548dc757aab1fd6a03c9d2087f6118c318/freezer.state
I0831 18:22:06.740717    3328 api_server.go:202] freezer state: "THAWED"
I0831 18:22:06.740717    3328 api_server.go:239] Checking apiserver healthz at https://127.0.0.1:53000/healthz ...
I0831 18:22:06.749297    3328 api_server.go:265] https://127.0.0.1:53000/healthz returned 200: ok
I0831 18:22:06.749297    3328 tunnel.go:57] Checking for tunnels to cleanup...
I0831 18:22:06.774848    3328 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube

And at this point, the tunnel just sits there and no tunneling works and also no user input is requested. I also tried running the tunnel in an admin-console, but to no avail.

[pfeigl]

Oh I think I found something. Running that last command directly in the bash gives the following error:

C:\WINDOWS\system32>docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
Template parsing error: template: :1: unexpected "/" in operand

Looking at the command, it seems like there is a problem with " beeing inside of the actual inspect string.

I just digged a little deeper. It feels like this is the core of the problem:
docker/cli#3241

Sofar I couldn't get the command to run, even after fixing the quotes, it still errors out on an unexpted / with the following error

PS C:\Users\philipp.feigl> docker inspect --format '{{ (index (index .NetworkSettings.Ports "22/tcp") 0).HostPort }}' minikube
Template parsing error: template: :1: unexpected "/" in operand

[pfeigl]

Ok, some more digging, the correct command seems to be the following

docker container inspect -f '{{(index (index .NetworkSettings.Ports \"22/tcp\") 0).HostPort}}' minikube

The following changes where needed:

• Removed outside double quotes
• Escaped double quotes in filter

With that the actual port was reported back as expected

The line that probably needs to be changed is this:

minikube/pkg/drivers/kic/oci/network.go

Line 179 in 4f005bd

rr, err = runCmd(exec.Command(ociBin, "container", "inspect", "-f", fmt.Sprintf("'{{(index (index .NetworkSettings.Ports \"%d/tcp\") 0).HostPort}}'", contPort), ociID))

What I'm not sure because I'm currently not having an old docker installation available is, if we can simply add the escaping of the quotes and it'll still work with older versions and also newer version or whether we have to check the docker version and have different implementations.

[pgschr]

@pfeigl you fixed the problem with your change? did you recompile the executable?

[pfeigl]

No, tbh I just analyzed the problem and tested the fixed command line command on the bash directly. I never changed / compiled the minikube code.

[speedyankur]

Also facing the same issue

[thanili]

Any updates on that? I am also facing the same issue :/