Running on WSL2 via ssh driver and docker runtime

[Spenhouet]

Steps to reproduce the issue:

I want to use minikube on my laptop and run everything on another Windows system's WSL2.

minikube start --driver=ssh --ssh-ip-address=10.196.0.103 -p aira-208

I did set up ssh access to that other Windows system following these two guides:

• https://www.hanselman.com/blog/how-to-ssh-into-a-windows-10-machine-from-linux-or-windows-or-anywhere
• https://www.concurrency.com/blog/may-2019/key-based-authentication-for-openssh-on-windows

I did set the default console so that the ssh connection will open to the WSL.

If I access the system via

ssh spe@10.196.0.103

I get directly connected to the WSL console of the Windows system (without any password needed).
I did expect minikube to then directly execute / work on the WSL system.

The WSL system has a docker runtime.

Full output of failed command:

ssh connection errors:

....
I0603 18:10:05.673271   11660 retry.go:31] will retry after 276.165072ms: new client: new client: Error creating new ssh host from driver: Error getting ssh host name for driver: IP address is not set
....
I0603 18:10:09.781665   11660 main.go:126] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none], no supported methods remain
....

I0603 18:10:05.558173 11660 out.go:278] Setting OutFile to fd 92 ... I0603 18:10:05.609253 11660 out.go:291] Setting ErrFile to fd 96... I0603 18:10:05.626464 11660 out.go:285] Setting JSON to false I0603 18:10:05.633263 11660 start.go:108] hostinfo: {"hostname":"spen","uptime":86003,"bootTime":1622650602,"procs":270,"os":"windows","platform":"Microsoft Windows 10 Home","platformFamily":"Standalone Workstation","platformVersion":"10.0.19043 Build 19043","kernelVersion":"10.0.19043 Build 19043","kernelArch":"x86_64","virtualizationSystem":"","virtualizationRole":"","hostId":"e9d2dde5-0ceb-4f17-9120-97c918de5f93"} W0603 18:10:05.633263 11660 start.go:116] gopshost.Virtualization returned error: not implemented yet I0603 18:10:05.644068 11660 out.go:157] 😄 [aira-208] minikube v1.19.0 auf Microsoft Windows 10 Home 10.0.19043 Build 19043 😄 [aira-208] minikube v1.19.0 auf Microsoft Windows 10 Home 10.0.19043 Build 19043 I0603 18:10:05.673271 11660 ssh_runner.go:149] Run: systemctl --version I0603 18:10:05.673271 11660 retry.go:31] will retry after 276.165072ms: new client: new client: Error creating new ssh host from driver: Error getting ssh host name for driver: IP address is not set I0603 18:10:05.960154 11660 retry.go:31] will retry after 540.190908ms: new client: new client: Error creating new ssh host from driver: Error getting ssh host name for driver: IP address is not set I0603 18:10:06.507837 11660 retry.go:31] will retry after 655.06503ms: new client: new client: Error creating new ssh host from driver: Error getting ssh host name for driver: IP address is not set I0603 18:10:07.172370 11660 retry.go:31] will retry after 791.196345ms: new client: new client: Error creating new ssh host from driver: Error getting ssh host name for driver: IP address is not set I0603 18:10:07.964762 11660 driver.go:322] Setting default libvirt URI to qemu:///system I0603 18:10:07.994314 11660 out.go:157] ✨ Using the ssh driver based on existing profile ✨ Using the ssh driver based on existing profile I0603 18:10:07.994500 11660 start.go:276] selected driver: ssh I0603 18:10:07.995044 11660 start.go:718] validating driver "ssh" against &{Name:aira-208 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.20@sha256:0250dab3644403384bd54f566921c6b57138eecffbb861f9392feef9b2ec44f6 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:ssh HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress:10.196.0.103 SSHUser:spe SSHKey:ssh-rsa ***** SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.20.2 ClusterName:aira-208 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.20.2 ControlPlane:true Worker:true}] Addons:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop: ExposedPorts:[] ListenAddress: Network: MultiNodeRequested:false} I0603 18:10:07.995711 11660 start.go:729] status for ssh: {Installed:true Healthy:true Running:false NeedsImprovement:false Error: Reason: Fix: Doc:} I0603 18:10:09.395514 11660 start_flags.go:270] config: {Name:aira-208 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.20@sha256:0250dab3644403384bd54f566921c6b57138eecffbb861f9392feef9b2ec44f6 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:ssh HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress:10.196.0.103 SSHUser:spe SSHKey:ssh-rsa ***** SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.20.2 ClusterName:aira-208 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.20.2 ControlPlane:true Worker:true}] Addons:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop: ExposedPorts:[] ListenAddress: Network: MultiNodeRequested:false} I0603 18:10:09.423928 11660 out.go:157] 👍 Starting control plane node aira-208 in cluster aira-208 👍 Starting control plane node aira-208 in cluster aira-208 I0603 18:10:09.424791 11660 preload.go:97] Checking if preload exists for k8s version v1.20.2 and runtime docker I0603 18:10:09.425325 11660 preload.go:105] Found local preload: C:\Users\spenh\.minikube\cache\preloaded-tarball\preloaded-images-k8s-v10-v1.20.2-docker-overlay2-amd64.tar.lz4 I0603 18:10:09.425325 11660 cache.go:54] Caching tarball of preloaded images I0603 18:10:09.425861 11660 preload.go:131] Found C:\Users\spenh\.minikube\cache\preloaded-tarball\preloaded-images-k8s-v10-v1.20.2-docker-overlay2-amd64.tar.lz4 in cache, skipping download I0603 18:10:09.425861 11660 cache.go:57] Finished verifying existence of preloaded tar for v1.20.2 on docker I0603 18:10:09.426391 11660 profile.go:148] Saving config to C:\Users\spenh\.minikube\profiles\aira-208\config.json ... I0603 18:10:09.428896 11660 cache.go:185] Successfully downloaded all kic artifacts I0603 18:10:09.429956 11660 start.go:313] acquiring machines lock for aira-208: {Name:mke84f9ca55575557f14736813d56e85bcb6ee42 Clock:{} Delay:500ms Timeout:13m0s Cancel:} I0603 18:10:09.430648 11660 start.go:317] acquired machines lock for "aira-208" in 0s I0603 18:10:09.431238 11660 start.go:93] Skipping create...Using existing machine configuration I0603 18:10:09.431754 11660 fix.go:55] fixHost starting: I0603 18:10:09.473534 11660 fix.go:108] recreateIfNeeded on aira-208: state=Running err= W0603 18:10:09.473534 11660 fix.go:134] unexpected machine state, will restart: I0603 18:10:09.477426 11660 out.go:157] 🏃 Updating the running ssh "aira-208" bare metal machine ... 🏃 Updating the running ssh "aira-208" bare metal machine ... I0603 18:10:09.477971 11660 machine.go:88] provisioning docker machine ... I0603 18:10:09.477971 11660 main.go:126] libmachine: Waiting for SSH to be available... I0603 18:10:09.478502 11660 main.go:126] libmachine: Getting to WaitForSSH function... I0603 18:10:09.490391 11660 main.go:126] libmachine: Using SSH client type: native I0603 18:10:09.490927 11660 main.go:126] libmachine: &{{{ 0 [] [] []} root [] 0xae1f00 [] 0s} 10.196.0.103 22 } I0603 18:10:09.490927 11660 main.go:126] libmachine: About to run SSH command: exit 0 I0603 18:10:09.781665 11660 main.go:126] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none], no supported methods remain I0603 18:10:13.094572 11660 main.go:126] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none], no supported methods remain ....

I replaced the ssh key with stars.

I have no further idea on what the issue is. When I do the ssh connection by hand it works fine (the ssh connection). But minikube does seem to fail to establish a ssh connection. Not sure what else I could do?

[sharifelgamal]

cc @afbjorklund, who is the SSH driver expert

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[afbjorklund]

Seems to be more related to the other SSH issues, try with --native-ssh=false if still relevant

[vitvavra]

@afbjorklund Yes, it is definitelly relevant to the other issues. But the quick fix is:
Delete %user%\.minikube\ folder (not sure which part is causing it)
Run using minikube start --driver=ssh --native-ssh=false --ssh-user=<youruser> --ssh-ip-address=<ip of machine>
And make sure you have systemd enabled Systemd
And also make sure you have the machine accessible using the root user as the --ssh-user is ignored due to another bug #10618