vmware: dialing TCP: ssh: handshake failed / Client.Timeout exceeded while awaiting headers

[CaledoniaProject]

Running latest version, I have multiple issues with minikube and it's not working at all. The command line is:

https_proxy=http://192.168.154.200:6666 minikube start --docker-env http_proxy=http://192.168.154.200:6666 --docker-env https_proxy=http://192.168.154.200:6666 --docker-env no_proxy=192.168.154.0/24 --vm-driver=vmwarefusion -v=10 --alsologtostderr

1. weird config file location issue

This is the first error message I encountered:

W0204 21:36:13.423240    4392 root.go:146] Error reading config file at /Users/XX/.minikube/config/config.json: open /Users/XX/.minikube/config/config.json: no such file or directory

Clearly the file is generated and saved at ~/.minikube/machines/minikube/config.json, was it a backward compatible feature?

2. SSH login issue

I got tons of ssh errors:

dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

I had to login with password tcuser and add the key to authorized_keys, WTF?

3. kube is not started at all

Installation is finally completed, now I started to get this:

I0204 21:37:22.674386    4392 kubernetes.go:119] error getting Pods with label selector "k8s-app=kube-proxy" [Get https://192.168.154.163:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy: Service Unavailable]
I0204 21:37:23.177517    4392 round_trippers.go:386] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: minikube/v0.0.0 (darwin/amd64) kubernetes/$Format" 'https://192.168.154.163:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy'

Inside the VM:

$ curl 127.0.0.1:8443
curl: (7) Failed to connect to 127.0.0.1 port 8443: Connection refused

What should I do?

[CaledoniaProject]

Attached complete log. 1.txt

Also, changing driver to xhyve has no effect, same result.

[gbraad]

Maybe you have messed up your SSH settings. Check local settings or if you might a stray binary on path provided by something like brew, etc.

…

On Mon, Feb 4, 2019, 14:53 Aaron Lewis ***@***.*** wrote: Attached complete log. Rename to log.txt and view it. [image: 111] <https://user-images.githubusercontent.com/1357701/52212400-5a4b5480-28c7-11e9-89e0-5a23497db444.jpg> Also, changing driver to xhyve has no effect, same result. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3620 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAHZuO0D7vi0Rv58Et49dR3cskI7i1Nks5vKDtxgaJpZM4ahNv1> .

[balopat]

1. Changing the driver after you created a machine won't change a thing - if the machine was created with vmwarefusion, it will stay that way, even if you run --vm-driver=xhyve, you'll have to remove the machine with minikube delete and recreate it with minikube start.

2. Sometimes it takes a long time on my mac to the cluster to stabilize - not sure why yet - the roundtripper messages flow for a couple of minutes - it's worthwile waiting until the very end - minikube will either fail or succeed - can you rerun and see if it fails?

3. If there is failure, I would start looking into your docker proxy setup - logging onto the VM, what is the output of the following commands?

docker ps -a 
docker run hello-world 

[balopat]

dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], > no supported methods remain

This is odd - I haven't seen it in the logs - how did you get this exactly?

[CaledoniaProject]

@gbraad @balopat I don't have a stray binary. But I have ~/.ssh/config in place to determine which ssh private key to use. And minikube used the wrong key, perhaps?

Here's my ssh config

Host 192.168.*
   StrictHostKeyChecking no
   UserKnownHostsFile=/dev/null
   ControlMaster no
   IdentityFile ~/.ssh/trusted/id_rsa

Anyway, it seems to happen with vmwarefusion driver only. I have removed the machine with the delete command, going to recreate it.

[CaledoniaProject]

I might figured it out:

I forget to start the proxy, so the first run on docker pull failed

	[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.13.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1

I started minikube again, it did not attempt to pull the docker image again. It went straightly to start all services ...

I0205 10:48:19.297687   49495 ssh_runner.go:101] SSH: sudo kubeadm init phase control-plane all --config /var/lib/kubeadm.yaml
I0205 10:48:19.318773   49495 utils.go:224] > [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0205 10:48:19.318824   49495 utils.go:224] > [control-plane] Creating static Pod manifest for "kube-apiserver"
I0205 10:48:19.325622   49495 utils.go:224] > [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0205 10:48:19.326484   49495 utils.go:224] > [control-plane] Creating static Pod manifest for "kube-scheduler"
I0205 10:48:19.327807   49495 ssh_runner.go:101] SSH: sudo kubeadm init phase etcd local --config /var/lib/kubeadm.yaml
I0205 10:48:19.358632   49495 utils.go:224] > [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0205 10:48:19.368870   49495 loader.go:359] Config loaded from file /Users/c0debreak/.kube/config
I0205 10:48:19.372829   49495 round_trippers.go:386] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: minikube/v0.0.0 (darwin/amd64) kubernetes/$Format" 'https://192.168.64.3:8443/api/v1/namespaces/kube-system/pods?labelSelector=k8s-app%3Dkube-proxy'

[CaledoniaProject]

Still not working. Failed at the RBAC stage:

I0205 16:37:16.517251    1190 loader.go:359] Config loaded from file /Users/XXX/.kube/config
I0205 16:37:16.523612    1190 round_trippers.go:386] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: minikube/v0.0.0 (darwin/amd64) kubernetes/$Format" 'https://192.168.64.7:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/minikube-rbac?timeout=1m0s'

port 8443 is up in the VM:

$ curl -k 'https://192.168.64.7:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s'
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "clusterrolebindings.rbac.authorization.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"clusterrolebindings\" in API group \"rbac.authorization.k8s.io\" at the cluster scope",
  "reason": "Forbidden",
  "details": {
    "group": "rbac.authorization.k8s.io",
    "kind": "clusterrolebindings"
  },
  "code": 403
}

Attached complete log: log.txt

[tstromberg]

Possibly a dupe of #3242

[balopat]

What kind of proxy are you running? It would be useful to know the details of your setup so that we can repro this.

[tstromberg]

Also related to #3145

[CaledoniaProject]

It's a squid proxy running on a remote server.

[CaledoniaProject]

I should start minikube with --docker-env only, removing the http_proxy=XXX env on minikube itself solved this problem.