[help] start minikube error

[xkcoding]

BUG REPORT :

Please provide the following details:

Environment:

Minikube version (use minikube version): v0.32.0

• OS (e.g. from /etc/os-release): MacOS mojave 10.14.2 (18C54)
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): virtualbox
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): "Boot2DockerURL": "file:///Users/yangkai.shen/.minikube/cache/iso/minikube-v0.32.0.iso"
• Install tools: brew

What happened:

$ minikube start
Starting local Kubernetes v1.12.4 cluster...
Starting VM...
Downloading Minikube ISO
 178.88 MB / 178.88 MB [============================================] 100.00% 0s
Getting VM IP address...
E0109 17:30:57.090762   47169 start.go:211] Error parsing version semver:  Version string empty
Moving files into cluster...
Downloading kubeadm v1.12.4
Downloading kubelet v1.12.4
Finished Downloading kubeadm v1.12.4
Finished Downloading kubelet v1.12.4
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Stopping extra container runtimes...
Starting cluster components...
E0109 17:38:49.423563   47169 start.go:343] Error starting cluster:  kubeadm init error
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI
 running command: : Process exited with status 2
================================================================================
An error has occurred. Would you like to opt in to sending anonymized crash
information to minikube to help prevent future errors?
To opt out of these messages, run the command:
	minikube config set WantReportErrorPrompt false
================================================================================
Please enter your response [Y/n]: %

Output of minikube logs (if applicable):

Jan 09 09:51:22 minikube kubelet[4040]: I0109 09:51:22.503828    4040 server.go:408] Version: v1.12.4
Jan 09 09:51:22 minikube kubelet[4040]: I0109 09:51:22.504056    4040 plugins.go:99] No cloud provider specified.
Jan 09 09:51:22 minikube kubelet[4040]: F0109 09:51:22.504123    4040 server.go:262] failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory
Jan 09 09:51:22 minikube systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Jan 09 09:51:22 minikube systemd[1]: kubelet.service: Failed with result 'exit-code'.
Jan 09 09:51:32 minikube systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Jan 09 09:51:32 minikube systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 85.
Jan 09 09:51:32 minikube systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Jan 09 09:51:32 minikube systemd[1]: Started kubelet: The Kubernetes Node Agent.
Jan 09 09:51:32 minikube kubelet[4053]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 09 09:51:32 minikube kubelet[4053]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 09 09:51:32 minikube kubelet[4053]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 09 09:51:32 minikube kubelet[4053]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 09 09:51:32 minikube kubelet[4053]: Flag --allow-privileged has been deprecated, will be removed in a future version
Jan 09 09:51:32 minikube kubelet[4053]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 09 09:51:32 minikube kubelet[4053]: Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jan 09 09:51:32 minikube kubelet[4053]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.

[jo-me]

Same issue on Windows 7 with a completely new installation of the latest versions of minikube and kubectl.

Could it be due to a proxy issue?

[xkcoding]

Same issue on Windows 7 with a completely new installation of the latest versions of minikube and kubectl.

Could it be due to a proxy issue?

I have no idea about proxy.... and I can't find a way to solve this issue. I think maybe I should try some old version

[jo-me]

I was able to get it to run.
Aside from proxy issues I assume that there is also a versioning conflict because minikube assumes an older (1.12.x) version of Kubernetes while the latest kubectl is already at 1.13.1.

So I ended up starting like this to force the same k8s version:
minikube start --alsologtostderr --kubernetes-version v1.13.1

[xkcoding]

I use your command and get another issue

I0114 10:47:38.727892    4695 utils.go:224] ! error execution phase preflight: [preflight] Some fatal errors occurred:
I0114 10:47:38.727931    4695 utils.go:224] ! 	[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.13.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp :80: connect: connection refused
I0114 10:47:38.727941    4695 utils.go:224] ! , error: exit status 1
I0114 10:47:38.727963    4695 utils.go:224] ! 	[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.13.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp :80: connect: connection refused
I0114 10:47:38.727975    4695 utils.go:224] ! , error: exit status 1

I think I know that you say what proxy means

[xkcoding]

btw, I run it minikube start --alsologtostderr --kubernetes-version v1.13.1 --docker-env HTTP_PROXY=http://192.168.99.88:1087 --docker-env HTTPS_PROXY=http://192.168.99.88:1087 like this , still not working... 😭

[xkcoding]

😆
Eventually, I start minikube successfully.
I checked my proxy connection is OK, and I found a proxy issue #530

In this issue, I found a comment #530 (comment) can solve the proxy problem.

https://forums.docker.com/t/beginner-having-trouble-with-docker-behind-company-proxy/3968

Here is my complete command,that can be start successfully.

minikube start --alsologtostderr --kubernetes-version v1.13.1 --docker-env HTTP_PROXY=http://10.0.2.2:1087 --docker-env HTTPS_PROXY=http://10.0.2.2:1087 --docker-env NO_PROXY=10.0.2.2,192.168.99.100

copy to your Terminal and you will get

.....
I0114 13:35:19.102229    7316 utils.go:224] > Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
I0114 13:35:19.102246    7316 utils.go:224] >   https://kubernetes.io/docs/concepts/cluster-administration/addons/
I0114 13:35:19.102260    7316 utils.go:224] > You can now join any number of machines by running the following on each node
I0114 13:35:19.102287    7316 utils.go:224] > as root:
I0114 13:35:19.102307    7316 utils.go:224] >   kubeadm join localhost:8443 --token agf5vb.45h4fw6jqy7o2yff --discovery-token-ca-cert-hash sha256:c60ce3456d2ac3225edc119d1792d1b6c20b40b03cf0812ddf8d2fc867f21bf3
Verifying kubelet health ...I0114 13:35:19.134467    7316 ssh_runner.go:137] Run with output: sudo systemctl is-active kubelet
I0114 13:35:19.141912    7316 utils.go:224] > active

Verifying apiserver health ...I0114 13:35:19.149990    7316 kubeadm.go:99] https://192.168.99.101:8443/healthz response: <nil> &{Status:200 OK StatusCode:200 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Date:[Mon, 14 Jan 2019 05:35:19 GMT] Content-Length:[2] Content-Type:[text/plain; charset=utf-8]] Body:0xc0004b9d00 ContentLength:2 TransferEncoding:[] Close:false Uncompressed:false Trailer:map[] Request:0xc0002da500 TLS:0xc000354000}
Kubectl is now configured to use the cluster.
Loading cached images from config file.


Everything looks great. Please enjoy minikube!

@jo-me Thanks a lot~ 👍

[336655asd]

I tried your solution:

minikube start --alsologtostderr --kubernetes-version v1.13.2 --docker-env HTTP_PROXY=http://10.0.2.2:1087 --docker-env HTTPS_PROXY=http://10.0.2.2:1087 --docker-env NO_PROXY=10.0.2.2,192.168.99.100

but got a similar error like you:

I0117 10:05:40.117320   11862 utils.go:224] ! error execution phase preflight: [preflight] Some fatal errors occurred:
I0117 10:05:40.117493   11862 utils.go:224] ! 	[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.13.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 10.0.2.2:1087: connect: connection refused
I0117 10:05:40.118137   11862 utils.go:224] ! , error: exit status 1
I0117 10:05:40.118566   11862 utils.go:224] ! 	[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.13.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 10.0.2.2:1087: connect: connection refused
I0117 10:05:40.118978   11862 utils.go:224] ! , error: exit status 1

I'm also behind company proxy but I don't know how to fix this question.

This is my option:

• OS: Ubuntu 16.04
• minikube: v0.32.0
• VM Driver: virtualbox

[xkcoding]

@336655asd Ummm... 1087 is my computer proxy port, you should change this, and 192.168.99.100 you can use $(minikube ip) instead

you can try this

minikube start --alsologtostderr --kubernetes-version v1.13.1 --docker-env HTTP_PROXY=http://10.0.2.2:${your proxy port} --docker-env HTTPS_PROXY=http://10.0.2.2:${your proxy port} --docker-env NO_PROXY=10.0.2.2,$(minikube ip)

remember to replace ${your proxy port} to your real proxy port.

good luck!

[336655asd]

Yes, you are right! I pulled the image successfully as your suggestion.
I did this：

minikube start --alsologtostderr --kubernetes-version v1.13.2 --docker-env HTTP_PROXY=http://10.0.2.2:3128 --docker-env HTTPS_PROXY=http://10.0.2.2:3128 --docker-env NO_PROXY=10.0.2.2,192.168.99.100

and this is my proxy:
HTTP_PROXY=http://127.0.0.1:3128
However another question comes up now......

non-retriable error: creating clusterrolebinding: Post https://192.168.99.100:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: x509: cannot validate certificate for 192.168.99.100 because it doesn't contain any IP SANs
Error starting cluster:  timed out waiting to elevate kube-system RBAC privileges: Temporary Error: creating clusterrolebinding: Post https://192.168.99.100:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: net/http: TLS handshake timeout
creating clusterrolebinding: Post https://192.168.99.100:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: x509: cannot validate certificate for 192.168.99.100 because it doesn't contain any IP SANs

And during logs:

unable to load client CA file /var/lib/minikube/certs/ca.crt: open /var/lib/minikube/certs/ca.crt: no such file or directory 
failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory 

They have been there since the first time I started minikube.

The new error log is

Unable to register node "minikube" with API server: Post https://localhost:8443/api/v1/nodes: dial tcp 127.0.0.1:8443: connect: connection refused
......
Failed to get status for pod "kube-scheduler-minikube_kube-system(9729a196c4723b60ab401eaff722982d)": Get https://localhost:8443/api/v1/namespaces/kube-system/pods/kube-scheduler-minikube: dial tcp 127.0.0.1:8443: connect: connection refused
......
node "minikube" not found
......
pod_workers.go:190] Error syncing pod 9396654f-1a2c-11e9-9a9e-0800277cff9a ("kubernetes-dashboard-ccc79bfc9-7m8zd_kube-system(9396654f-1a2c-11e9-9a9e-0800277cff9a)"), skipping: failed to "StartContainer" for "kubernetes-dashboard" with CrashLoopBackOff: "Back-off 10s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-ccc79bfc9-7m8zd_kube-system(9396654f-1a2c-11e9-9a9e-0800277cff9a)"

Maybe there are some other questions in my configuration. I am trying to fix it.
Whatever thanks a lot!

[336655asd]

Finally I succeed!
The proxy I used is cntlm.
This time I edited the /etc/cntlm.conf.
I append 192.168.* to NoProxy this item, and succeed!
I run

minikube start --alsologtostderr --kubernetes-version v1.13.2 --docker-env HTTP_PROXY=http://10.0.2.2:3128 --docker-env HTTPS_PROXY=http://10.0.2.2:3128 --docker-env NO_PROXY=10.0.2.2,192.168.99.100

And my proxy configuration
HTTP_PROXY=http://127.0.0.1:3128

Please must remember to add your minikube IP to NO_PROXY.

[xkcoding]

bravo, congratulations !

😄