hyperkit: minikube fails to restart (kubeadm exited with status 1)

[cmoulliard]

Config

OS: MacOS
Driver: Hyperkit
minikube: minikube version: v0.31.0
Config:
minikube config set vm-driver hyperkit
minikube config set WantReportError true
minikube config set cpus 4
minikube config set kubernetes-version v1.13.0
minikube config set memory 5000

Problem

When we stop and start again minikube, then this error is reported

minikube start --logtostderr
I1217 14:35:21.992918   14554 notify.go:121] Checking for updates...
Starting local Kubernetes v1.13.0 cluster...
Starting VM...
I1217 14:35:22.380182   14554 cluster.go:74] Skipping create...Using existing machine configuration
I1217 14:35:22.476876   14554 cluster.go:82] Machine state:  Stopped
I1217 14:36:19.912947   14554 ssh_runner.go:57] Run: sudo rm -f /etc/docker/ca.pem
I1217 14:36:19.920127   14554 ssh_runner.go:57] Run: sudo mkdir -p /etc/docker
I1217 14:36:19.933421   14554 ssh_runner.go:57] Run: sudo rm -f /etc/docker/server.pem
I1217 14:36:19.938178   14554 ssh_runner.go:57] Run: sudo mkdir -p /etc/docker
I1217 14:36:19.950732   14554 ssh_runner.go:57] Run: sudo rm -f /etc/docker/server-key.pem
I1217 14:36:19.957115   14554 ssh_runner.go:57] Run: sudo mkdir -p /etc/docker
Getting VM IP address...
E1217 14:36:21.942750   14554 start.go:210] Error parsing version semver:  Version string empty
Moving files into cluster...
I1217 14:36:21.972096   14554 kubeadm.go:248] Container runtime flag provided with no value, using defaults.
I1217 14:36:21.973027   14554 ssh_runner.go:57] Run: sudo rm -f /usr/bin/kubeadm
I1217 14:36:21.973028   14554 ssh_runner.go:57] Run: sudo rm -f /usr/bin/kubelet
I1217 14:36:21.980397   14554 ssh_runner.go:57] Run: sudo mkdir -p /usr/bin
I1217 14:36:21.984965   14554 ssh_runner.go:57] Run: sudo mkdir -p /usr/bin
I1217 14:36:27.658867   14554 ssh_runner.go:57] Run: sudo rm -f /lib/systemd/system/kubelet.service
I1217 14:36:27.664545   14554 ssh_runner.go:57] Run: sudo mkdir -p /lib/systemd/system
I1217 14:36:27.674369   14554 ssh_runner.go:57] Run: sudo rm -f /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
I1217 14:36:27.680040   14554 ssh_runner.go:57] Run: sudo mkdir -p /etc/systemd/system/kubelet.service.d
I1217 14:36:27.689915   14554 ssh_runner.go:57] Run: sudo rm -f /var/lib/kubeadm.yaml
I1217 14:36:27.695759   14554 ssh_runner.go:57] Run: sudo mkdir -p /var/lib
I1217 14:36:27.707028   14554 ssh_runner.go:57] Run: sudo rm -f /etc/kubernetes/addons/storageclass.yaml
I1217 14:36:27.714111   14554 ssh_runner.go:57] Run: sudo mkdir -p /etc/kubernetes/addons
I1217 14:36:27.725955   14554 ssh_runner.go:57] Run: sudo rm -f /etc/kubernetes/manifests/addon-manager.yaml
I1217 14:36:27.731071   14554 ssh_runner.go:57] Run: sudo mkdir -p /etc/kubernetes/manifests/
I1217 14:36:27.741016   14554 ssh_runner.go:57] Run: sudo rm -f /etc/kubernetes/addons/storage-provisioner.yaml
I1217 14:36:27.746524   14554 ssh_runner.go:57] Run: sudo mkdir -p /etc/kubernetes/addons
I1217 14:36:27.757021   14554 ssh_runner.go:57] Run: sudo rm -f /etc/kubernetes/addons/dashboard-dp.yaml
I1217 14:36:27.763240   14554 ssh_runner.go:57] Run: sudo mkdir -p /etc/kubernetes/addons
I1217 14:36:27.774072   14554 ssh_runner.go:57] Run: sudo rm -f /etc/kubernetes/addons/dashboard-svc.yaml
I1217 14:36:27.779052   14554 ssh_runner.go:57] Run: sudo mkdir -p /etc/kubernetes/addons
I1217 14:36:27.790428   14554 ssh_runner.go:57] Run:
sudo systemctl daemon-reload &&
sudo systemctl enable kubelet &&
sudo systemctl start kubelet

Setting up certs...
I1217 14:36:27.934225   14554 certs.go:47] Setting up certificates for IP: 192.168.99.101
I1217 14:36:27.955288   14554 ssh_runner.go:57] Run: sudo rm -f /var/lib/minikube/certs/ca.crt
I1217 14:36:27.962727   14554 ssh_runner.go:57] Run: sudo mkdir -p /var/lib/minikube/certs/
I1217 14:36:27.977527   14554 ssh_runner.go:57] Run: sudo rm -f /var/lib/minikube/certs/ca.key
I1217 14:36:27.983774   14554 ssh_runner.go:57] Run: sudo mkdir -p /var/lib/minikube/certs/
I1217 14:36:28.000864   14554 ssh_runner.go:57] Run: sudo rm -f /var/lib/minikube/certs/apiserver.crt
I1217 14:36:28.013110   14554 ssh_runner.go:57] Run: sudo mkdir -p /var/lib/minikube/certs/
I1217 14:36:28.027316   14554 ssh_runner.go:57] Run: sudo rm -f /var/lib/minikube/certs/apiserver.key
I1217 14:36:28.034032   14554 ssh_runner.go:57] Run: sudo mkdir -p /var/lib/minikube/certs/
I1217 14:36:28.054529   14554 ssh_runner.go:57] Run: sudo rm -f /var/lib/minikube/certs/proxy-client-ca.crt
I1217 14:36:28.059377   14554 ssh_runner.go:57] Run: sudo mkdir -p /var/lib/minikube/certs/
I1217 14:36:28.072859   14554 ssh_runner.go:57] Run: sudo rm -f /var/lib/minikube/certs/proxy-client-ca.key
I1217 14:36:28.078665   14554 ssh_runner.go:57] Run: sudo mkdir -p /var/lib/minikube/certs/
I1217 14:36:28.090295   14554 ssh_runner.go:57] Run: sudo rm -f /var/lib/minikube/certs/proxy-client.crt
I1217 14:36:28.095375   14554 ssh_runner.go:57] Run: sudo mkdir -p /var/lib/minikube/certs/
I1217 14:36:28.107872   14554 ssh_runner.go:57] Run: sudo rm -f /var/lib/minikube/certs/proxy-client.key
I1217 14:36:28.112573   14554 ssh_runner.go:57] Run: sudo mkdir -p /var/lib/minikube/certs/
I1217 14:36:28.132298   14554 ssh_runner.go:57] Run: sudo rm -f /var/lib/minikube/kubeconfig
I1217 14:36:28.137555   14554 ssh_runner.go:57] Run: sudo mkdir -p /var/lib/minikube
Connecting to cluster...
Setting up kubeconfig...
I1217 14:36:28.403202   14554 config.go:125] Using kubeconfig:  /Users/dabou/.kube/config
Stopping extra container runtimes...
Machine exists, restarting cluster components...
I1217 14:36:28.565969   14554 ssh_runner.go:57] Run:
sudo kubeadm alpha phase certs all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase kubeconfig all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase controlplane all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase etcd local --config /var/lib/kubeadm.yaml

E1217 14:36:28.604772   14554 start.go:348] Error restarting cluster:  running cmd:
sudo kubeadm alpha phase certs all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase kubeconfig all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase controlplane all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase etcd local --config /var/lib/kubeadm.yaml
: Process exited with status 1

[cmoulliard]

FYI: The error is not related to the hypervisor used as I get the same problem using virtualbox, hyperkit or xhyve

Is it possible to get detail of the error Process exited with status 1 ? What should be the command to be executed within the vm to start kubeadm and to try to understand what is the root cause of the issue ?

[cmoulliard]

It appears that kubelet failed to restart

$ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Mon 2018-12-17 13:39:22 UTC; 8s ago
     Docs: http://kubernetes.io/docs/
  Process: 3080 ExecStart=/usr/bin/kubelet --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --authorization-mode=Webhook --client-ca-file=/var/lib/minikube/certs/ca.crt --fail-swap-on=false --kubeconfig=/etc/kubernetes/kubelet.conf --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --cgroup-driver=cgroupfs --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --hostname-override=minikube (code=exited, status=255)
 Main PID: 3080 (code=exited, status=255)
...
-- The start-up result is RESULT.
Dec 17 13:40:23 minikube kubelet[3169]: I1217 13:40:23.732420    3169 server.go:407] Version: v1.13.0
Dec 17 13:40:23 minikube kubelet[3169]: I1217 13:40:23.732701    3169 plugins.go:103] No cloud provider specified.
Dec 17 13:40:23 minikube kubelet[3169]: F1217 13:40:23.732789    3169 server.go:261] failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory
Dec 17 13:40:23 minikube systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Dec 17 13:40:23 minikube systemd[1]: kubelet.service: Failed with result 'exit-code'.
Dec 17 13:40:27 minikube kernel: hpet1: lost 318 rtc interrupts

as it can't find the file

Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory

[cmoulliard]

The kubeconfig file has been deleted between stop/start

minikube ssh
                         _             _
            _         _ ( )           ( )
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ ls -la /etc/kubernetes/
total 0
drwxr-xr-x  4 root root 0 Dec 17 13:36 .
drwxr-xr-x 30 root root 0 Dec 17 13:36 ..
drwxr-xr-x  2 root root 0 Dec 17 13:36 addons
drwxr-xr-x  2 root root 0 Dec 17 13:36 manifests
$ ls -la /etc/kubernetes/addons/
total 16
drwxr-xr-x 2 root root    0 Dec 17 13:36 .
drwxr-xr-x 4 root root    0 Dec 17 13:36 ..
-rw-r----- 1 root root 1514 Dec 17 13:36 dashboard-dp.yaml
-rw-r----- 1 root root  979 Dec 17 13:36 dashboard-svc.yaml
-rw-r----- 1 root root 1709 Dec 17 13:36 storage-provisioner.yaml
-rw-r----- 1 root root  273 Dec 17 13:36 storageclass.yaml
$ ls -la /etc/kubernetes/manifests/
total 4
drwxr-xr-x 2 root root    0 Dec 17 13:36 .
drwxr-xr-x 4 root root    0 Dec 17 13:36 ..
-rw-r----- 1 root root 1406 Dec 17 13:36 addon-manager.yaml

[ervikrant06]

Hitting the same issue with version 0.32.0 also.

minikube start
There is a newer version of minikube available (v0.32.0).  Download it here:
https://github.com/kubernetes/minikube/releases/tag/v0.32.0

To disable this notification, run the following:
minikube config set WantUpdateNotification false
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
Kubernetes version downgrade is not supported. Using version: v1.13.0
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
E0103 17:32:35.581862    6773 start.go:302] Error restarting cluster:  running cmd:
sudo kubeadm alpha phase certs all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase kubeconfig all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase controlplane all --config /var/lib/kubeadm.yaml &&
sudo /usr/bin/kubeadm alpha phase etcd local --config /var/lib/kubeadm.yaml
: Process exited with status 1

[jalchr]

Me too ...
Any resolutions ?

[liuqimin]

[PCasafont]

Still happening as of 0.33.1

[tstromberg]

@PCasafont - Do you mind posting the output from v0.33.x, along with what platform you are on? I'm sure bugs in this path still linger, but I haven't been able to reproduce this.

[cmoulliard]

Problem is resolved since 0.33 as I don't experiment anymore this issue and minikube restarts properly

minikube stop
Stopping local Kubernetes cluster...
Machine stopped.

minikube start
Starting local Kubernetes v1.13.2 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Stopping extra container runtimes...
Machine exists, restarting cluster components...
Verifying kubelet health ...
Verifying apiserver health ....
Kubectl is now configured to use the cluster.
Loading cached images from config file.


Everything looks great. Please enjoy minikube!

So I propose to close the ticket. Make sense ?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[tstromberg]

I believe this issue was resolved in the v1.1.0 release. Please try upgrading to the latest release of minikube, and if the same issue occurs, please re-open this bug. Thank you opening this bug report, and for your patience!