hyperkit time falls behind: failed to write or validate certificate "apiserver": the certificate is not valid yet

[adam-ah]

The exact command to reproduce the issue:

minikube start --vm-driver hyperkit

The full output of the command that failed:

09:52 $ minikube start --vm-driver hyperkit
😄  minikube v1.0.0 on darwin (amd64)
🤹  Downloading Kubernetes v1.14.0 images in the background ...
💡  Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
🏃  Re-using the currently running hyperkit VM for "minikube" ...
⌛  Waiting for SSH access ...
📶  "minikube" IP address is 192.168.64.2
🐳  Configuring Docker as the container runtime ...
🐳  Version of container runtime is 18.06.2-ce
⌛  Waiting for image downloads to complete ...
✨  Preparing Kubernetes environment ...
🚜  Pulling images required by Kubernetes v1.14.0 ...
❌  Unable to pull images, which may be OK: running cmd: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml: command failed: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml
stdout: 
stderr: failed to pull image "k8s.gcr.io/kube-apiserver:v1.14.0": output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate has expired or is not yet valid
, error: exit status 1
: Process exited with status 1
🔄  Relaunching Kubernetes v1.14.0 using kubeadm ... 

💣  Error restarting cluster: running cmd: sudo kubeadm init phase certs all --config /var/lib/kubeadm.yaml: command failed: sudo kubeadm init phase certs all --config /var/lib/kubeadm.yaml
stdout: [certs] Using certificateDir folder "/var/lib/minikube/certs/"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver-kubelet-client certificate and key on disk

stderr: error execution phase certs/apiserver: failed to write or validate certificate "apiserver": failure loading apiserver certificate: failed to load certificate: the certificate is not valid yet
: Process exited with status 1

😿  Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉  https://github.com/kubernetes/minikube/issues/new

The output of the minikube logs command:

Only older log items from a previous run:

Apr 13 09:05:47 minikube kubelet[2771]: E0413 09:05:47.321943    2771 pod_workers.go:190] Error syncing pod 68d780c8-5b60-11e9-9e3b-6685e8e735dc ("coredns-fb8b8dccf-dlw8m_kube-system(68d780c8-5b60-11e9-9e3b-6685e8e735dc)"), skipping: failed to "StartContainer" for "coredns" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=coredns pod=coredns-fb8b8dccf-dlw8m_kube-system(68d780c8-5b60-11e9-9e3b-6685e8e735dc)"
Apr 13 09:05:47 minikube kubelet[2771]: W0413 09:05:47.416481    2771 status_manager.go:485] Failed to get status for pod "kube-apiserver-minikube_kube-system(044dfb035c2f9fb76fe27db3840af10a)": Get https://localhost:8443/api/v1/namespaces/kube-system/pods/kube-apiserver-minikube: x509: certificate has expired or is not yet valid
Apr 13 09:05:47 minikube kubelet[2771]: E0413 09:05:47.611940    2771 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Node: Get https://localhost:8443/api/v1/nodes?fieldSelector=metadata.name%!D(MISSING)minikube&limit=500&resourceVersion=0: x509: certificate has expired or is not yet valid
Apr 13 09:05:47 minikube kubelet[2771]: E0413 09:05:47.813184    2771 reflector.go:126] object-"kube-system"/"default-token-p59x2": Failed to list *v1.Secret: Get https://localhost:8443/api/v1/namespaces/kube-system/secrets?fieldSelector=metadata.name%!D(MISSING)default-token-p59x2&limit=500&resourceVersion=0: x509: certificate has expired or is not yet valid
Apr 13 09:05:48 minikube kubelet[2771]: E0413 09:05:48.008928    2771 reflector.go:126] object-"kube-system"/"storage-provisioner-token-nzkjt": Failed to list *v1.Secret: Get https://localhost:8443/api/v1/namespaces/kube-system/secrets?fieldSelector=metadata.name%!D(MISSING)storage-provisioner-token-nzkjt&limit=500&resourceVersion=0: x509: certificate has expired or is not yet valid
Apr 13 09:05:48 minikube kubelet[2771]: E0413 09:05:48.208670    2771 reflector.go:126] object-"kube-system"/"coredns": Failed to list *v1.ConfigMap: Get https://localhost:8443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%!D(MISSING)coredns&limit=500&resourceVersion=0: x509: certificate has expired or is not yet valid
Apr 13 09:05:48 minikube kubelet[2771]: E0413 09:05:48.409973    2771 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.RuntimeClass: Get https://localhost:8443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: x509: certificate has expired or is not yet valid
Apr 13 09:05:48 minikube kubelet[2771]: E0413 09:05:48.608857    2771 reflector.go:126] object-"kube-system"/"kube-proxy-token-7hpcg": Failed to list *v1.Secret: Get https://localhost:8443/api/v1/namespaces/kube-system/secrets?fieldSelector=metadata.name%!D(MISSING)kube-proxy-token-7hpcg&limit=500&resourceVersion=0: x509: certificate has expired or is not yet valid
Apr 13 09:05:48 minikube kubelet[2771]: W0413 09:05:48.808401    2771 status_manager.go:485] Failed to get status for pod "kube-proxy-lcfcm_kube-system(365e9714-5b7e-11e9-af96-6685e8e735dc)": Get https://localhost:8443/api/v1/namespaces/kube-system/pods/kube-proxy-lcfcm: x509: certificate has expired or is not yet valid
Apr 13 09:05:48 minikube kubelet[2771]: E0413 09:05:48.856007    2771 controller.go:115] failed to ensure node lease exists, will retry in 7s, error: Get https://localhost:8443/apis/coordination.k8s.io/v1beta1/namespaces/kube-node-lease/leases/minikube?timeout=10s: x509: certificate has expired or is not yet valid
Apr 13 09:05:49 minikube kubelet[2771]: E0413 09:05:49.011792    2771 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Get https://localhost:8443/api/v1/services?limit=500&resourceVersion=0: x509: certificate has expired or is not yet valid
Apr 13 09:05:49 minikube kubelet[2771]: E0413 09:05:49.225416    2771 reflector.go:126] object-"kube-system"/"kube-proxy": Failed to list *v1.ConfigMap: Get https://localhost:8443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%!D(MISSING)kube-proxy&limit=500&resourceVersion=0: x509: certificate has expired or is not yet valid

==> kubernetes-dashboard <==
2019/04/13 09:04:30 Using in-cluster config to connect to apiserver
2019/04/13 09:04:30 Starting overwatch
2019/04/13 09:04:30 Using service account token for csrf signing
2019/04/13 09:05:00 Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service account's configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout
Refer to our FAQ and wiki pages for more information: https://github.com/kubernetes/dashboard/wiki/FAQ

==> storage-provisioner <==
F0413 09:05:01.113510       1 main.go:37] Error getting server version: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout

The operating system version:

MacOS Mojave 10.14.2 (18C54)

[tstromberg]

I suspect the clock on your computer may be behind the real world. Do you mind sharing the output of:

sntp time.google.com

[tstromberg]

On second thought, it actually may just be the time within the VM. The minikube logs command should include this, but if it doesn't, you can just run minikube ssh date

If the VM clock is way behind, just run minikube delete and a new VM will be created.

[adam-ah]

10:04 $ sntp time.google.com
sntp 4.2.8p10@1.3728-o Tue Mar 21 14:36:42 UTC 2017 (136.200.1~2544)
kod_init_kod_db(): Cannot open KoD db file /var/db/ntp-kod: No such file or directory
Send to [2001:4860:4806:4::]:123 failed, No route to host
2019-05-16 10:04:09.568370 (-1000) -0.002928 +/- 0.002455 time.google.com 216.239.35.4 s1 no-leap

10:04 $ minikube ssh date
Sat Apr 13 09:13:52 UTC 2019

I think the date might have been the issue, I'm re-running the start now.

Do you think it would be worthwhile to check for this upon start? I suspect this might not be a very unique case: start minikube on a laptop, put it away, work on something else, come back to minikube, start again, the time will be grossly off.
A simple compare of minikube ssh date could warn the user to delete the VM?

I think this might answer your question(s) @tstromberg

Thanks for the quick help, minikube delete was a good workaround.

[tstromberg]

OK. Your VM clock is way behind the times then. I'm surprised that hyperkit hasn't brought it forward to the proper time. Do you mind including the "dmesg" and "kernel" sections of minikube logs output?

I'll leave this issue open because it isn't solved, but to workaround it, you can use minikube delete.

[tstromberg]

@adam-ah great idea. Opened #4264 for it.

[tstromberg]

Apparently this is a dupe of #1378

[adam-ah]

Apparently this is a dupe of #1378

Good find! I'm not sure about your policy but this seems to be a more concise workaround and problem identification bug than the other so if possible, leaving this might be useful?
I know some refrain from keeping duplicates, but I kind of like the idea so people can find both problems when searching for bugs.

[tstromberg]

This issue appears to be a duplicate of #1378, so I will close this in preference to that issue so that we may centralize the information around it.

[whyvez]

Stopping and restarting minikube fixed this issue for me.

[Bnjmn83]

• make sure no VPN is active
• minikube delete
• restart
• minikube start --vm-driver="hyperv" --hyperv-virtual-switch="myexternalswitch" --v=7 -- alsologtostderr