Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubeadm init: failure loading apiserver certificate: the certificate is not valid yet #2703

Closed
kylewin opened this issue Apr 11, 2018 · 15 comments
Labels
co/kubeadm Issues relating to kubeadm

Comments

@kylewin
Copy link

kylewin commented Apr 11, 2018

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Please provide the following details:

Environment:

Minikube version (use minikube version): minikube version: v0.26.0

  • OS (e.g. from /etc/os-release): MacOS Siera 10.12
  • VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): virtualbox
  • ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): "Boot2DockerURL": "file:///Users/122462/.minikube/cache/iso/minikube-v0.26.0.iso",
  • Install tools: brew cask install minikube
  • Others:
    The above can be generated in one go with the following commands (can be copied and pasted directly into your terminal):
minikube version
echo "";
echo "OS:";
cat /etc/os-release
echo "";
echo "VM driver": 
grep DriverName ~/.minikube/machines/minikube/config.json
echo "";
echo "ISO version";
grep -i ISO ~/.minikube/machines/minikube/config.json

What happened:
Executed minikube start
What you expected to happen:
minikube successfully starts
How to reproduce it (as minimally and precisely as possible):
Execute minikube start
Output of minikube logs (if applicable): (not actually minikube log, I got this when executed minikube start)

E0411 16:57:04.402812    4049 start.go:276] Error starting cluster:  kubeadm init error sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap  running command: : running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap
 output: [init] Using Kubernetes version: v1.10.0
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
[certificates] Using the existing ca certificate and key.
	[WARNING Swap]: running with swap on is not supported. Please disable swap
Flag --admission-control has been deprecated, Use --enable-admission-plugins or --disable-admission-plugins instead. Will be removed in a future version.
failure loading apiserver certificate: the certificate is not valid yet
: running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap
.: Process exited with status 1

Anything else do we need to know: VirtualBox 5.2.8

@kylehodgetts
Copy link

kylehodgetts commented Apr 11, 2018

I've got this here too.

I even tried minikube delete and minikube start and from fresh this issue still raises its head.

After running minikube logs, this line caught my eye.
minikube kubelet[3329]: error: failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory

I'm not sure if this has significance or not

edit: Boom. Here we go:

output: Error: unknown flag: --ignore-preflight-errors
Usage:
  kubeadm init [flags]

@mrballcb
Copy link

mrballcb commented Apr 11, 2018

I'm getting the same kubeadm error (unknown flag --ignore-preflight-errors) when trying to start kube 1.8.10 in minikube. I can start kube 1.8.0 with minikube 0.25.2, the highest 1.8.x that it supports. But I cannot start kube 1.8.10 with minikube 0.26.0 because the arg to kubeadm is not supported.

When you look at the kubeadm command minikube is trying to run, in Jan 2018 it was changed from “--skip-preflight-checks” to a bunch of “--ignore-preflight-errors" args. The variable kubeadmInitTemplate in pkg/minikube/bootstrapper/kubeadm/templates.go is hard coded to use it regardless of what kube version you tell it to use. That appears to be a new 1.9.x and 1.10.x arg. The kubeadm downloaded for 1.8.x does not support the new arg, hence the error. The kubeadm downloaded for 1.9.x does, but something chokes when starting these things up, and it’s not clear to me on what.

@leanne73
Copy link

+1

I also got the same unknown flag: --ignore-preflight-errors error when trying to start a k8s v1.8 cluster with minikube 0.26.

I was able to work around it temporarily by downgrading to minikube 0.25.2, thanks for the tip @mrballcb!

@afbjorklund
Copy link
Collaborator

afbjorklund commented Apr 13, 2018

Needs to be conditionalized for kubeadm 1.8, the old arg was deprecated in 1.9 that was why. #2403

Flag --skip-preflight-checks has been deprecated, it is now equivalent to --ignore-preflight-errors=all

@afbjorklund
Copy link
Collaborator

afbjorklund commented Apr 13, 2018

Obviously storing the kubeadm log and testing the releases with 1.5.x, 1.6.x, 1.7.x, 1.8.x, 1.9.x, 1.10.x would also be good (although I guess at least some of those must be truly deprecated by now... ?)

Localkube says it supports 1.3.x and up, but then again it has been deprecated itself - so what does it know. If I read the Kubernetes docs correctly, they say that 1.7.x and earlier are now end-of-life ? But not 1.8

https://github.com/kubernetes/community/blob/master/contributors/design-proposals/release/versioning.md

@kylewin
Copy link
Author

kylewin commented Apr 13, 2018

Minikube v0.26 works fine on Ubuntu 16.04. This is not fair :(

@afbjorklund
Copy link
Collaborator

@mrballcb : The choking thing on 1.9 is probably #2716 ? (workaround for CRI, that got fixed in 1.10)

@KowalczykBartek
Copy link

as workaround - I started minikube from this release https://github.com/kubernetes/minikube/releases/tag/v0.25.1
At least works on hight sierra 10.13.3

@mgdevstack
Copy link

Minikube 0.26.0 cluster initialization still failing. My dependency is on k8s-v1.10 cluster.

How soon can we expect new version release with few important fixes like #2721

Is anyone able to run Kubernetes cluster 1.10 with Minikube 0.26.0 ?

@javajon
Copy link

javajon commented Apr 30, 2018

Since Minikube version 0.26.0 I am getting this error too with "minikube start" on Windows 10 + VirtualBox 5.2.8 (latest). I have been sticking with 0.25.2.

Related to this report I captured my details in issue #2818.

@MaLub
Copy link

MaLub commented May 6, 2018

I'll get same issue here:
output: [init] Using Kubernetes version: v1.10.0 [init] Using Authorization modes: [Node RBAC] [preflight] Running pre-flight checks. [certificates] Using the existing ca certificate and key. [WARNING Swap]: running with swap on is not supported. Please disable swap Flag --admission-control has been deprecated, Use --enable-admission-plugins or --disable-admission-plugins instead. Will be removed in a future version. failure loading apiserver certificate: the certificate is not valid yet : running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI .: Process exited with status 1

Go back to 0.25.1 works for me.

Mac OS 1.12.6

@edwardzxw
Copy link

+1
Met with the exactly same problem on CentOS 7 + VirtualBox 5.1, downgrade to 0.25.2 fixed this problem.

@raghur
Copy link

raghur commented Jun 8, 2018

Still the same error on 0.27, Win 8.1 and Vbox 5.1; Going back to 0.25.2 as that's the last known good version for me.

Why is minikube upgrade always so error prone :(.. Thankfully, its easy to create a new profile and test with that rather than muck with an existing working installation.

E0608 09:53:15.150680    3544 start.go:276] Error starting cluster:  kubeadm init error sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI  running command: : running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI
 output: [init] Using Kubernetes version: v1.10.0
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
[certificates] Using the existing ca certificate and key.
                                                                                                    [WARNING Swap]: running with swap on is not supported. Please disable swap
Flag --admission-control has been deprecated, Use --enable-admission-plugins or --disable-admission-plugins instead. Will be removed in a future version.
failure loading apiserver certificate: the certificate is not valid yet
                                                          : running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI
.: Process exited with status 1

@Rouche
Copy link

Rouche commented Jun 25, 2018

I dont know if its related...
Everything works fine with 0.28 and kubernetes 1.10.0 Windows 10 Pro + Virtualbox.

But when i try to enable Audit : /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory

Start command:
minikube start --extra-config=apiserver.Authorization.Mode=RBAC --extra-config=apiserver.Audit.LogOptions.Path=/var/log/audit.log --extra-config=apiserver.Audit.PolicyFile=/etc/kubernetes/addons/audit-policy.yaml

@ixtendio
Copy link

The same problem for me.

I'm running minikube 0.28 and kubectl 1.10.3 on MacOS High Sierra 10.13.5

I used both commands:

minikube start --extra-config=apiserver.Authorization.Mode=RBAC --extra-config=apiserver.Audit.LogOptions.Path=/var/log/audit.log --extra-config=apiserver.Audit.PolicyFile=/etc/kubernetes/addons/audit-policy.yaml

and

minikube start --extra-config=apiserver.authorization-mode=RBAC --extra-config=apiserver.audit-log-path=/var/log/audit.log --extra-config=apiserver.audit-policy-file=/etc/kubernetes/addons/audit-policy.yaml

This is the log file:

Jun 25 19:24:08 minikube kubelet[2616]: F0625 19:24:08.210031 2616 server.go:233] failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory
Jun 25 19:24:08 minikube systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Jun 25 19:24:08 minikube systemd[1]: kubelet.service: Failed with result 'exit-code'.

@tstromberg tstromberg changed the title Unable to start minikube v0.26 kubeadm init: failure loading apiserver certificate: the certificate is not valid yet Sep 19, 2018
@tstromberg tstromberg added the co/kubeadm Issues relating to kubeadm label Sep 19, 2018
cdrage added a commit to cdrage/kubeadm-ansible that referenced this issue Oct 17, 2018
In January 2018, the parameter was changed.

See: kubernetes/minikube#2703

Removes the error caused by:

```
                 Auditing=true|false (ALPHA - default=false)\n                                                      CoreDNS=true|false (default=true)\n                                                      DynamicKubeletConfig=true|false (BETA - default=false)\n  -h, --help                                          help for join\n      --ignore-preflight-errors str
ings               A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks.\n      --node-name string                              Specify the node name.\n      --tls-bootstrap-token string                    A token used for TLS bootstrapping.\n      --token string
             Use this token for both discovery-token and tls-bootstrap-token.\n\nGlobal Flags:\n      --rootfs string   [EXPERIMENTAL] The path to the 'real' host root filesystem.\n  -v, --v Level         log level for V logs\n\nerror: unknown flag: --skip-preflight-errors", "stderr_lines": ["Error: unknown flag: --skip-preflight-errors", "Usage:", "  kubeadm joi
n [flags]", "", "Flags:", "      --apiserver-advertise-address string            If the node should host a new control plane instance, the IP address the API Server will advertise it's listening on.", "      --apiserver-bind-port int32                     If the node should host a new control plane instance, the port for the API Server to bind to. (default 6443)"
, "      --config string                                 Path to kubeadm config file.", "      --cri-socket string                             Specify the CRI socket to connect to. (default \"/var/run/dockershim.sock\")", "      --discovery-file string                         A file or url from which to load cluster information.", "      --discovery-token string
                       A token used to validate cluster information fetched from the api server.", "      --discovery-token-ca-cert-hash strings          For token-based discovery, validate that the root CA public key matches this hash (format: \"<type>:<value>\").", "      --discovery-token-unsafe-skip-ca-verification   For token-based discovery, allow joining w
ithout --discovery-token-ca-cert-hash pinning.", "      --experimental-control-plane                    Create a new control plane instance on this node", "      --feature-gates string                          A set of key=value pairs that describe feature gates for various features. Options are:", "                                                      Auditing=t
rue|false (ALPHA - default=false)", "                                                      CoreDNS=true|false (default=true)", "                                                      DynamicKubeletConfig=true|false (BETA - default=false)", "  -h, --help                                          help for join", "      --ignore-preflight-errors strings
A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks.", "      --node-name string                              Specify the node name.", "      --tls-bootstrap-token string                    A token used for TLS bootstrapping.", "      --token string
Use this token for both discovery-token and tls-bootstrap-token.", "", "Global Flags:", "      --rootfs string   [EXPERIMENTAL] The path to the 'real' host root filesystem.", "  -v, --v Level         log level for V logs", "", "error: unknown flag: --skip-preflight-errors"], "stdout": "", "stdout_lines": []}
        to retry, use: --limit @/home/wikus/seafile/files/dev/linux/ansible-playbooks/kubeadm-ansible/site.retry
```

This PR changes is to the new parameter
cdrage added a commit to cdrage/kubeadm-ansible that referenced this issue Oct 17, 2018
In January 2018, the parameter was changed.

See: kubernetes/minikube#2703

Removes the error caused by:

```
                 Auditing=true|false (ALPHA - default=false)\n                                                      CoreDNS=true|false (default=true)\n                                                      DynamicKubeletConfig=true|false (BETA - default=false)\n  -h, --help                                          help for join\n      --ignore-preflight-errors str
ings               A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks.\n      --node-name string                              Specify the node name.\n      --tls-bootstrap-token string                    A token used for TLS bootstrapping.\n      --token string
             Use this token for both discovery-token and tls-bootstrap-token.\n\nGlobal Flags:\n      --rootfs string   [EXPERIMENTAL] The path to the 'real' host root filesystem.\n  -v, --v Level         log level for V logs\n\nerror: unknown flag: --skip-preflight-errors", "stderr_lines": ["Error: unknown flag: --skip-preflight-errors", "Usage:", "  kubeadm joi
n [flags]", "", "Flags:", "      --apiserver-advertise-address string            If the node should host a new control plane instance, the IP address the API Server will advertise it's listening on.", "      --apiserver-bind-port int32                     If the node should host a new control plane instance, the port for the API Server to bind to. (default 6443)"
, "      --config string                                 Path to kubeadm config file.", "      --cri-socket string                             Specify the CRI socket to connect to. (default \"/var/run/dockershim.sock\")", "      --discovery-file string                         A file or url from which to load cluster information.", "      --discovery-token string
                       A token used to validate cluster information fetched from the api server.", "      --discovery-token-ca-cert-hash strings          For token-based discovery, validate that the root CA public key matches this hash (format: \"<type>:<value>\").", "      --discovery-token-unsafe-skip-ca-verification   For token-based discovery, allow joining w
ithout --discovery-token-ca-cert-hash pinning.", "      --experimental-control-plane                    Create a new control plane instance on this node", "      --feature-gates string                          A set of key=value pairs that describe feature gates for various features. Options are:", "                                                      Auditing=t
rue|false (ALPHA - default=false)", "                                                      CoreDNS=true|false (default=true)", "                                                      DynamicKubeletConfig=true|false (BETA - default=false)", "  -h, --help                                          help for join", "      --ignore-preflight-errors strings
A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks.", "      --node-name string                              Specify the node name.", "      --tls-bootstrap-token string                    A token used for TLS bootstrapping.", "      --token string
Use this token for both discovery-token and tls-bootstrap-token.", "", "Global Flags:", "      --rootfs string   [EXPERIMENTAL] The path to the 'real' host root filesystem.", "  -v, --v Level         log level for V logs", "", "error: unknown flag: --skip-preflight-errors"], "stdout": "", "stdout_lines": []}
        to retry, use: --limit @/home/wikus/seafile/files/dev/linux/ansible-playbooks/kubeadm-ansible/site.retry
```

This PR changes is to the new parameter
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
co/kubeadm Issues relating to kubeadm
Projects
None yet
Development

No branches or pull requests