kubeadm init waiting for the control plane to become ready on CentOS 7.2 with kubeadm 1.6.1

[rushabh268]

After downloading kubeadm 1.6.1 and starting kubeadm init, it gets stuck at [apiclient] Created API client, waiting for the control plane to become ready

kubeadm init --kubernetes-version v1.6.1 --apiserver-advertise-address=10.X.X.X
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.6.1
[init] Using Authorization mode: RBAC
[preflight] Running pre-flight checks
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [<hostname> kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.X.X.X]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[apiclient] Created API client, waiting for the control plane to become ready

I have the following 10-kubeadm.conf

cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=192.168.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_EXTRA_ARGS --cgroup-driver=systemd

So, its no longer a cgroup issue. Also, I have flushed the iptables rules and disabled selinux. I have also specified the IP address of the interface which I want to use for my master but it still doesn't go through.

From the logs,

Apr 06 12:55:55 hostname kubelet[5174]: I0406 12:55:55.087703    5174 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Apr 06 12:55:55 hostname kubelet[5174]: I0406 12:55:55.146554    5174 kubelet_node_status.go:77] Attempting to register node hostname
Apr 06 12:55:55 hostname kubelet[5174]: E0406 12:55:55.147133    5174 kubelet_node_status.go:101] Unable to register node "hostname" with API server: Post https://10.X.X.X:6443/api/v1/nodes: dial tcp 10.X.X.X:6443: getsockopt: connection refused
Apr 06 12:55:55 hostname kubelet[5174]: E0406 12:55:55.553801    5174 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.X.X.X:6443/api/v1/services?resourceVersion=0: dial tcp 10.X.X.X:6443: getsockopt: connection refused
Apr 06 12:55:55 hostname kubelet[5174]: E0406 12:55:55.555837    5174 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.X.X.X:6443/api/v1/nodes?fieldSelector=metadata.name%3Dhostname&resourceVersion=0: dial tcp 10.X.X.X:6443: getsockopt: connection refused
Apr 06 12:55:55 hostname kubelet[5174]: E0406 12:55:55.556271    5174 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.X.X.X:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dhostname&resourceVersion=0: dial tcp 10.X.X.X:6443: getsockopt: connection refused
Apr 06 12:55:55 hostname kubelet[5174]: E0406 12:55:55.828198    5174 event.go:208] Unable to write event: 'Post https://10.X.X.X:6443/api/v1/namespaces/default/events: dial tcp 10.X.X.X:6443: getsockopt: connection refused' (may retry after sleeping)
Apr 06 12:55:56 hostname kubelet[5174]: E0406 12:55:56.555099    5174 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.X.X.X:6443/api/v1/services?resourceVersion=0: dial tcp 10.X.X.X:6443: getsockopt: connection refused
Apr 06 12:55:56 hostname kubelet[5174]: E0406 12:55:56.556772    5174 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.X.X.X:6443/api/v1/nodes?fieldSelector=metadata.name%3Dhostname&resourceVersion=0: dial tcp 10.X.X.X:6443: getsockopt: connection refused
Apr 06 12:55:56 hostname kubelet[5174]: E0406 12:55:56.557978    5174 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.X.X.X:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dhostname&resourceVersion=0: dial tcp 10.X.X.X:6443: getsockopt: connection refused
Apr 06 12:55:56 hostname kubelet[5174]: I0406 12:55:56.760733    5174 kubelet.go:1752] skipping pod synchronization - [Failed to start ContainerManager systemd version does not support ability to start a slice as transient unit]
Apr 06 12:55:56 hostname kubelet[5174]: W0406 12:55:56.858684    5174 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
Apr 06 12:55:56 hostname kubelet[5174]: E0406 12:55:56.858931    5174 kubelet.go:2067] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 06 12:55:57 hostname kubelet[5174]: E0406 12:55:57.556067    5174 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.X.X.X:6443/api/v1/services?resourceVersion=0: dial tcp 10.X.X.X:6443: getsockopt: connection refused
Apr 06 12:55:57 hostname kubelet[5174]: E0406 12:55:57.557441    5174 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.X.X.X:6443/api/v1/nodes?fieldSelector=metadata.name%3Dhostname&resourceVersion=0: dial tcp 10.X.X.X:6443: getsockopt: connection refused
Apr 06 12:55:57 hostname kubelet[5174]: E0406 12:55:57.558822    5174 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.X.X.X:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dhostname&resourceVersion=0: dial tcp 10.X.X.X:6443: getsockopt: connection refused
Apr 06 12:55:58 hostname kubelet[5174]: I0406 12:55:58.347460    5174 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Apr 06 12:55:58 hostname kubelet[5174]: I0406 12:55:58.405762    5174 kubelet_node_status.go:77] Attempting to register node hostname
Apr 06 12:55:58 hostname kubelet[5174]: E0406 12:55:58.406037    5174 kubelet_node_status.go:101] Unable to register node "hostname" with API server: Post https://10.X.X.X:6443/api/v1/nodes: dial tcp 10.X.X.X:6443: getsockopt: connection refused
Apr 06 12:55:58 hostname kubelet[5174]: E0406 12:55:58.556829    5174 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.X.X.X:6443/api/v1/services?resourceVersion=0: dial tcp 10.X.X.X:6443: getsockopt: connection refused

Versions

kubeadm version (use kubeadm version):
kubeadm version
kubeadm version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e", GitTreeState:"clean", BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

• Cloud provider or hardware configuration:
  Bare metal nodes

• OS (e.g. from /etc/os-release):
  cat /etc/redhat-release
  CentOS Linux release 7.2.1511 (Core)

• Kernel (e.g. uname -a):
  uname -a
  Linux hostname 3.10.0-327.18.2.el7.x86_64 kubeadm join on slave node fails preflight checks #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

• Others:
  docker -v
  Docker version 1.12.6, build 96d83a5/1.12.6
  rpm -qa | grep kube
  kubelet-1.6.1-0.x86_64
  kubernetes-cni-0.5.1-0.x86_64
  kubeadm-1.6.1-0.x86_64
  kubectl-1.6.1-0.x86_64

What happened?

Kubeadm getting stuck waiting for control plane to get ready

What you expected to happen?

It should have gone through and finished the init

[Yengas]

I'm having the same issue. I also tried removing the Network ARGS as suggested on another issue. It still hangs at waiting for control plane to be ready.

[githubvick]

Did you reload Daemon and restart kubelet service after making the changes. It worked after changing the driver and commenting network. It takes 10 to 11 minutes for the control plane to get ready for the first time for me, I would suggest leaving it for 15 minutes for the first time.

[rushabh268]

I did reload the daemon and restarted the kubelet service every single time. I have even left the setup undisturbed for entire night but it was still waiting for the control plane.

[Yengas]

I've reloaded daemon (systemctl daemon-reload) and restarted kubelet aswell. I run kubeadm reset, edit the service configuration, reload the daemon, then run thekubeadm init.

Apiserver and etcd docker containers fail to run after commenting out the networking options. I also tried installing weave-net manually so the cni config directory would be populated, but that didn't work neither. To do this i installed weave, ran weave setup and weave launch. I don't really know how Kubeadm configures Docker to use the CNI settings, but there maybe an step i'm missing here.

[phagunbaya]

Looks like kubelet cannot reach kube api server.

[Yengas]

I noticed that etcd wasn't able to listen on port 2380, i've followed these steps again and my cluster started up:

• Run kubeadm reset to remove any changes made to the server.
• Get back the machine to initial state. By re installing kubeadm(to get back the original config files).
• Remove docker containers related to kubernetes if there are any.
• Get and install weave. Don't run it.
• Reboot the server.
• Make sure kubelet doesn't run.
• Run weave setup and weave launch.
• Run kubeadm init.

If you want to get rid of managing weave by hand...

• Run weave reset
• Apply the weave addon for kubernetes 1.6.
• Reboot the server.

Kubeadm join should work on other servers.

[jruels]

@Yengas Can you please provide more details on the weave steps? Did you run them on all nodes, or just the master?

[Yengas]

@jruels just the master node. Weave is just a single binary. The setup command without any args, downloads weave docker images and creates the CNI config. The launch command without any args starts the weave containers on host only.

[rushabh268]

@Yengas Still not sure, what you mean by - "get and install weave. Don't run it" I cannot obviously do kubectl apply -f https://git.io/weave-kube-1.6 so how do I install weave?

[mikedanese]

What do the apiserver logs say?

[jruels]

@rushabh268
To install weave run the following on the master
sudo curl -L git.io/weave -o /usr/local/bin/weave && chmod a+x /usr/local/bin/weave
Then run
weave setup
When that completes run
weave launch

[mikedanese]

you don't need to do that. kubectl apply -f https://git.io/weave-kube-1.6 should be sufficient.

[rushabh268]

The API server logs say the exact same as I mentioned in the bug. Also, I cannot do kubectl because the Kubernetes is not installed

[rushabh268]

@jruels I'll try it out and update this thread!