kubeadm init not working and kubelet not running

[gobenpark]

@BumwooPark commented on Fri Apr 20 2018

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug
/kind feature

What happened:
command kubeadm init

then showing

5: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.

Unfortunately, an error has occurred:
	timed out waiting for the condition

This error is likely caused by:
	- The kubelet is not running
	- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
	- Either there is no internet connection, or imagePullPolicy is set to "Never",
	  so the kubelet cannot pull or find the following control plane images:
		- k8s.gcr.io/kube-apiserver-amd64:v1.10.1
		- k8s.gcr.io/kube-controller-manager-amd64:v1.10.1
		- k8s.gcr.io/kube-scheduler-amd64:v1.10.1
		- k8s.gcr.io/etcd-amd64:3.1.12 (only if no external etcd endpoints are configured)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
	- 'systemctl status kubelet'
	- 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster

What you expected to happen:

[bootstraptoken] Using token: <token>
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: kube-dns
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run (as a regular user):

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  http://kubernetes.io/docs/admin/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>

How to reproduce it (as minimally and precisely as possible):
kubeadm reset and after kubeadm init

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Kernel (e.g. uname -a):
  Linux 3.10.0-693.21.1.el7.x86_64 kubeadm join on slave node fails preflight checks #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
  allowing to https://kubernetes.io/docs/setup/independent/install-kubeadm/
• Others:
  i had

swapoff -a
and add Environment="KUBELET_EXTRA_ARGS=--fail-swap-on=false" in 10-kubeadm.conf
and docker & kubelet cgroup set systemd same and other checking cgroupfs
but not starting kubelet

i had set after command
systemctl daemon-reload
systemctl restart kubelet

but not working

my systemctl status kubelet

● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf, 90-local-extras.conf
   Active: activating (auto-restart) (Result: exit-code) since 금 2018-04-20 15:27:00 KST; 6s ago
     Docs: http://kubernetes.io/docs/
  Process: 17247 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
 Main PID: 17247 (code=exited, status=255)

 4월 20 15:27:00 ems01 kubelet[17247]: --runtime-cgroups string                              Optional absolute name of cgroups to create and run the runtime in.
 4월 20 15:27:00 ems01 kubelet[17247]: --seccomp-profile-root string                         <Warning: Alpha feature> Directory path for seccomp profiles. (default "/var/lib/kubelet/seccomp")
 4월 20 15:27:00 ems01 kubelet[17247]: --stderrthreshold severity                            logs at or above this threshold go to stderr (default 2)
 4월 20 15:27:00 ems01 kubelet[17247]: -v, --v Level                                             log level for V logs
 4월 20 15:27:00 ems01 kubelet[17247]: --version version[=true]                              Print version information and quit
 4월 20 15:27:00 ems01 kubelet[17247]: --vmodule moduleSpec                                  comma-separated list of pattern=N settings for file-filtered logging
 4월 20 15:27:00 ems01 systemd[1]: Unit kubelet.service entered failed state.
 4월 20 15:27:00 ems01 kubelet[17247]: --volume-plugin-dir string                            The full path of the directory in which to search for additional third party volume plugins (default "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/")
 4월 20 15:27:00 ems01 kubelet[17247]: F0420 15:27:00.231291   17247 server.go:145] unknown flag: --require-kubeconfig
 4월 20 15:27:00 ems01 systemd[1]: kubelet.service failed.
(xiilab) [root@ems01 ~]# systemctl status kubelet -l
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf, 90-local-extras.conf
   Active: activating (auto-restart) (Result: exit-code) since 금 2018-04-20 15:27:51 KST; 5s ago
     Docs: http://kubernetes.io/docs/
  Process: 17441 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
 Main PID: 17441 (code=exited, status=255)

 4월 20 15:27:51 ems01 kubelet[17441]: --rotate-certificates                                 <Warning: Beta feature> Auto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches.
 4월 20 15:27:51 ems01 kubelet[17441]: --runonce                                             If true, exit after spawning pods from static pod files or remote urls. Exclusive with --enable-server
 4월 20 15:27:51 ems01 kubelet[17441]: --runtime-cgroups string                              Optional absolute name of cgroups to create and run the runtime in.
 4월 20 15:27:51 ems01 kubelet[17441]: --seccomp-profile-root string                         <Warning: Alpha feature> Directory path for seccomp profiles. (default "/var/lib/kubelet/seccomp")
 4월 20 15:27:51 ems01 kubelet[17441]: --stderrthreshold severity                            logs at or above this threshold go to stderr (default 2)
 4월 20 15:27:51 ems01 kubelet[17441]: -v, --v Level                                             log level for V logs
 4월 20 15:27:51 ems01 kubelet[17441]: --version version[=true]                              Print version information and quit
 4월 20 15:27:51 ems01 kubelet[17441]: --vmodule moduleSpec                                  comma-separated list of pattern=N settings for file-filtered logging
 4월 20 15:27:51 ems01 kubelet[17441]: --volume-plugin-dir string                            The full path of the directory in which to search for additional third party volume plugins (default "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/")
 4월 20 15:27:51 ems01 kubelet[17441]: F0420 15:27:51.492949   17441 server.go:145] unknown flag: --require-kubeconfig

---

@k8s-ci-robot commented on Fri Apr 20 2018

@BumwooPark: There are no sig labels on this issue. Please add a sig label.

A sig label can be added by either:

1. mentioning a sig: @kubernetes/sig-<group-name>-<group-suffix>
   e.g., @kubernetes/sig-contributor-experience-<group-suffix> to notify the contributor experience sig, OR

2. specifying the label manually: /sig <group-name>
   e.g., /sig scalability to apply the sig/scalability label

Note: Method 1 will trigger an email to the group. See the group list.
The <group-suffix> in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[chuckha]

@BumwooPark The error in the logs is here

 4월 20 15:27:00 ems01 kubelet[17247]: F0420 15:27:00.231291   17247 server.go:145] unknown flag: --require-kubeconfig

Few questions;

1. can you paste the systemd files for the kubelet?
2. how did you get those systemd files?

[gobenpark]

@chuckha

1. 10-kubeadm.conf

1 [Service]
2 Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true --cgroup-driver=systemd"
3 Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
4 Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
5 Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
6 Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
7 Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
8 Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
9 Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"
10 ExecStart=
11 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUB ELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS
12 Environment="KUBELET_EXTRA_ARGS=--fail-swap-on=false"

2. install to yum  reference by https://kubernetes.io/docs/setup/independent/install-kubeadm/

my kubeadm version is

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:26:04Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?

and use Physical machine not VM

[chuckha]

What is the output of yum list kubelet ? Another way to phrase that question is: what version of the kubelet got installed? Your kubeadm client might be 1.10.1, but it looks like the kubelet is old.

I don't see any yum update on this page https://kubernetes.io/docs/setup/independent/install-kubeadm/ so perhaps your yum is out of date?

[chuckha]

The --require-kubeconfig flag was deprecated at some point (probably 1.9 or 1.8) and removed in 1.10:

kubernetes/kubernetes#58367

[gobenpark]

command yum list kubelet is kubelet.x86_64 1.10.1-0 @kubernetes
and yum list installed is same version

[gobenpark]

@chuckha

I installed kubeadm & kubelet again today
It works well wonderfully.... :0

thanks to your help 😀

[rbehravesh]

@BumwooPark Could you please describe how did you solve this issue?

[sunfan666]

Reboot well wonderfully.........😌

[NullIsNot0]

Check if Docker service is running:
systemctl enable docker
If not, start it and enable service to automatically start after reboot:
systemctl start docker
systemctl enable docker

[rkm1990]

in my case it was issue with change in hostname.
I changed my hostname after installation.This could have caused the issue.
Hence I removed and installed kuberadm and the issue was fixed.

[oudream]

apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat </etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update

kubeadm init error The kubelet is not running

install kubelet.service

apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

[oudream]

Note: This dropin only works with kubeadm and kubelet v1.11+

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"

This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically

EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env

This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use

the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.

EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

[kaushalendra-kumar]

@BumwooPark Could you please describe how did you solve this issue?

I also faced this issu. solved using https://kubernetes.io/docs/setup/independent/install-kubeadm/