kubeadm init not working and kubelet not running #761

gobenpark · 2018-04-20T06:48:05Z

@BumwooPark commented on Fri Apr 20 2018

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug
/kind feature

What happened:
command kubeadm init

then showing

5: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp [::1]:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.

Unfortunately, an error has occurred:
	timed out waiting for the condition

This error is likely caused by:
	- The kubelet is not running
	- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
	- Either there is no internet connection, or imagePullPolicy is set to "Never",
	  so the kubelet cannot pull or find the following control plane images:
		- k8s.gcr.io/kube-apiserver-amd64:v1.10.1
		- k8s.gcr.io/kube-controller-manager-amd64:v1.10.1
		- k8s.gcr.io/kube-scheduler-amd64:v1.10.1
		- k8s.gcr.io/etcd-amd64:3.1.12 (only if no external etcd endpoints are configured)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
	- 'systemctl status kubelet'
	- 'journalctl -xeu kubelet'
couldn't initialize a Kubernetes cluster

What you expected to happen:

[bootstraptoken] Using token: <token>
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: kube-dns
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run (as a regular user):

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  http://kubernetes.io/docs/admin/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>

How to reproduce it (as minimally and precisely as possible):
kubeadm reset and after kubeadm init

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version):
Cloud provider or hardware configuration:
OS (e.g. from /etc/os-release):

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Kernel (e.g. uname -a):
Linux 3.10.0-693.21.1.el7.x86_64 kubeadm join on slave node fails preflight checks #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Install tools:
allowing to https://kubernetes.io/docs/setup/independent/install-kubeadm/
Others:
i had

swapoff -a
and add Environment="KUBELET_EXTRA_ARGS=--fail-swap-on=false" in 10-kubeadm.conf
and docker & kubelet cgroup set systemd same and other checking cgroupfs
but not starting kubelet

i had set after command
systemctl daemon-reload
systemctl restart kubelet

but not working

my systemctl status kubelet

● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf, 90-local-extras.conf
   Active: activating (auto-restart) (Result: exit-code) since 금 2018-04-20 15:27:00 KST; 6s ago
     Docs: http://kubernetes.io/docs/
  Process: 17247 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
 Main PID: 17247 (code=exited, status=255)

 4월 20 15:27:00 ems01 kubelet[17247]: --runtime-cgroups string                              Optional absolute name of cgroups to create and run the runtime in.
 4월 20 15:27:00 ems01 kubelet[17247]: --seccomp-profile-root string                         <Warning: Alpha feature> Directory path for seccomp profiles. (default "/var/lib/kubelet/seccomp")
 4월 20 15:27:00 ems01 kubelet[17247]: --stderrthreshold severity                            logs at or above this threshold go to stderr (default 2)
 4월 20 15:27:00 ems01 kubelet[17247]: -v, --v Level                                             log level for V logs
 4월 20 15:27:00 ems01 kubelet[17247]: --version version[=true]                              Print version information and quit
 4월 20 15:27:00 ems01 kubelet[17247]: --vmodule moduleSpec                                  comma-separated list of pattern=N settings for file-filtered logging
 4월 20 15:27:00 ems01 systemd[1]: Unit kubelet.service entered failed state.
 4월 20 15:27:00 ems01 kubelet[17247]: --volume-plugin-dir string                            The full path of the directory in which to search for additional third party volume plugins (default "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/")
 4월 20 15:27:00 ems01 kubelet[17247]: F0420 15:27:00.231291   17247 server.go:145] unknown flag: --require-kubeconfig
 4월 20 15:27:00 ems01 systemd[1]: kubelet.service failed.
(xiilab) [root@ems01 ~]# systemctl status kubelet -l
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf, 90-local-extras.conf
   Active: activating (auto-restart) (Result: exit-code) since 금 2018-04-20 15:27:51 KST; 5s ago
     Docs: http://kubernetes.io/docs/
  Process: 17441 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
 Main PID: 17441 (code=exited, status=255)

 4월 20 15:27:51 ems01 kubelet[17441]: --rotate-certificates                                 <Warning: Beta feature> Auto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches.
 4월 20 15:27:51 ems01 kubelet[17441]: --runonce                                             If true, exit after spawning pods from static pod files or remote urls. Exclusive with --enable-server
 4월 20 15:27:51 ems01 kubelet[17441]: --runtime-cgroups string                              Optional absolute name of cgroups to create and run the runtime in.
 4월 20 15:27:51 ems01 kubelet[17441]: --seccomp-profile-root string                         <Warning: Alpha feature> Directory path for seccomp profiles. (default "/var/lib/kubelet/seccomp")
 4월 20 15:27:51 ems01 kubelet[17441]: --stderrthreshold severity                            logs at or above this threshold go to stderr (default 2)
 4월 20 15:27:51 ems01 kubelet[17441]: -v, --v Level                                             log level for V logs
 4월 20 15:27:51 ems01 kubelet[17441]: --version version[=true]                              Print version information and quit
 4월 20 15:27:51 ems01 kubelet[17441]: --vmodule moduleSpec                                  comma-separated list of pattern=N settings for file-filtered logging
 4월 20 15:27:51 ems01 kubelet[17441]: --volume-plugin-dir string                            The full path of the directory in which to search for additional third party volume plugins (default "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/")
 4월 20 15:27:51 ems01 kubelet[17441]: F0420 15:27:51.492949   17441 server.go:145] unknown flag: --require-kubeconfig

@k8s-ci-robot commented on Fri Apr 20 2018

@BumwooPark: There are no sig labels on this issue. Please add a sig label.

A sig label can be added by either:

mentioning a sig: @kubernetes/sig-<group-name>-<group-suffix>
e.g., @kubernetes/sig-contributor-experience-<group-suffix> to notify the contributor experience sig, OR
specifying the label manually: /sig <group-name>
e.g., /sig scalability to apply the sig/scalability label

Note: Method 1 will trigger an email to the group. See the group list.
The <group-suffix> in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

The text was updated successfully, but these errors were encountered:

chuckha · 2018-04-20T13:22:08Z

@BumwooPark The error in the logs is here

 4월 20 15:27:00 ems01 kubelet[17247]: F0420 15:27:00.231291   17247 server.go:145] unknown flag: --require-kubeconfig

Few questions;

can you paste the systemd files for the kubelet?
how did you get those systemd files?

gobenpark · 2018-04-20T15:21:58Z

@chuckha

10-kubeadm.conf

1 [Service]
2 Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true --cgroup-driver=systemd"
3 Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
4 Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
5 Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
6 Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
7 Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
8 Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
9 Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"
10 ExecStart=
11 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUB ELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS
12 Environment="KUBELET_EXTRA_ARGS=--fail-swap-on=false"

install to yum reference by https://kubernetes.io/docs/setup/independent/install-kubeadm/

my kubeadm version is

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:26:04Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?

and use Physical machine not VM

chuckha · 2018-04-20T16:28:48Z

What is the output of yum list kubelet ? Another way to phrase that question is: what version of the kubelet got installed? Your kubeadm client might be 1.10.1, but it looks like the kubelet is old.

I don't see any yum update on this page https://kubernetes.io/docs/setup/independent/install-kubeadm/ so perhaps your yum is out of date?

chuckha · 2018-04-20T16:29:45Z

The --require-kubeconfig flag was deprecated at some point (probably 1.9 or 1.8) and removed in 1.10:

kubernetes/kubernetes#58367

gobenpark · 2018-04-21T07:13:39Z

command yum list kubelet is kubelet.x86_64 1.10.1-0 @kubernetes
and yum list installed is same version

gobenpark · 2018-04-23T01:00:07Z

@chuckha

I installed kubeadm & kubelet again today
It works well wonderfully.... :0

thanks to your help 😀

rbehravesh · 2018-04-30T09:42:34Z

@BumwooPark Could you please describe how did you solve this issue?

sunfan666 · 2018-10-11T09:26:02Z

Reboot well wonderfully.........😌

NullIsNot0 · 2018-11-28T12:08:23Z

Check if Docker service is running:
systemctl enable docker
If not, start it and enable service to automatically start after reboot:
systemctl start docker
systemctl enable docker

rkm1990 · 2019-05-30T10:12:13Z

in my case it was issue with change in hostname.
I changed my hostname after installation.This could have caused the issue.
Hence I removed and installed kuberadm and the issue was fixed.

oudream · 2019-06-18T00:56:30Z

apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat </etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update

kubeadm init error The kubelet is not running

install kubelet.service

apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

oudream · 2019-06-18T00:56:55Z

Note: This dropin only works with kubeadm and kubelet v1.11+

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"

This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically

EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env

This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use

the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.

EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

kaushalendra-kumar · 2020-06-27T11:13:42Z

@BumwooPark Could you please describe how did you solve this issue?

I also faced this issu. solved using https://kubernetes.io/docs/setup/independent/install-kubeadm/

gobenpark mentioned this issue Apr 20, 2018

kubeadm init not working and kubelet not running kubernetes/kubernetes#62897

Closed

gobenpark closed this as completed Apr 23, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubeadm init not working and kubelet not running #761

kubeadm init not working and kubelet not running #761

gobenpark commented Apr 20, 2018

chuckha commented Apr 20, 2018 •

edited

Loading

gobenpark commented Apr 20, 2018 •

edited

Loading

chuckha commented Apr 20, 2018

chuckha commented Apr 20, 2018 •

edited

Loading

gobenpark commented Apr 21, 2018

gobenpark commented Apr 23, 2018

rbehravesh commented Apr 30, 2018

sunfan666 commented Oct 11, 2018

NullIsNot0 commented Nov 28, 2018

rkm1990 commented May 30, 2019

oudream commented Jun 18, 2019

oudream commented Jun 18, 2019

kaushalendra-kumar commented Jun 27, 2020

kubeadm init not working and kubelet not running #761

kubeadm init not working and kubelet not running #761

Comments

gobenpark commented Apr 20, 2018

chuckha commented Apr 20, 2018 • edited Loading

gobenpark commented Apr 20, 2018 • edited Loading

chuckha commented Apr 20, 2018

chuckha commented Apr 20, 2018 • edited Loading

gobenpark commented Apr 21, 2018

gobenpark commented Apr 23, 2018

rbehravesh commented Apr 30, 2018

sunfan666 commented Oct 11, 2018

NullIsNot0 commented Nov 28, 2018

rkm1990 commented May 30, 2019

oudream commented Jun 18, 2019

kubeadm init error The kubelet is not running

install kubelet.service

oudream commented Jun 18, 2019

Note: This dropin only works with kubeadm and kubelet v1.11+

This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically

This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use

the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.

kaushalendra-kumar commented Jun 27, 2020

chuckha commented Apr 20, 2018 •

edited

Loading

gobenpark commented Apr 20, 2018 •

edited

Loading

chuckha commented Apr 20, 2018 •

edited

Loading