Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

none on Fedora 30: failed to get the kubelet's cgroup: cpu and memory cgroup hierarchy not unified #5127

Closed
lehh opened this issue Aug 18, 2019 · 10 comments
Labels
co/kubelet Kubelet config issues co/none-driver kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done.

Comments

@lehh
Copy link

lehh commented Aug 18, 2019

The exact command to reproduce the issue:
After installing minikube I ran:
sudo minikube config set vm-driver none
sudo minikube start

The full output of the command that failed
😄  minikube v1.3.1 on Fedora 30
🤹  Running on localhost (CPUs=6, Memory=7955MB, Disk=25070MB) ...
ℹ️   OS release is Fedora 30 (Workstation Edition)
🐳  Preparing Kubernetes v1.15.2 on Docker 1.13.1 ...
    ▪ kubelet.resolv-conf=/run/systemd/resolve/resolv.conf
🚜  Pulling images ...
🚀  Launching Kubernetes ... 

💣 Error starting cluster: cmd failed: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--data-minikube,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap

: running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--data-minikube,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap
output: [init] Using Kubernetes version: v1.15.2
[preflight] Running pre-flight checks
[WARNING Firewalld]: firewalld is active, please ensure ports [8443 10250] are open or your cluster may not function correctly
[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING FileExisting-socat]: socat not found in system path
[WARNING Hostname]: hostname "minikube" could not be reached
[WARNING Hostname]: hostname "minikube": lookup minikube on 192.168.1.1:53: no such host
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs/"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [minikube localhost] and IPs [192.168.1.15 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [minikube localhost] and IPs [192.168.1.15 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp [::1]:10248: connect: connection refused.

Unfortunately, an error has occurred:
timed out waiting for the condition

This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
: running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--data-minikube,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap
.: exit status 1

😿 Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉 https://github.com/kubernetes/minikube/issues/new/choose

The output of the "minikube logs" command
Aug 18 19:09:41 Fedora dockerd-current[2930]: time="2019-08-18T19:09:41.212979339-03:00" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n"
Aug 18 19:09:41 Fedora dockerd-current[2930]: time="2019-08-18T19:09:41.214861674-03:00" level=warning msg="failed to retrieve docker-init version: unknown output format: tini version 0.18.0\n"
Aug 18 19:09:41 Fedora dockerd-current[2930]: time="2019-08-18T19:09:41.308356033-03:00" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n"
Aug 18 19:09:41 Fedora dockerd-current[2930]: time="2019-08-18T19:09:41.308980547-03:00" level=warning msg="failed to retrieve docker-init version: unknown output format: tini version 0.18.0\n"
Aug 18 19:09:41 Fedora dockerd-current[2930]: time="2019-08-18T19:09:41.340250132-03:00" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n"
Aug 18 19:09:41 Fedora dockerd-current[2930]: time="2019-08-18T19:09:41.340833984-03:00" level=warning msg="failed to retrieve docker-init version: unknown output format: tini version 0.18.0\n"
Aug 18 19:09:41 Fedora dockerd-current[2930]: time="2019-08-18T19:09:41.497722794-03:00" level=error msg="Handler for GET /v1.26/containers/k8s.gcr.io/kube-apiserver:v1.15.2/json returned error: No such container: k8s.gcr.io/kube-apiserver:v1.15.2"
Aug 18 19:09:41 Fedora dockerd-current[2930]: time="2019-08-18T19:09:41.498051114-03:00" level=error msg="Handler for GET /v1.26/containers/k8s.gcr.io/kube-apiserver:v1.15.2/json returned error: No such container: k8s.gcr.io/kube-apiserver:v1.15.2"
Aug 18 19:09:41 Fedora dockerd-current[2930]: time="2019-08-18T19:09:41.524744388-03:00" level=error msg="Handler for GET /v1.26/containers/k8s.gcr.io/kube-controller-manager:v1.15.2/json returned error: No such container: k8s.gcr.io/kube-controller-manager:v1.15.2"
Aug 18 19:09:41 Fedora dockerd-current[2930]: time="2019-08-18T19:09:41.525018153-03:00" level=error msg="Handler for GET /v1.26/containers/k8s.gcr.io/kube-controller-manager:v1.15.2/json returned error: No such container: k8s.gcr.io/kube-controller-manager:v1.15.2"
Aug 18 19:09:41 Fedora dockerd-current[2930]: time="2019-08-18T19:09:41.549290068-03:00" level=error msg="Handler for GET /v1.26/containers/k8s.gcr.io/kube-scheduler:v1.15.2/json returned error: No such container: k8s.gcr.io/kube-scheduler:v1.15.2"
Aug 18 19:09:41 Fedora dockerd-current[2930]: time="2019-08-18T19:09:41.549570271-03:00" level=error msg="Handler for GET /v1.26/containers/k8s.gcr.io/kube-scheduler:v1.15.2/json returned error: No such container: k8s.gcr.io/kube-scheduler:v1.15.2"
Aug 18 19:09:41 Fedora dockerd-current[2930]: time="2019-08-18T19:09:41.571476664-03:00" level=error msg="Handler for GET /v1.26/containers/k8s.gcr.io/kube-proxy:v1.15.2/json returned error: No such container: k8s.gcr.io/kube-proxy:v1.15.2"
Aug 18 19:09:41 Fedora dockerd-current[2930]: time="2019-08-18T19:09:41.571820593-03:00" level=error msg="Handler for GET /v1.26/containers/k8s.gcr.io/kube-proxy:v1.15.2/json returned error: No such container: k8s.gcr.io/kube-proxy:v1.15.2"
Aug 18 19:09:41 Fedora dockerd-current[2930]: time="2019-08-18T19:09:41.595521785-03:00" level=error msg="Handler for GET /v1.26/containers/k8s.gcr.io/pause:3.1/json returned error: No such container: k8s.gcr.io/pause:3.1"
Aug 18 19:09:41 Fedora dockerd-current[2930]: time="2019-08-18T19:09:41.595836927-03:00" level=error msg="Handler for GET /v1.26/containers/k8s.gcr.io/pause:3.1/json returned error: No such container: k8s.gcr.io/pause:3.1"
Aug 18 19:09:41 Fedora dockerd-current[2930]: time="2019-08-18T19:09:41.618804200-03:00" level=error msg="Handler for GET /v1.26/containers/k8s.gcr.io/etcd:3.3.10/json returned error: No such container: k8s.gcr.io/etcd:3.3.10"
Aug 18 19:09:41 Fedora dockerd-current[2930]: time="2019-08-18T19:09:41.619174668-03:00" level=error msg="Handler for GET /v1.26/containers/k8s.gcr.io/etcd:3.3.10/json returned error: No such container: k8s.gcr.io/etcd:3.3.10"
Aug 18 19:09:41 Fedora dockerd-current[2930]: time="2019-08-18T19:09:41.641446281-03:00" level=error msg="Handler for GET /v1.26/containers/k8s.gcr.io/coredns:1.3.1/json returned error: No such container: k8s.gcr.io/coredns:1.3.1"
Aug 18 19:09:41 Fedora dockerd-current[2930]: time="2019-08-18T19:09:41.641819171-03:00" level=error msg="Handler for GET /v1.26/containers/k8s.gcr.io/coredns:1.3.1/json returned error: No such container: k8s.gcr.io/coredns:1.3.1"
Aug 18 19:09:41 Fedora dockerd-current[2930]: time="2019-08-18T19:09:41.713819391-03:00" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n"
Aug 18 19:09:41 Fedora dockerd-current[2930]: time="2019-08-18T19:09:41.714342708-03:00" level=warning msg="failed to retrieve docker-init version: unknown output format: tini version 0.18.0\n"

==> container status <==
sudo: crictl: command not found
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

==> dmesg <==
[ +0.000009] snd_hda_intel 0000:01:00.1: spurious response 0x0:0x0, last cmd=0x378901
[ +7.628119] radeon_dp_aux_transfer_native: 284 callbacks suppressed
[Aug18 16:14] radeon_dp_aux_transfer_native: 32 callbacks suppressed
[ +38.971214] snd_hdac_bus_update_rirb: 450 callbacks suppressed
[ +0.000007] snd_hda_intel 0000:01:00.1: spurious response 0x0:0x0, last cmd=0x770740
[ +0.000005] snd_hda_intel 0000:01:00.1: spurious response 0x200:0x0, last cmd=0xb70740
[ +0.000004] snd_hda_intel 0000:01:00.1: spurious response 0x0:0x0, last cmd=0x377200
[ +0.000014] snd_hda_intel 0000:01:00.1: spurious response 0x0:0x0, last cmd=0x578901
[ +0.000003] snd_hda_intel 0000:01:00.1: spurious response 0x0:0x0, last cmd=0x777200
[ +0.000003] snd_hda_intel 0000:01:00.1: spurious response 0x0:0x0, last cmd=0x778901
[ +0.000003] snd_hda_intel 0000:01:00.1: spurious response 0x0:0x0, last cmd=0x977200
[ +0.000003] snd_hda_intel 0000:01:00.1: spurious response 0x0:0x0, last cmd=0x978901
[ +0.000003] snd_hda_intel 0000:01:00.1: spurious response 0x0:0x0, last cmd=0xb77200
[ +0.000003] snd_hda_intel 0000:01:00.1: spurious response 0x0:0x0, last cmd=0xb78901
[ +1.262119] radeon_dp_aux_transfer_native: 32 callbacks suppressed
[ +5.352300] radeon_dp_aux_transfer_native: 74 callbacks suppressed
[ +8.373222] radeon_dp_aux_transfer_native: 32 callbacks suppressed
[Aug18 16:33] radeon_dp_aux_transfer_native: 32 callbacks suppressed
[ +0.687724] hrtimer: interrupt took 2998022 ns
[ +1.614347] snd_hdac_bus_update_rirb: 444 callbacks suppressed
[ +0.000010] snd_hda_intel 0000:01:00.1: spurious response 0x0:0x0, last cmd=0x970740
[ +0.000010] snd_hda_intel 0000:01:00.1: spurious response 0x600:0x0, last cmd=0x377200
[ +0.000010] snd_hda_intel 0000:01:00.1: spurious response 0x0:0x0, last cmd=0x578901
[ +0.000028] snd_hda_intel 0000:01:00.1: spurious response 0x0:0x0, last cmd=0xd78901
[ +0.000008] snd_hda_intel 0000:01:00.1: spurious response 0x0:0x0, last cmd=0x270e00
[ +0.000004] snd_hda_intel 0000:01:00.1: spurious response 0x0:0x0, last cmd=0x370100
[ +0.000007] snd_hda_intel 0000:01:00.1: spurious response 0x0:0x0, last cmd=0x470d01
[ +0.000006] snd_hda_intel 0000:01:00.1: spurious response 0x0:0x0, last cmd=0x470e00
[ +0.000007] snd_hda_intel 0000:01:00.1: spurious response 0x0:0x0, last cmd=0x570882
[ +0.000007] snd_hda_intel 0000:01:00.1: spurious response 0x0:0x0, last cmd=0x670e00

==> kernel <==
19:18:41 up 3:06, 1 user, load average: 0.83, 0.66, 0.62
Linux Fedora 5.1.16-300.fc30.x86_64 #1 SMP Wed Jul 3 15:06:51 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
PRETTY_NAME="Fedora 30 (Workstation Edition)"

==> kubelet <==
-- Logs begin at Mon 2018-12-03 20:05:46 -02, end at Sun 2019-08-18 19:18:41 -03. --
Aug 18 19:18:40 Fedora kubelet[23073]: Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 18 19:18:40 Fedora kubelet[23073]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 18 19:18:40 Fedora kubelet[23073]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 18 19:18:40 Fedora kubelet[23073]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 18 19:18:40 Fedora kubelet[23073]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 18 19:18:40 Fedora kubelet[23073]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 18 19:18:40 Fedora kubelet[23073]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 18 19:18:40 Fedora kubelet[23073]: Flag --resolv-conf has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 18 19:18:40 Fedora kubelet[23073]: I0818 19:18:40.553689 23073 server.go:425] Version: v1.15.2
Aug 18 19:18:40 Fedora kubelet[23073]: I0818 19:18:40.554024 23073 plugins.go:103] No cloud provider specified.
Aug 18 19:18:40 Fedora kubelet[23073]: F0818 19:18:40.559978 23073 server.go:273] failed to run Kubelet: failed to get the kubelet's cgroup: cpu and memory cgroup hierarchy not unified. cpu: /, memory: /system.slice/kubelet.service
Aug 18 19:18:40 Fedora systemd[1]: kubelet.service: Main process exited, code=exited, status=255/EXCEPTION
Aug 18 19:18:40 Fedora systemd[1]: kubelet.service: Failed with result 'exit-code'.
Aug 18 19:18:41 Fedora systemd[1]: kubelet.service: Service RestartSec=600ms expired, scheduling restart.
Aug 18 19:18:41 Fedora systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 718.
Aug 18 19:18:41 Fedora systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Aug 18 19:18:41 Fedora systemd[1]: Started kubelet: The Kubernetes Node Agent.
Aug 18 19:18:41 Fedora kubelet[23165]: Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 18 19:18:41 Fedora kubelet[23165]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 18 19:18:41 Fedora kubelet[23165]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 18 19:18:41 Fedora kubelet[23165]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 18 19:18:41 Fedora kubelet[23165]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 18 19:18:41 Fedora kubelet[23165]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 18 19:18:41 Fedora kubelet[23165]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 18 19:18:41 Fedora kubelet[23165]: Flag --resolv-conf has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 18 19:18:41 Fedora kubelet[23165]: I0818 19:18:41.298262 23165 server.go:425] Version: v1.15.2
Aug 18 19:18:41 Fedora kubelet[23165]: I0818 19:18:41.298624 23165 plugins.go:103] No cloud provider specified.
Aug 18 19:18:41 Fedora kubelet[23165]: F0818 19:18:41.306308 23165 server.go:273] failed to run Kubelet: failed to get the kubelet's cgroup: cpu and memory cgroup hierarchy not unified. cpu: /, memory: /system.slice/kubelet.service
Aug 18 19:18:41 Fedora systemd[1]: kubelet.service: Main process exited, code=exited, status=255/EXCEPTION
Aug 18 19:18:41 Fedora systemd[1]: kubelet.service: Failed with result 'exit-code'.

The operating system version: Fedora 30 Workstation.

I configured the user permissions as stated by @afbjorklund on issue #5099 but I'm still getting errors.
I opened a new issue since I'm using Fedora and the error seems different from the later mentioned issue.
Obs: Running minikube using the virtualbox driver worked without problems.

@afbjorklund
Copy link
Collaborator

Did you open the required ports in firewalld, as suggested by kubeadm ?

[WARNING Firewalld]: firewalld is active, please ensure ports [8443 10250] are open or your cluster may not function correctly

@tstromberg
Copy link
Contributor

I believe this is the fatal error:

Aug 18 19:18:41 Fedora kubelet[23165]: F0818 19:18:41.306308   23165 server.go:273] failed to run Kubelet: failed to get the kubelet's cgroup: cpu and memory cgroup hierarchy not unified.  cpu: /, memory: /system.slice/kubelet.service

This sounds like https://bugzilla.redhat.com/show_bug.cgi?id=1692131

@tstromberg tstromberg added co/kubelet Kubelet config issues co/none-driver labels Aug 20, 2019
@tstromberg
Copy link
Contributor

My guess is that this can be overcome setting kubelet flags (or changing the vm driver). I'm not sure which yet though.

@tstromberg tstromberg changed the title Error starting cluster: cmd failed: sudo /usr/bin/kubeadm init none on Fedora 30: failed to get the kubelet's cgroup: cpu and memory cgroup hierarchy not unified Aug 20, 2019
@tstromberg tstromberg added the priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. label Aug 20, 2019
@lehh
Copy link
Author

lehh commented Aug 21, 2019

Hello. Thank you for the answers.

So, I tried to run minikube with these flags (Font):
--extra-config=kubelet.runtime-cgroups=/systemd/system.slice
--extra-config=kubelet.kubelet-cgroups=/systemd/system.slice

And the error changed to:

failed to get the container runtime's cgroup: failed to get container name for docker process: cpu and memory cgroup hierarchy not unified.  cpu: /system.slice, memory: /system.slice/docker.service

Then, after following what jfpucheu commented in this issue:

Create a 11-cgroups.conf file at /etc/systemd/system/kubelet.service.d/ with:
[Service] 
CPUAccounting=true 
MemoryAccounting=true

Then run:

systemctl daemon-reload
systemctl restart kubelet

I started minikube with the previous flags plus this one:
--extra-config=kubelet.cgroup-driver=systemd

I got this new error:

😿  Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉  https://github.com/kubernetes/minikube/issues/new/choose
❌  Problems detected in kube-apiserver:
    error: unable to load server certificate: open /var/lib/minikube/certs/apiserver.crt: permission denied
❌  Problems detected in kube-addon-manager:
    error: Error loading config file "/var/lib/minikube/kubeconfig": open /var/lib/minikube/kubeconfig: permission denied
    error: Error loading config file "/var/lib/minikube/kubeconfig": open /var/lib/minikube/kubeconfig: permission denied
    error: Error loading config file "/var/lib/minikube/kubeconfig": open /var/lib/minikube/kubeconfig: permission denied

I'm running with sudo.
I will continue trying later after I come back from work.
Even though, I hope this helps somehow.

@cousine
Copy link

cousine commented Aug 28, 2019

I believe this is the fatal error:

Aug 18 19:18:41 Fedora kubelet[23165]: F0818 19:18:41.306308   23165 server.go:273] failed to run Kubelet: failed to get the kubelet's cgroup: cpu and memory cgroup hierarchy not unified.  cpu: /, memory: /system.slice/kubelet.service

This sounds like https://bugzilla.redhat.com/show_bug.cgi?id=1692131

I had the same issue and I can confirm its related to https://bugzilla.redhat.com/show_bug.cgi?id=1692131 , enabling DefaultCPUAccounting in /etc/systemd/system.conf and restarting resolves the issue.

P.S. This is originally an issue with kubeadm and kubelet not minikube.

@afbjorklund
Copy link
Collaborator

As long as we have none, we will end up getting kubeadm issues... But maybe we can detect this situation, and offer a better solution?

@tstromberg tstromberg added the kind/bug Categorizes issue or PR as related to a bug. label Sep 19, 2019
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 18, 2019
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jan 17, 2020
@fejta-bot
Copy link

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@k8s-ci-robot
Copy link
Contributor

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
co/kubelet Kubelet config issues co/none-driver kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done.
Projects
None yet
Development

No branches or pull requests

6 participants