CPU usage is not shown in minikube v1.25.1 when --kubernetes-version=v1.21.2 is specified

[Noksa]

What Happened?

CPU metrics are not working anymore. They always are zero for all containers.
It works the same for none and docker drivers. About other drivers I'm not sure.

➜  ~ minikube version
minikube version: v1.25.1
commit: 3e64b11ed75e56e4898ea85f96b2e4af0301f43d

➜  ~ minikube start --driver docker --kubernetes-version=v1.21.2
😄  minikube v1.25.1 on Linuxmint 20.1
    ▪ KUBECONFIG=/home/alexandr/.kube/config:/home/alexandr/.kube/k3s_config:/home/alexandr/Work/DEBUG/rke/kube_config_cluster.yml
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.21.2 preload ...
    > preloaded-images-k8s-v16-v1...: 499.08 MiB / 499.08 MiB  100.00% 9.94 MiB
🔥  Creating docker container (CPUs=2, Memory=16000MB) ...
🐳  Preparing Kubernetes v1.21.2 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass

❗  /home/linuxbrew/.linuxbrew/bin/kubectl is version 1.23.3, which may have incompatibilites with Kubernetes 1.21.2.
    ▪ Want kubectl v1.21.2? Try 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
➜  ~ minikube addons enable metrics-server
    ▪ Using image k8s.gcr.io/metrics-server/metrics-server:v0.4.2
🌟  The 'metrics-server' addon is enabled

CPU usage is always 0 for all containers all time:

➜  ~ kubectl top pod -A
NAMESPACE     NAME                              CPU(cores)   MEMORY(bytes)
kube-system   coredns-558bd4d5db-jc4qb          0m           8Mi
kube-system   kube-proxy-97jkt                  0m           5Mi
kube-system   metrics-server-77c99ccb96-rgld6   0m           3Mi
kube-system   storage-provisioner               0m           6Mi

Or it just says that there are no resources found:

➜  ~ kubectl top pod -A
No resources found

At the same time it works with v1.22.0 minikube (as example):

➜  ~ minikube version
minikube version: v1.22.0
commit: a03fbcf166e6f74ef224d4a63be4277d017bb62e

➜  ~ minikube start --driver docker --kubernetes-version=v1.21.2
😄  minikube v1.22.0 on Linuxmint 20.1
    ▪ KUBECONFIG=/home/alexandr/.kube/config:/home/alexandr/.kube/k3s_config:/home/alexandr/Work/DEBUG/rke/kube_config_cluster.yml
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🎉  minikube 1.25.1 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.25.1
💡  To disable this notice, run: 'minikube config set WantUpdateNotification false'

💾  Downloading Kubernetes v1.21.2 preload ...
    > preloaded-images-k8s-v11-v1...: 502.14 MiB / 502.14 MiB  100.00% 6.09 MiB
    > gcr.io/k8s-minikube/kicbase...: 361.09 MiB / 361.09 MiB  100.00% 3.50 MiB
🔥  Creating docker container (CPUs=2, Memory=16000MB) ...
🐳  Preparing Kubernetes v1.21.2 on Docker 20.10.7 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass

❗  /home/linuxbrew/.linuxbrew/bin/kubectl is version 1.23.3, which may have incompatibilites with Kubernetes 1.21.2.
    ▪ Want kubectl v1.21.2? Try 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
➜  ~ minikube addons enable metrics-server
    ▪ Using image k8s.gcr.io/metrics-server/metrics-server:v0.4.2
🌟  The 'metrics-server' addon is enabled

CPU and MEMORY are not zero:

➜  ~ kubectl top pod -A
NAMESPACE     NAME                               CPU(cores)   MEMORY(bytes)
kube-system   coredns-558bd4d5db-ph4b9           7m           11Mi
kube-system   etcd-minikube                      22m          28Mi
kube-system   kube-apiserver-minikube            107m         359Mi
kube-system   kube-controller-manager-minikube   21m          43Mi
kube-system   kube-proxy-5j88l                   25m          17Mi
kube-system   kube-scheduler-minikube            3m           17Mi
kube-system   metrics-server-77c99ccb96-dpsbq    0m           3Mi
kube-system   storage-provisioner                2m           8Mi

Attach the log file

log.txt

Operating System

Ubuntu

Driver

Docker

[marksartdev]

Me too. I run with different kubernetes versions.

OS: macOS
Driver: docker, hiperkit

[spowelljr]

Hi @Noksa, thanks for reporting your issue with minikube!

In 1.24 we started increasing the kubelet.housekeeping-interval which seemed to alleviate some CPU usage, but this is likely slowing the output of CPU and memory metrics.

Could you try running minikube start --driver docker --kubernetes-version=v1.21.2 --extra-config=kubelet.housekeeping-interval=10s and see if that resolves the reporting metrics.

[marksartdev]

Hi @spowelljr, thanks.

It resolved the problem for me.

[spowelljr]

Glad that worked @marksartdev, the next minikube release will include a --disable-optimizations flag.

--disable-optimizations=false: If set, disables optimizations that are set for local Kubernetes. Including decreasing CoreDNS replicas from 2 to 1 and increasing kubeadm housekeeping-interval from 10s to 5m. Defaults to false.

[Noksa]

@spowelljr
Thanks, the --extra-config=kubelet.housekeeping-interval=10s flag works for me as well.

[spowelljr]

Glad that worked @Noksa, I'm going to close this issue, but feel free to respond if you have any further questions, thanks!

[subbu-clustrex]

@spowelljr I am running minikube by using this minikube start --driver docker --kubernetes-version=v1.21.2 --extra-config=kubelet.housekeeping-interval=10s. It gives metrics for the pods that are in the default namespace. After some time it gives an error for the same pod
$ kubectl top pods
error: Metrics not available for pod default/nginx-67dff5cc7f-57f2l, age: 13m30.207072682s

[PrayagS]

Thanks @spowelljr for the trick. It worked for me.

[usersina]

Not quite similar to the original issue but I just needed to enable the metrics-server add on in my case. (It's not enabled by default)

minikube addons enable metrics-server

kubectl top pod --all-namespaces