pod_phase tag on kubernetes.pods.running is misleading

[aquiladayc]

Observation

In my env, there's two failing pods.

NAMESPACE NAME                      READY   STATUS                     
default   postgres-A 0/1     Pending                 
test      postgres-B 1/2     ImagePullBackOff 
default   postgres-C 1/1     Running                 

kubernetes.pods.running returns 2.

This is because kubernetes.pods.running counts pods which has at least 1 container running inside.
Code link: https://github.com/DataDog/integrations-core/blob/master/kubelet/datadog_checks/kubelet/kubelet.py#L461

What is the problem?

pod_phase tag is quite confusing and non-sense for kubernetes.pods.running with 2 reasons.

Firstly, this metrics is counting Running pods. So metrics value is correct. However, as it only counts Running pod, it doesn't care about pod phase.
Secondly, pod_phase: pending for this metrics is not actually "Pending" as shown above.

This confusion becomes an issue when a user wants to count pending pods.

Suggestion for fix

Remove pod_phase tag from kubernetes.pods.running
Add something like fully_running and value can be yes or no (true or false)

[clamoriniere]

Hi @aquiladayc
Thanks for reporting this issue.

to get pod by phase we rely on other integrations "kubernetes_state" or kubernetes_state_core" that generate the metrics: kubernetes_state.pod.status_phase`.

it is because the kubernetes.pods.running is generating by a check that run on the node agent which communicate with the local kubelet. But this kubelet only see the pod already assigned to it.

kubernetes_state.pod.status_phase is generated by a "cluster check" that runs by communicating with the kubernetes api-server and so have a full visibility on pod: unschedule, pending, running.