Log something about OOMKilled containers

[sylr]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind feature

What happened:

Container gets killed because it tries to use more memory than allowed.

What you expected to happen:

Have an OOMKilled event tied to the pod and logs about this

/sig node

[sylr]

@dims in #sig-sheduling mentionned this https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/docker_container.go#L356-L359

[dims]

related discussion in sig-schedling slack https://kubernetes.slack.com/messages/C09TP78DV/

[sylr]

https://kubernetes.slack.com/archives/C09TP78DV/p1539255149000100 to be exact but it might fall outside the slack allowed history for free tier pretty soon.

[nikopen]

FYI there is a related Prometheus metric that is able to grep OOMKilled:
it's kube_pod_container_status_terminated_reason and kube_pod_container_status_last_terminated_reason

https://github.com/kubernetes/kube-state-metrics/blob/master/Documentation/pod-metrics.md

reason=<OOMKilled|Error|Completed|ContainerCannotRun>

It might be a bit tricky to implement as an alert, thus kube_pod_container_status_last_terminated_reason was created to grep the previous status as the OOMKilled can be so fast that is not grepped.
kubernetes/kube-state-metrics#344

Still, it would be nice if a logline was printed by k8s by default.

[nikopen]

kube_pod_container_status_last_terminated_reason is not yet released in a version, I'll ping to see if they're able to cut a release

[anderson4u2]

@nikopen Hi, I just tried kube_pod_container_status_last_terminated_reason in version 1.4.0 and it seems to work? i.e. use sum_over_time(kube_pod_container_status_terminated_reason{reason!="Completed"}[5m]) > 0 to detect a recent non-graceful termination.

[nikopen]

Indeed, it seems to work :)

@brancz do you know why this happens? also tried it in 1.3.1.

    - alert: OOMKilled
      expr: sum_over_time(kube_pod_container_status_terminated_reason{reason="OOMKilled"}[5m]) > 0
      for: 1m
      labels:
        severity: warning
      annotations:
        description:  Pod {{$labels.pod}} in {{$labels.namespace}} got OOMKilled

[brancz]

Because the kube_pod_container_status_terminated_reason metric was only introduced in 1.4: https://github.com/kubernetes/kube-state-metrics/releases/tag/v1.4.0-rc.0

[nikopen]

@brancz my test is in 1.3.1 and kube_pod_container_status_terminated_reason works, was it already there perhaps?

[brancz]

Strange, you're right, looks like we have a mistake in the changelog. The code seems to be there in 1.3.1: https://github.com/kubernetes/kube-state-metrics/blob/v1.3.1/collectors/pod.go#L125

[boosty]

@anderson4u2 I am a bit confused by your last comment. You wrote:

just tried kube_pod_container_status_last_terminated_reason in version 1.4.0

But in the example below you use kube_pod_container_status_terminated_reason, not kube_pod_container_status_last_terminated_reason.

So as far as I see, the new (very useful) metric kube_pod_container_status_last_terminated_reason is still unreleased.