Detect "cascading evictions" of DaemonSet pods

[mac-chaffee]

When pods in a DaemonSet get repeatedly evicted (due to using too much RAM or ephemeral-storage), it seems there's currently no way to detect that situation.

• Humans looking at kubectl get pods output won't detect it because DaemonSets immediately delete Evicted pods
• KubePodCrashLooping won't detect the issue because the Evicted pods are deleted and new ones are created.
• KubeDaemonSetRolloutStuck/KubeDaemonSetNotScheduled/KubeDaemonSetMisScheduled won't detect the issue because all the pods are up-to-date and being scheduled just fine. Just that they get killed a few seconds after creation.

I think we should add an alert for this kind of situation, but unsure how to do it.

• We could get the cluster-wide eviction rate with kubelet_evictions, but it doesn't include the pod name or anything.
• kube_daemonset_status_number_unavailable has a chance of detecting the issue, but only if the evictions consistently happen around the 30-second interval when that is measured. In the two cascading-eviction events I've witnessed, only one of them had the evictions happening fast enough to show up on a graph consistently (still didn't trigger any existing alerts).

If we can't think of a better way to detect this, we be forced to change some of the upstream metrics better-detect evictions.

[mac-chaffee]

Maybe we just need a generic "elevated rate of evictions" alert? For example, those spikes on the right are what we'd be trying to detect:

sum(rate(kubelet_evictions[1d])) > 0.0001 seems to work for my specific situation. Thoughts on adding that rule?

[github-actions]

This issue has not had any activity in the past 30 days, so the
stale label has been added to it.

• The stale label will be removed if there is new activity
• The issue will be closed in 7 days if there is no new activity
• Add the keepalive label to exempt this issue from the stale check action

Thank you for your contributions!