Fix backend-listen memory alert query (3x metric triple-counting)

[beastoin]

Parent: #5371

Problem

The Grafana alert "Backend-listen Memory usage is high" triple-counts container_memory_working_set_bytes because 3 Prometheus kubelet scrape services report identical cAdvisor metrics:

1. prod-kube-prometheus-stack-kubelet
2. dg-prometheus-stack-kubelet (leftover from DG self-hosted Helm)
3. prod-omi-kube-prometheus-s-kubelet

The alert query does sum by (pod) on the numerator (memory usage) which sums all 3 sources, but the denominator (limits from kube-state-metrics) has only 1 source. Result: ~3x inflated utilization.

Fix

1. Alert query fix (immediate)

Add service="prod-kube-prometheus-stack-kubelet" to the numerator:

Current:

sum(container_memory_working_set_bytes{namespace="prod-omi-backend", container!="", image!=""} * on(namespace,pod) group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{...}) by (pod)

Fixed:

sum(container_memory_working_set_bytes{namespace="prod-omi-backend", container!="", image!="", service="prod-kube-prometheus-stack-kubelet"} * on(namespace,pod) group_left(workload, workload_type) namespace_workload_pod:kube_pod_owner:relabel{...}) by (pod)

2. Deduplicate Prometheus scrapes (follow-up)

Investigate and remove the duplicate kubelet scrape configs:

• dg-prometheus-stack-kubelet — likely installed with Deepgram self-hosted Helm chart
• prod-omi-kube-prometheus-s-kubelet — likely a second prometheus-stack install

Note: This triple-counting likely affects ALL memory/CPU alerts across the cluster, not just backend-listen.

Verification

After fix, the hottest pod should report ~27% (not ~82%).

Driver: @mon-agent
CC: @thaingnguyen

[beastoin]

CC @thainguyensunya — heads up on upcoming monitoring changes:

1. Alert query fix: Adding service="prod-kube-prometheus-stack-kubelet" filter to the "Backend-listen Memory usage is high" alert to fix triple-counting from duplicate Prometheus scrape services
2. Duplicate scrape cleanup: The cluster has 3 kubelet scrape services (prod-kube-prometheus-stack-kubelet, dg-prometheus-stack-kubelet, prod-omi-kube-prometheus-s-kubelet) reporting identical cAdvisor metrics. The dg-prometheus-stack one appears to be a leftover from the Deepgram self-hosted Helm install. This triple-counting likely affects all memory/CPU alerts cluster-wide, not just backend-listen.

I'll proceed with the alert query fix. The scrape deduplication may need your input on which prometheus-stack installs to keep vs remove.

[beastoin]

Alert query fix applied

Done. Updated the Grafana alert rule eew9yw583gnwgd ("Backend-listen Memory usage is high"):

• Added service="prod-kube-prometheus-stack-kubelet" filter to container_memory_working_set_bytes in the numerator
• Verified fixed query: max pod utilization now shows ~27% (was falsely reporting ~82%)
• 0 pods above 70% threshold — alert is no longer firing
• 0 firing alerts total across the cluster

Remaining work: Deduplicating the 3 Prometheus kubelet scrape services is a larger task that may affect other alerts cluster-wide. Leaving that for @thainguyensunya to assess.