Investigating Unexpected Memory Usage in Kubernetes Pods Causing Restarts

[Kaushikdhola]

I’m encountering an issue with memory management in my Kubernetes setup, where specific pods are consuming unexpected amounts of memory, leading to frequent restarts. I’d appreciate any guidance on diagnosing this issue and understanding what might be causing the memory consumption.

Problem Description

In our current Kubernetes setup, certain pods are writing heavily to memory, eventually consuming the allotted memory limits, which causes them to be terminated and restarted by the system. This behavior disrupts the workload, and we are unclear on what specifically is writing to memory or why this memory consumption is happening.

Since my end goal is to store this data on a Persistent Volume (PVC) instead, if we can identify the process causing this memory usage, I could redirect it to disk storage and avoid filling up memory.

Here’s what I have attempted to understand and troubleshoot this issue:

Using kubectl describe and kubectl top:

• Used kubectl describe pod to check for any signs of memory leaks or OOMKill events. Executed kubectl top pod to monitor current memory consumption and identify spikes.

Setting Resource Limits:

• Defined resource limits in the pod’s YAML file to ensure it doesn’t exceed a set amount of memory. However, the pod still eventually reaches this limit, indicating continuous high memory use.

In-pod Process Analysis:

• Accessed the pod with kubectl exec and ran top -o %MEM to identify any memory-heavy processes. While this provides visibility into memory usage, it has not yet clarified why the process is writing so heavily to memory.

Questions:

• Is there an internal method or Kubernetes-native option to trace memory usage within a pod to pinpoint which processes or files are consuming memory?
• If not, are there any recommended debugging techniques using Kubernetes features that could help identify memory usage sources?

[tengqm]

Is there an internal method or Kubernetes-native option to trace memory usage within a pod to pinpoint which processes or files are consuming memory?

You can use the /metrics/resource API endpoint on the kubelet to check the pod_memory_working_set_bytes metric. The metrics have the labels like namespace and pod.
This API is also exposed on the API server, using a proxy method.
Checking the data on the API server saves you some effort related to node authn/authz.

You can use other monitoring tools like Prometheus if that suits your need.