Kubernetes Best Practices for Pods Resizing (setting right resources)

Resources of User Pods

• Memory Request = [Appropriate value based on the usage over last 30 days or more identified using Prometheus and Grafana]
• CPU Request = [Appropriate value based on the usage over last 30 days or more identified using Prometheus and Grafana]
• Memory Limit = Memory Request
  • Only set Memory Limit higher than the Memory Request for those pods which rarely have intense spikes.
    • Those pods can be identified based on the trend over time using Prometheus and Grafana and/or the nature of the pod.
• CPU Limit = None
  • Only set CPU Limit for those pods which consume all of the CPU resources.
    • Those pods can be identified based on the trend over time using Prometheus and Grafana and/or the alerts/events related to node failure or pod eviction.

Pod Eviction Order

• Keeping in mind the pod eviction order below, based on the nature of pod, assign the right QoS class which is based on CPU and Memory requests and limits set and assign the right priority class which is based an integer value assigned to a class which is then assigned to a pod.

Based on QoS class (Quality of Service Class)

• Evicted first - BestEffort pods – None of the containers in the pod have any CPU and Memory Request and Limit set.
• Evicted next - Burstable pods - At least one container in the pod has a Memory or CPU Request or Limit.
  • Those whose usage exceeds their requests are evicted before those that are within their requests.
• Evicted last - Guaranteed pods – Every container in the pod has CPU Request = CPU Limit and Memory Request = Memory Limit.

Based on PriorityClass

• All pods in each QoS class (BestEffort, Burstable, Guaranteed) are sorted based on PriorityClass where lowest gets evicted first and highest gets scheduled first when all other things are equal among the pods.

Resources of System Pods

• To make sure that Kubernetes system daemons (e.g., kubelet, container runtime, etc) and OS system daemons (e.g., sshd, udev, etc) don't starve for resources, we can update resources reserved for them as well as eviction thresholds using Node Allocatable Kubelet configuration setting. It is enabled by default but we can increase the values. It covers CPU, Memory and Storage.
  • For example, in case of AWS EKS, in user data of AWS EC2 launch template of each AWS EKS node group, update bootstrap command with --kubelet-extra-args for kubeReserved, systemReserved, and evictionHard as follows: /etc/eks/bootstrap.sh my-cluster --kubelet-extra-args '--kube-reserved=cpu=200m,memory=500Mi,ephemeral-storage=1Gi --system-reserved=cpu=200m,memory=500Mi,ephemeral-storage=1Gi --eviction-hard=memory.available<100Mi,nodefs.available<10%'

Resources of Nodes

• Choose the right instance type and right number of instances based on the usage over last 30 days or more identified using Prometheus and Grafana.