Pending pod triggers new node instead of evict a pod with lower priority

[mmingorance-dh]

Hi all,
Yesterday we started to test cluster-autoscaler with priorityClasses and podPriority to always have some available extra capacity in our cluster, however whenever a new pod comes up and is in pending state, this one triggers a new node with cluster-autoscaler instead of replacing any pod from the "paused" deployment running with lower priorityClass.

This is configuration I added to my cluster:

    authorizationMode: RBAC
    authorizationRbacSuperUser: admin
    runtimeConfig:
      scheduling.k8s.io/v1alpha1: "true"
      admissionregistration.k8s.io/v1beta1: "true"
      autoscaling/v2beta1: "true"

kubelet:
    featureGates:
      PodPriority: "true"
 
 
kubeAPIServer:
    featureGates:
      PodPriority: "true"

kubeAPIServer:
   admissionControl:
   - Priority
   - NamespaceLifecycle
    - LimitRanger
    - ServiceAccount
    - PersistentVolumeLabel
    - DefaultStorageClass
    - ResourceQuota
    - DefaultTolerationSeconds

kubeControllerManager:
    horizontalPodAutoscalerDownscaleDelay: 1h0m0s
    horizontalPodAutoscalerUseRestClients: true

As I could see in my masters, these features seems to be enabled:
--feature-gates=PodPriority=true
--enable-admission-plugins=Priority,NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,ValidatingAdmissionWebhook,NodeRestriction,ResourceQuota

Is there anything else that I'm missing in my config?
The overscaling deployment is the same one you can find in cluster-autoscaler FAQ.

[losipiuk]

Thanks for reporting that.

Are the "paused" pods actually being created?
Please ensure that there is actually a single node with enough "paused" pods with lower priority, that preempting them will free enough resources to schedule the "workload" pod.
If both are true the problem is with scheduler configuration I guess. If not we need to fix that first :)

[mmingorance-dh]

Thanks for helping.

The "paused" pods are getting created and they take over of 6000m CPU and 10000Mi Memory each.
I run 10 pods in the deployment, so I guess that should be enough space for any other pod of our private applications to be created.

By the way, we are using kops 1.10 and cluster-autoscaler 1.3.0

[losipiuk]

so I guess that should be enough space for any other pod of our private applications to be created.

I don't know :) Depends how be resource requests the applications have.
Also to be sure. The application pods have the priority set to default (0)?

Also could you please check if nominatedNodeName is set in status on "workload" pods?

[losipiuk]

I am not sure if that is possible but you may also try to run scheduler with log verbosity set to at least 3 and look for preemption related log messages.
This way we could rule out that preemption is not enabled.
If that is the case you should see "Pod priority feature is not enabled or preemption is disabled by scheduler configuration." in logs.

Btw. Which version of k8s and CA are you using?

[mmingorance-dh]

we are using kops 1.10 (with Kubernetes 1.10.6) and cluster-autoscaler 1.3.0
I found out that in Kops you can also enable the following field:

kubeScheduler:
   featureGates
     podPriority: "true"

Maybe that's the reason why my "paused" pods are not being rescheduled. I will enable this field and I'll let you know.

[mmingorance-dh]

Regarding the application pods's priority, we have defined a default priorityClass in our cluster, therefore all new pods will get this class as default.
Also I read in the Kubernetes documentation that when enabling podPriority, all existing pods automatically gets priority 0 set.

[mmingorance-dh]

I found out how to make it works. There are a couple of parameters more to set than what cluster-autoscaler describes.
This is the right configuration for this:

kubeAPIServer:
  runtimeConfig:
      scheduling.k8s.io/v1alpha1: "true"
      admissionregistration.k8s.io/v1beta1: "true"
      autoscaling/v2beta1: "true"
  admissionControl:
  - Priority
  featureGates:
    PodPriority: "true"
kubelet:
  featureGates:
     PodPriority: "true"
kubeScheduler:
  featureGates:
    PodPriority: "true"
kubeControllerManager:
  horizontalPodAutoscalerUseRestClients: true
  featureGates:
     PodPriority: "true"

That will enable podPriority and Preemption in your cluster.

Thank you for helping me!

[aleksandra-malinowska]

@mmingorance-dh thanks for posting the solution!

[aarongorka]

@mmingorance-dh there are a few typos in your yaml (missing colon, duplicated keys and inconsistent case), should be:

  kubeAPIServer:
    runtimeConfig:
        scheduling.k8s.io/v1alpha1: "true"
        admissionregistration.k8s.io/v1beta1: "true"
        autoscaling/v2beta1: "true"
    admissionControl:
      - Priority
    featureGates:
      PodPriority: "true"
  kubelet:
    featureGates:
       PodPriority: "true"
  kubeScheduler:
    featureGates:
      PodPriority: "true"
  kubeControllerManager:
    horizontalPodAutoscalerUseRestClients: true
    featureGates:
       PodPriority: "true"

[mmingorance-dh]

@aarongorka thanks for catching that. I just updated my comment as well.

[mmingorance-dh]

Updated.
Thanks @aarongorka

[njfix6]

@mmingorance-dh Which config file do I set that config in? the kops config or the cluster auto scaler config?