[SPARK-51250][K8S] Add Support for K8s PriorityClass Configuration fo…

[zemin-piao]

What changes were proposed in this pull request?

• Introduce a new spark configuration spark.kubernetes.priorityclass.name.
• Create a new FeatureStep class PriorityClassFeatureStep that passes configured priority class name to driver and executor pods.
• Tests for PriorityClassFeatureStep.scala

Why are the changes needed?

Currently when running spark on k8s, with spark-submit approach, in order to specify priority class name, the only possible way is to specify in the pod template. In a cluster with numerous priority classes, it means multiple pod template files are to be used, each of which has variant priority class name only.

One could dynamically generate the pod templates based on priority classes, however this introduces additional overhead on user side (e.g. building a process to dynamically generate templates just for priority class name, making sure dynamically generated templates are still valid)

Furthermore, for SPARK on YARN there is a configuration spark.yarn.priority that is flexible and easy to use for users. It is good to achieve the similar level of simplicity for users moving from SPARK on YARN to SPARK on K8s.

Does this PR introduce any user-facing change?

Yes. With this PR, users can use spark.kubernetes.priorityclass.name to specify the priority class name for driver and executor pods.

How was this patch tested?

Tests were added in this pull request

Was this patch authored or co-authored using generative AI tooling?

No

[zemin-piao]

Bump :P

[zemin-piao]

@yaooqinn if you have time could you maybe have a look? Thanks a lot!

[yaooqinn]

Kubernetes has a large number of APIs, and I'm quite sure which ones should be added to the Spark defaults. I think using spark.kubernetes.driver.pod.featureSteps is beneficial.

Also cc @dongjoon-hyun

[dongjoon-hyun]

I'm not sure about this part of the PR claim. When does this happen, @zemin-piao ?

In a cluster with numerous priority classes

[zemin-piao]

Kubernetes has a large number of APIs, and I'm quite sure which ones should be added to the Spark defaults. I think using spark.kubernetes.driver.pod.featureSteps is beneficial.

Also cc @dongjoon-hyun

Indeed in the released this is how I did priority class setting by using the customised feature step. I made this PR with this new spark config, with the thought that setting priority class is a generic thing for spark on k8s.

[zemin-piao]

I'm not sure about this part of the PR claim. When does this happen, @zemin-piao ?

In a cluster with numerous priority classes

IMO it happens when multiple business criticalities needs to be defined within a cluster. This can also happen if in a cluster, a value based prioritisation needs to happen.

[zemin-piao]

@yaooqinn @dongjoon-hyun thanks for your questions. Lemme know if more clarification and discussion needed