[SPARK-52399][Kubernetes] Support local ephemeral-storage resource

[ashangit]

What changes were proposed in this pull request?

Add capability to set the local ephemeral-storage resource on driver and executor pods

Why are the changes needed?

On kubernetes when a node is low on ephemeral storage the kubelete will killed pods using too much ephemeral storage (more than the requested one). As driver and executor does not set any ephemeral storage they can be evicted even if the big usage comes from other pods
Being able to set the ephemeral storage requests will ensure pods are not evicted

Does this PR introduce any user-facing change?

Yes added spark.kubernetes.driver.request.ephemeral.storage and spark.kubernetes.executor.request.ephemeral.storage those 2 new configs.
But if those configs are not set the spark jobs will works as it was before without setting the ephemeral storage

How was this patch tested?

Added some unittest and also deployed a spark job on our kubernetes cluster using those parameters.
The ephemeral storage was well added to the driver and executor:

% k get po -o yaml test-nico-driver|grep -A 10 resources 
--
    resources:
      limits:
        cpu: "2"
        ephemeral-storage: 5Gi
        memory: 8692Mi
      requests:
        cpu: "2"
        ephemeral-storage: 5Gi
        memory: 8692Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
% k get po -o yaml test-nico-0aeee3973a4d6156-exec-1|grep -A 10 resources
--
    resources:
      limits:
        cpu: "2"
        ephemeral-storage: 50Gi
        memory: 9716Mi
      requests:
        cpu: "2"
        ephemeral-storage: 50Gi
        memory: 9716Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File

Was this patch authored or co-authored using generative AI tooling?

No

[ashangit]

Not sure which release I should target (same on other doc/config section)

[dongjoon-hyun]

Thank you for making a PR, @ashangit .

However, Apache Spark 3.0+ has been providing a more generalized way to control this kind of K8s specific setting via Pod Template instead of adding many configurations one by one. There are so many things to control like this. For example, priorityClassName.

• #22146

Please make and use pod template like the following, pod.yml.

apiVersion: v1
Kind: Pod
spec:
  containers:
  - name: pod
    resources:
      limits:
        ephemeral-storage: 10Gi

You can use like this in Apache Spark .

-c spark.kubernetes.driver.podTemplateFile=pod.yml
-c spark.kubernetes.executor.podTemplateFile=pod.yml

[ashangit]

Hi, relying on pod template doesn't seem to work when trying to modify the spark-kubernetes-driver/spark-kubernetes-executor as spark is adding it's generated pod config to the list of containers, see spark/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala at 80657974ee691a36c5a761629260a65b2fb7fa03 · apache/spark and spark/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala at 80657974ee691a36c5a761629260a65b2fb7fa03 · apache/spark

Due to this it ends up by trying to submit a pod with 2 times the spark-kubernetes-* container.
Ex. of error log thrown by the kube api: Pod "test-nico-driver" is invalid: [spec.containers[1].image: Required value, spec.containers[2].name: Duplicate value: "spark-kubernetes-driver"]

[dongjoon-hyun]

Hi, relying on pod template doesn't seem to work when trying to modify the spark-kubernetes-driver/spark-kubernetes-executor as spark is adding it's generated pod config to the list of containers, see spark/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala at 80657974ee691a36c5a761629260a65b2fb7fa03 · apache/spark and spark/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala at 80657974ee691a36c5a761629260a65b2fb7fa03 · apache/spark

Due to this it ends up by trying to submit a pod with 2 times the spark-kubernetes-* container. Ex. of error log thrown by the kube api: Pod "test-nico-driver" is invalid: [spec.containers[1].image: Required value, spec.containers[2].name: Duplicate value: "spark-kubernetes-driver"]

I'm interested in your bug reports. Could you file a JIRA issue with the above one with the procedure you did, please?

FYI, Apache Spark has a pod template integration test here. We can improve the test coverage first.

• PodTemplateSuite: https://github.com/apache/spark/blob/master/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/PodTemplateSuite.scala

[ashangit]

Hi, Found the issue
I was relying on this pod template which was not working

spec:
  containers:
  - name: spark-kubernetes-driver
    resources:
      limits:
        ephemeral-storage: 10Gi
...

Relying on this one work fine:

spec:
  containers:
  - resources:
      limits:
        ephemeral-storage: 10Gi

[dongjoon-hyun]

Great! Thank you for sharing and closing the PR, @ashangit .