[SPARK-46945][K8S] Add spark.kubernetes.legacy.useReadWriteOnceAccessMode for old K8s clusters

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to introduce a legacy configuration for K8s PVC access mode to mitigate migrations issues in old K8s clusters.

Why are the changes needed?

• The default value of spark.kubernetes.legacy.useReadWriteOnceAccessMode is false in Apache Spark 4.
• To help the users who cannot upgrade their K8s versions.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manually.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

Hi, @yaooqinn . Could you review this, please?

The recent Apache Spark 3.5.1 release discussion in dev mailing list reminds me your error reporting. I added this legacy configuration with different default values.

• false in master
• true in branch-3.5 and branch-3.4.

[yaooqinn]

Looks cool to have it.

[pan3793]

... in old K8s clusters

I suppose the "old" means K8s version < 1.22? would be helpful to mention it in the PR description or configuration docs.

[yaooqinn]

Thanks @dongjoon-hyun @pan3793

Merged to master.

[dongjoon-hyun]

Thank you, @yaooqinn and @pan3793 .

[dongjoon-hyun]

Hi, @pan3793 . Apache Spark 4.0 only supports K8s 1.26+ already via SPARK-44119

• [SPARK-44119][K8S][DOCS] Drop K8s v1.25 and lower version support #43069

I suppose the "old" means K8s version < 1.22? would be helpful to mention it in the PR description or configuration docs.

[dongjoon-hyun]

I believe the user will know the capability at the first Spark 4.0 run if their cluster is not ready for ReadWriterOncePod. When Apache Spark 4.0.0 comes, I expect K8s v1.27+ mostly because of the following end of support plan in the public cloud.

K8s	AKS	GKE	EKS
1.26	2024-03	2024-06	2024-06

[pan3793]

I believe the user will know the capability at the first Spark 4.0 run if their cluster is not ready for ReadWriterOncePod.

That makes sense, thanks for the explanation.