Spark 4.0.x

[razvan]

Which new version of Apache Spark-on-Kubernetes should we support?

Spark 4.0.0 has been released and it requires a slightly different approach to Kubernetes deployments.

Identified Problems

HBase connector

Open PR for Spark 4 support apache/hbase-connectors#130

Trying locally, doesn't build with Scala 2.13 (scalatest_2.13 doesn't exist):

181.0 [ERROR] Failed to execute goal on project hbase-spark: Could not resolve dependencies for project org.apache.hbase.connectors.spark:hbase-spark:jar:1.0.1-stackable0.0.0-dev: Could not find artifact org.scalatest:scalatest_2.13:jar:3.0.5 in nexus (https://build-repo.stackable.tech/repository/maven-public/) -> [Help 1]

Apache Iceberg

Not currently supported. Watch this issue for updates

Spark Changes

Release notes

This blog post provides a good overview of the changes.

JDK support:

• 11 minimum
• 17 recommended
• 21 supported

Scala support:

• 2.12 dropped
• 2.13 is supported

Some notable Kubernetes related changes since v3.5.6 from git commit log:

❯ git log --oneline --grep kubernetes v3.5.6..v4.0.0
...
14225cb55ba [SPARK-52027][K8S][DOCS] Update `YuniKorn` docs with `1.6.3`
c4d33712448 [SPARK-51000][K8S][DOCS] Drop K8s v1.29 Support
07301ddb889 [SPARK-50135][BUILD] Upgrade ZooKeeper to 3.9.3
620f16ed0af [SPARK-48400][CORE] Promote `PrometheusServlet` to `DeveloperApi`
47c0b55e06d [SPARK-49190][K8S][FOLLOW-UP] Add how to populate executor log url vars with env vars
72fe58d716d [SPARK-47675][K8S][TESTS] Use AWS SDK v2 `2.23.19` in K8s IT

Changes required

Are there any upstream changes that we need to support?
e.g. new features, changed features, deprecated features etc.

Implementation checklist

• Update the Docker image
• Update documentation to include supported version(s)
• Update and test getting started guide with updated version(s)
• Update operator to support the new version (if needed)
• Update integration tests to test use the new versions (in addition or replacing old versions
• Update examples to use new versions

Size

M