[BUG] - pyspark-notebook no longer builds for Spark 3.1.3, 3.1.2, 2.4.8

[ionicsolutions]

What docker image(s) are you using?

pyspark-notebook

OS system and architecture running docker image

any

What Docker command are you running?

docker build ./docker-stacks/pyspark-notebook
  --build-arg spark_version="3.1.3" \
  --build-arg spark_checksum="2AF22A096CAA616F7C2045D2CA2291DCBBB9112233434C2249944FBD6CE85AF58743A479A86BA8454E86EF3A3D5730664466EE6D12D05AB03C9F2128CCD8AB1B" \
  --build-arg openjdk_version="11" \
  --build-arg hadoop_version="3.2"

How to Reproduce the problem?

Building the pyspark-notebook Docker image for 3.1.3 (as well as 3.1.2 or 2.4.8) is no longer possible as PR #1727 introduced a mandatory ${scala_version} variable to the file path that's used to download the respective Spark archive:

docker-stacks/pyspark-notebook/Dockerfile

Line 35 in aeb9402

RUN wget -q "https://archive.apache.org/dist/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}-scala${scala_version}.tgz" && \

that is not applicable to these older versions of Spark.

Neither 3.1.3 nor 3.1.2 are offered by the Apache Spark project for specific Scala versions (https://archive.apache.org/dist/spark/spark-3.1.3/ and https://archive.apache.org/dist/spark/spark-3.1.2/), nor are the Hadoop-binaries for 2.4.8 provided with specific Scala versions (https://archive.apache.org/dist/spark/spark-2.4.8/).

Command output

No response

Expected behavior

The notebook should build for all versions of Spark, at least the ones that are actively supported/maintained by the Apache Spark project.

Actual behavior

File download fails because the specified file simply does not exist.

Anything else?

This regression breaks builds for the Anovos library (cf. https://github.com/anovos/anovos/runs/7518229159?check_suite_focus=true)

[ionicsolutions]

Pinging @Bidek56, @mathbunnyru as the author/reviewer of PR #1727 that appears to have introduced this breaking change.

[Bidek56]

@ionicsolutions It's not a bug it's a known feature that is documented.
The reason for the change is the fact that Apache started to release builds for a specific Scala version.
If you need to build with prior version of Spark, please use a prior version of Dockerfile from this repo.

[ionicsolutions]

Thanks for the quick response! I'm aware that Apache started to release builds for specific Scala versions but don't you think that at least the currently maintained versions of Spark should be supported? The Anovos project was quite surprised by the sudden failure of their notebook builds, everything used to be fine until about a month ago.

Even 3.2.2 comes with archives that do not specifiy a Scala version in the filename, but assume 2.12 to be the default: https://archive.apache.org/dist/spark/spark-3.2.2/

Would it be conceivable to add a simple "if $scala_version is not empty add scala specification to filepath else leave out" statement to the Dockerfile? I'm sure there are plenty of downstream consumers that would appreciate that.

[Bidek56]

It makes sense, want to do a PR? Thanks

[Bidek56]

@ionicsolutions How would you structure the if?
Because we can have Spark 3.3 with Scala 2.12 or 2.13 but older versions only work with Scala 2.12
We want to default to Scala 2.13 since it's the latest but for older versions we need to use Scala 2.12 and Java 11, not 17.
It's gets complex quickly and we need to make sure it works for all. :)

[ionicsolutions]

@Bidek56 I'll see whether there's a way to solve it at least for 3.1 and 3.2. Or we need to convince the Apache Spark folks to add some symlinks ;-)

[Bidek56]

Asking Apache team to make a change will be a challenge but good luck.
I think you are better off asking end users to upgrade to Spark 3.3 and Scala 2.13. :)

[bjornjorgensen]

"pyspark-notebook no longer builds for Spark 3.1.3, 3.1.2, 2.4.8"

It`s only Spark 3.1.3 from yours list that are supported by Spark for now.

Any reasons you need outdated software?

[ionicsolutions]

@bjornjorgensen 3.2.2 doesn't work either, it has a similar issue (Apache doesn't specify Scala 2.12 for 3.2.x in the filename, they just omit the Scala version entirely).

That said, the Anovos project by Mobilewalla (for which I'm contracted as an external advisor) supports older versions of Spark as they are apparently still in use in many enterprise settings. They even support 2.4.x... Hence, they'd like to build Docker images for these legacy versions as well. The docker-stacks Dockerfiles have served them really well so far as a straightforward, reliable way to build Jupyter environments for various Spark/Java/Python combinations.

If you're only interested in supporting the latest version of Spark and not be backwards-compatible, I can fully understand this from your point of view as maintainers. When I filed the issue, I wasn't aware that the breaking change in #1727 was a conscious decision. Just let me know :-)

[bjornjorgensen]

@Bidek56 SO dockerfile if else condition with external arguments

[Bidek56]

@bjornjorgensen How would you structure the if? Since we need to support multiple Scala, Spark and Java versions.

[ionicsolutions]

@Bidek56 I think the only issue that needs to be solved that if Scala is set to 2.12 (or is left blank) it has to be omitted from the filename for versions prior to 3.3.x. Then it would be possible to install all currently maintained versions of Spark in the various configurations made available by the Apache Spark project.

Alternatively, we could use the "target" method advertised on SO. This would make the Dockerfile a bit longer but would have the benefit that no complicated nested ifs need to be written and each supported minor Spark version gets its own section that can easily be adapted/removed as its status changes.