Skip to content

Conversation

nopzdk
Copy link
Contributor

@nopzdk nopzdk commented Sep 18, 2025

updated spark version and modified download url to dlcdn for speedier download.

updated spark version and modified download url to dlcdn for speedier download.
FROM alpine:3.22

ENV SPARK_VERSION=3.5.1 \
ENV SPARK_VERSION=3.5.6 \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

even upstream only supports 3.5.1
so I am not going to merge above, it would need more extensive testing, checking what changed if there aren't any incompats

@nopzdk can you take above change away?
I will be able to merge the rest

@tarzanek
Copy link
Contributor

the spark upgrade has to be a new PR
we could try to get to 3.5.7, but besides the change you try here
we should change it for all testing ( so e.g. https://github.com/scylladb/scylla-migrator/blob/master/dockerfiles/spark/Dockerfile#L3 and then in all other places, even binaries depend on 3.5.1 - https://github.com/scylladb/scylla-migrator/blob/master/build.sbt#L5 , so spark upgrade HAS to be a separate change - I checked changelogs and for this repo the change seems OK, but we'd need first update https://github.com/scylladb/spark-scylladb-connector/blob/scylla-4.x/project/Versions.scala#L17 , along with jetty (it was updated in 3.5.x) - so there the change might be harder)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants