Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Golang to 1.23 and Spark to 3.5.2 #2150

Closed
wants to merge 2 commits into from
Closed

Update Golang to 1.23 and Spark to 3.5.2 #2150

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
aae6455
Upgrade Go from 1.22.5 to 1.23.0
jacobsalway Sep 3, 2024
6f7336b
Update Spark from 3.5.0 to 3.5.2
jacobsalway Sep 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Update Spark from 3.5.0 to 3.5.2 
Signed-off-by: Jacob Salway <jacob.salway@gmail.com>
  • Loading branch information
@jacobsalway
jacobsalway committed Sep 9, 2024
commit 6f7336bcf05e321208f4d5017803d8c65238719d
4 changes: 2 additions & 2 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,9 @@
# limitations under the License.
#

ARG SPARK_IMAGE=spark:3.5.0
ARG SPARK_IMAGE=spark:3.5.2

FROM golang:1.23.0 AS builder
FROM golang:1.23 AS builder

WORKDIR /workspace

Expand Down
10 changes: 5 additions & 5 deletions config/samples/v1beta1_sparkapplication.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,16 @@ metadata:
spec:
type: Scala
mode: cluster
image: spark:3.5.0
image: spark:3.5.2
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.0.jar
sparkVersion: 3.5.0
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
sparkVersion: 3.5.2
driver:
labels:
version: 3.5.0
version: 3.5.2
serviceAccount: spark-operator-spark
executor:
labels:
version: 3.5.0
version: 3.5.2
instances: 1
10 changes: 5 additions & 5 deletions config/samples/v1beta2_scheduledsparkapplication.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,23 +11,23 @@ spec:
template:
type: Scala
mode: cluster
image: spark:3.5.0
image: spark:3.5.2
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.0.jar
sparkVersion: 3.5.0
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
sparkVersion: 3.5.2
restartPolicy:
type: Never
driver:
labels:
version: 3.5.0
version: 3.5.2
cores: 1
coreLimit: 1200m
memory: 512m
serviceAccount: spark-operator-spark
executor:
labels:
version: 3.5.0
version: 3.5.2
instances: 1
cores: 1
coreLimit: 1200m
Expand Down
10 changes: 5 additions & 5 deletions config/samples/v1beta2_sparkapplication.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,16 @@ metadata:
spec:
type: Scala
mode: cluster
image: spark:3.5.0
image: spark:3.5.2
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.0.jar
sparkVersion: 3.5.0
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
sparkVersion: 3.5.2
driver:
labels:
version: 3.5.0
version: 3.5.2
serviceAccount: spark-operator-spark
executor:
labels:
version: 3.5.0
version: 3.5.2
instances: 1
10 changes: 5 additions & 5 deletions examples/spark-pi-configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,11 @@ metadata:
spec:
type: Scala
mode: cluster
image: spark:3.5.0
image: spark:3.5.2
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.0.jar
sparkVersion: 3.5.0
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
sparkVersion: 3.5.2
restartPolicy:
type: Never
volumes:
Expand All @@ -34,7 +34,7 @@ spec:
name: test-configmap
driver:
labels:
version: 3.5.0
version: 3.5.2
cores: 1
coreLimit: 1200m
memory: 512m
Expand All @@ -44,7 +44,7 @@ spec:
mountPath: /opt/spark/config
executor:
labels:
version: 3.5.0
version: 3.5.2
instances: 1
cores: 1
memory: 512m
Expand Down
10 changes: 5 additions & 5 deletions examples/spark-pi-custom-resource.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,11 @@ metadata:
spec:
type: Scala
mode: cluster
image: spark:3.5.0
image: spark:3.5.2
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.0.jar
sparkVersion: 3.5.0
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
sparkVersion: 3.5.2
restartPolicy:
type: Never
volumes:
Expand All @@ -35,7 +35,7 @@ spec:
type: Directory
driver:
labels:
version: 3.5.0
version: 3.5.2
cores: 1
coreLimit: 1200m
memory: 512m
Expand All @@ -45,7 +45,7 @@ spec:
mountPath: /tmp
executor:
labels:
version: 3.5.0
version: 3.5.2
instances: 1
cores: 1
memory: 512m
Expand Down
10 changes: 5 additions & 5 deletions examples/spark-pi-dynamic-allocation.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,23 +21,23 @@ metadata:
spec:
type: Scala
mode: cluster
image: spark:3.5.0
image: spark:3.5.2
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.0.jar
sparkVersion: 3.5.0
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
sparkVersion: 3.5.2
arguments:
- "50000"
driver:
labels:
version: 3.5.0
version: 3.5.2
cores: 1
coreLimit: 1200m
memory: 512m
serviceAccount: spark-operator-spark
executor:
labels:
version: 3.5.0
version: 3.5.2
instances: 1
cores: 1
coreLimit: 1200m
Expand Down
10 changes: 5 additions & 5 deletions examples/spark-pi-kube-scheduler.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,21 +21,21 @@ metadata:
spec:
type: Scala
mode: cluster
image: spark:3.5.0
image: spark:3.5.2
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.0.jar
sparkVersion: 3.5.0
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
sparkVersion: 3.5.2
driver:
labels:
version: 3.5.0
version: 3.5.2
cores: 1
coreLimit: 1200m
memory: 512m
serviceAccount: spark-operator-spark
executor:
labels:
version: 3.5.0
version: 3.5.2
instances: 2
cores: 1
coreLimit: 1200m
Expand Down
8 changes: 4 additions & 4 deletions examples/spark-pi-python.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,20 +22,20 @@ spec:
type: Python
pythonVersion: "3"
mode: cluster
image: spark:3.5.0
image: spark:3.5.2
imagePullPolicy: IfNotPresent
mainApplicationFile: local:///opt/spark/examples/src/main/python/pi.py
sparkVersion: 3.5.0
sparkVersion: 3.5.2
driver:
labels:
version: 3.5.0
version: 3.5.2
cores: 1
coreLimit: 1200m
memory: 512m
serviceAccount: spark-operator-spark
executor:
labels:
version: 3.5.0
version: 3.5.2
instances: 1
cores: 1
coreLimit: 1200m
Expand Down
10 changes: 5 additions & 5 deletions examples/spark-pi-scheduled.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,23 +25,23 @@ spec:
template:
type: Scala
mode: cluster
image: spark:3.5.0
image: spark:3.5.2
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.0.jar
sparkVersion: 3.5.0
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
sparkVersion: 3.5.2
restartPolicy:
type: Never
driver:
labels:
version: 3.5.0
version: 3.5.2
cores: 1
coreLimit: 1200m
memory: 512m
serviceAccount: spark-operator-spark
executor:
labels:
version: 3.5.0
version: 3.5.2
instances: 1
cores: 1
coreLimit: 1200m
Expand Down
10 changes: 5 additions & 5 deletions examples/spark-pi-volcano.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,21 +21,21 @@ metadata:
spec:
type: Scala
mode: cluster
image: spark:3.5.0
image: spark:3.5.2
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.0.jar
sparkVersion: 3.5.0
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
sparkVersion: 3.5.2
driver:
labels:
version: 3.5.0
version: 3.5.2
cores: 1
coreLimit: 1200m
memory: 512m
serviceAccount: spark-operator-spark
executor:
labels:
version: 3.5.0
version: 3.5.2
instances: 2
cores: 1
coreLimit: 1200m
Expand Down
12 changes: 6 additions & 6 deletions examples/spark-pi-yunikorn.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,25 +21,25 @@ metadata:
spec:
type: Scala
mode: cluster
image: spark:3.5.0
image: spark:3.5.2
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.0.jar
sparkVersion: 3.5.0
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
sparkVersion: 3.5.2
driver:
labels:
version: 3.5.0
version: 3.5.2
cores: 1
coreLimit: 1200m
memory: 512m
serviceAccount: spark-operator-spark
executor:
labels:
version: 3.5.0
version: 3.5.2
instances: 2
cores: 1
coreLimit: 1200m
memory: 512m
batchScheduler: yunikorn
batchSchedulerOptions:
queue: root.default
queue: root.default
10 changes: 5 additions & 5 deletions examples/spark-pi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,21 +21,21 @@ metadata:
spec:
type: Scala
mode: cluster
image: spark:3.5.0
image: spark:3.5.2
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.0.jar
sparkVersion: 3.5.0
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
sparkVersion: 3.5.2
driver:
labels:
version: 3.5.0
version: 3.5.2
cores: 1
coreLimit: 1200m
memory: 512m
serviceAccount: spark-operator-spark
executor:
labels:
version: 3.5.0
version: 3.5.2
instances: 1
cores: 1
coreLimit: 1200m
Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
module github.com/kubeflow/spark-operator

go 1.23.0
go 1.23

require (
cloud.google.com/go/storage v1.43.0
Expand Down
Loading