chore(spark): update container images for 25.7.0 #1094
Conversation
spark-k8s/versions.py
Outdated
| "hbase": "2.6.1", # current Stackable LTS version | ||
| "aws_java_sdk_bundle": "1.12.367", # https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.3.6 | ||
| "azure_storage": "7.0.1", # https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure/3.3.6 | ||
| "hadoop": "3.4.1", # Hadoop version defined in ../hbase/versions.py to reduce build time and disk requirements |
There was a problem hiding this comment.
Oh nice, I didn't realise we could bump it for an existing version.
Hopefully we can do the same for Hive (otherwise I have to keep 3.3.6 around).
There was a problem hiding this comment.
I'll add a comment in there because it's misleading. Spark is not built against this version at all. This argument is only used to extract the hadoop/cloud connectors from the Hadoop LTS version and add them to the Spark classpath.
CHANGELOG.md
Outdated
| @@ -57,6 +57,7 @@ All notable changes to this project will be documented in this file. | |||
|
|
|||
| - ci: Remove Nexus steps from build, mirror and release workflows ([#1056]). | |||
| Also remove the old release workflow. | |||
| - spark: remove 3.5.2 and update dependencies for 3.5.5 ([#1094]) | |||
There was a problem hiding this comment.
I think update dependencies for 3.5.5 can go under the Changes heading.
|
Currently doesn't build because of missing HBase 2.6.2 image |
spark-k8s/versions.py
Outdated
| "azure_storage": "7.0.1", # https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure/3.3.6 | ||
| "hadoop": "3.4.1", # Current Stackable LTS version. Source of the AWS and Azure artifacts to Spark's classpath. | ||
| "hbase": "2.6.2", # Current Stackable LTS version. Used to build the HBase connector. | ||
| "aws_java_sdk_bundle": "1.12.720", # https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.4.1 |
There was a problem hiding this comment.
Just curious, this mentions:
aws-java-sdk-bundle-1.12.x.jarbecomesbundle-2.x.y.jar
And I do see: bundle-2.24.6.jar in /stackable/hadoop-3.4.1/share/hadoop/tools/lib/.
I only see aws-java-sdk-bundle-1.12.367.jar for hadoop 3.3.6.
There was a problem hiding this comment.
Description
Part of #1080
Definition of Done Checklist
Note
Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant.
Please make sure all these things are done and tick the boxes
TIP: Running integration tests with a new product image
The image can be built and uploaded to the kind cluster with the following commands:
See the output of
baketo retrieve the image tag for<image-tagged-with-the-major-version>.