[SPARK-38262][BUILD] Upgrade Google guava to version 30.0-jre #35584
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Can one of the admins verify this patch? |
medb
reviewed
Feb 21, 2022
| flatbuffers-java/1.12.0//flatbuffers-java-1.12.0.jar | ||
| generex/1.0.2//generex-1.0.2.jar | ||
| gmetric4j/1.0.10//gmetric4j-1.0.10.jar | ||
| gson/2.2.4//gson-2.2.4.jar | ||
| guava/14.0.1//guava-14.0.1.jar | ||
| guava/30.0-jre//guava-30.0-jre.jar |
There was a problem hiding this comment.
Any reason to not use the latest 31.0.1-jre version?
There was a problem hiding this comment.
Yes, this PR is just to get rid of two CVE. I do not want to destroy existing code base. If you are a developer who needs and uses some of what is in newer code, then I recommend that you implement this yourself when you need it.
|
In fact, many people have tried similar jobs, such as SPARK-36676, but due to the cascading dependence of third-party libraries on Guava 14.0.1 (such as hive-2.3.9 and hadoop-2.7.4), in order to ensure the compatibility, it is not possible to upgrade the Guava version directly like this pr. |
|
@LuciferYang Thank you :) I see that none of the other PRs that can be found have come up with the fact that today there are two CVEs for Google guava. What's a little weird about this is that I chose version 30.0-jre which is the first one without any known security holes. All the tests have passed. The difference is that for example 29326 or 33989 the tests fail. |
|
@bjornjorgensen Can you test with |
|
also ping @sunchao |
|
@LuciferYang Oh yes, that might explain why my tests get green. But if we look at the problem differently. We actually use hadoop-common-2.7.4.jar |
|
Not only Hadoop, but Hive 2.3.x also has dependency on Guava 14.0.1. For Hadoop we can at least change Guava version accordingly but there's no workaround for Hive. |
|
I have updated the JIRA for this issue SPARK-38262 I will close this PR now. Thanks to @medb @LuciferYang and @sunchao for the help on this one :) |
Closed
dongjoon-hyun
pushed a commit
that referenced
this pull request
Sep 12, 2024
### What changes were proposed in this pull request? This PR upgrades Spark's built-in Guava from 14 to 33.2.1-jre Currently, Spark uses Guava 14 because the previous built-in Hive 2.3.9 is incompatible with new Guava versions. HIVE-27560 (apache/hive#4542) makes Hive 2.3.10 compatible with Guava 14+ (thanks to LuciferYang) ### Why are the changes needed? It's a long-standing issue, see prior discussions at #35584, #36231, and #33989 ### Does this PR introduce _any_ user-facing change? Yes, some user-faced error messages changed. ### How was this patch tested? GA passed. Closes #42493 from pan3793/guava. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
attilapiros
pushed a commit
to attilapiros/spark
that referenced
this pull request
Oct 4, 2024
### What changes were proposed in this pull request? This PR upgrades Spark's built-in Guava from 14 to 33.2.1-jre Currently, Spark uses Guava 14 because the previous built-in Hive 2.3.9 is incompatible with new Guava versions. HIVE-27560 (apache/hive#4542) makes Hive 2.3.10 compatible with Guava 14+ (thanks to LuciferYang) ### Why are the changes needed? It's a long-standing issue, see prior discussions at apache#35584, apache#36231, and apache#33989 ### Does this PR introduce _any_ user-facing change? Yes, some user-faced error messages changed. ### How was this patch tested? GA passed. Closes apache#42493 from pan3793/guava. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
himadripal
pushed a commit
to himadripal/spark
that referenced
this pull request
Oct 19, 2024
### What changes were proposed in this pull request? This PR upgrades Spark's built-in Guava from 14 to 33.2.1-jre Currently, Spark uses Guava 14 because the previous built-in Hive 2.3.9 is incompatible with new Guava versions. HIVE-27560 (apache/hive#4542) makes Hive 2.3.10 compatible with Guava 14+ (thanks to LuciferYang) ### Why are the changes needed? It's a long-standing issue, see prior discussions at apache#35584, apache#36231, and apache#33989 ### Does this PR introduce _any_ user-facing change? Yes, some user-faced error messages changed. ### How was this patch tested? GA passed. Closes apache#42493 from pan3793/guava. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
prabhjyotsingh
pushed a commit
to acceldata-io/spark3
that referenced
this pull request
Feb 8, 2025
This PR upgrades Spark's built-in Guava from 14 to 33.2.1-jre Currently, Spark uses Guava 14 because the previous built-in Hive 2.3.9 is incompatible with new Guava versions. HIVE-27560 (apache/hive#4542) makes Hive 2.3.10 compatible with Guava 14+ (thanks to LuciferYang) It's a long-standing issue, see prior discussions at apache#35584, apache#36231, and apache#33989 Yes, some user-faced error messages changed. GA passed. Closes apache#42493 from pan3793/guava. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 1f24b2d)
prabhjyotsingh
pushed a commit
to acceldata-io/spark3
that referenced
this pull request
Feb 8, 2025
This PR upgrades Spark's built-in Guava from 14 to 33.2.1-jre Currently, Spark uses Guava 14 because the previous built-in Hive 2.3.9 is incompatible with new Guava versions. HIVE-27560 (apache/hive#4542) makes Hive 2.3.10 compatible with Guava 14+ (thanks to LuciferYang) It's a long-standing issue, see prior discussions at apache#35584, apache#36231, and apache#33989 Yes, some user-faced error messages changed. GA passed. Closes apache#42493 from pan3793/guava. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 1f24b2d)
prabhjyotsingh
pushed a commit
to acceldata-io/spark3
that referenced
this pull request
Feb 8, 2025
This PR upgrades Spark's built-in Guava from 14 to 33.2.1-jre Currently, Spark uses Guava 14 because the previous built-in Hive 2.3.9 is incompatible with new Guava versions. HIVE-27560 (apache/hive#4542) makes Hive 2.3.10 compatible with Guava 14+ (thanks to LuciferYang) It's a long-standing issue, see prior discussions at apache#35584, apache#36231, and apache#33989 Yes, some user-faced error messages changed. GA passed. Closes apache#42493 from pan3793/guava. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 1f24b2d)
prabhjyotsingh
pushed a commit
to acceldata-io/spark3
that referenced
this pull request
Feb 8, 2025
This PR upgrades Spark's built-in Guava from 14 to 33.2.1-jre Currently, Spark uses Guava 14 because the previous built-in Hive 2.3.9 is incompatible with new Guava versions. HIVE-27560 (apache/hive#4542) makes Hive 2.3.10 compatible with Guava 14+ (thanks to LuciferYang) It's a long-standing issue, see prior discussions at apache#35584, apache#36231, and apache#33989 Yes, some user-faced error messages changed. GA passed. Closes apache#42493 from pan3793/guava. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 1f24b2d) (cherry picked from commit e5cc252)
shubhluck
pushed a commit
to acceldata-io/spark3
that referenced
this pull request
May 16, 2025
This PR upgrades Spark's built-in Guava from 14 to 33.2.1-jre Currently, Spark uses Guava 14 because the previous built-in Hive 2.3.9 is incompatible with new Guava versions. HIVE-27560 (apache/hive#4542) makes Hive 2.3.10 compatible with Guava 14+ (thanks to LuciferYang) It's a long-standing issue, see prior discussions at apache#35584, apache#36231, and apache#33989 Yes, some user-faced error messages changed. GA passed. Closes apache#42493 from pan3793/guava. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 1f24b2d)
senthh
pushed a commit
to acceldata-io/spark3
that referenced
this pull request
May 26, 2025
This PR upgrades Spark's built-in Guava from 14 to 33.2.1-jre Currently, Spark uses Guava 14 because the previous built-in Hive 2.3.9 is incompatible with new Guava versions. HIVE-27560 (apache/hive#4542) makes Hive 2.3.10 compatible with Guava 14+ (thanks to LuciferYang) It's a long-standing issue, see prior discussions at apache#35584, apache#36231, and apache#33989 Yes, some user-faced error messages changed. GA passed. Closes apache#42493 from pan3793/guava. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 1f24b2d)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Bump Google guava from version 14.0.1 to 30.0-jre
Release notes for Google guava 30.0
Why are the changes needed?
Spark is using com.google.guava:guava version 14.0.1 which has two security issues.
CVE-2018-10237
CVE-2020-8908
Does this PR introduce any user-facing change?
NO
How was this patch tested?
All existing tests must pass.