Skip to content

[SPARK-44811][BUILD] Upgrade Guava to 33.2.1-jre #42493

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

[SPARK-44811][BUILD] Upgrade Guava to 33.2.1-jre #42493

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e14900d
[SPARK-44811][BUILD] Bump Guava 33.2.1-jre
pan3793 May 10, 2024
c75e76a
nit
pan3793 Sep 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion assembly/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@
<!--
Because we don't shade dependencies anymore, we need to restore Guava to compile scope so
that the libraries Spark depend on have it available. We'll package the version that Spark
uses (14.0.1) which is not the same as Hadoop dependencies, but works.
uses which is not the same as Hadoop dependencies, but works.
-->
<dependency>
<groupId>com.google.guava</groupId>
Expand Down
1 change: 1 addition & 0 deletions core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -558,6 +558,7 @@
<include>org.eclipse.jetty:jetty-util</include>
<include>org.eclipse.jetty:jetty-server</include>
<include>com.google.guava:guava</include>
<include>com.google.guava:failureaccess</include>
<include>com.google.protobuf:*</include>
<!-- While shading new dependencies, please also update LICENSE/NOTICE files in the
core/src/main/resources/META-INF if necessary -->
Expand Down
7 changes: 6 additions & 1 deletion dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ breeze-macros_2.13/2.1.0//breeze-macros_2.13-2.1.0.jar
breeze_2.13/2.1.0//breeze_2.13-2.1.0.jar
bundle/2.24.6//bundle-2.24.6.jar
cats-kernel_2.13/2.8.0//cats-kernel_2.13-2.8.0.jar
checker-qual/3.42.0//checker-qual-3.42.0.jar
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we need to update LICENSE-binary and NOTICE-binary for new jars, @pan3793 .

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please proceed as a follow-up, @pan3793 .

chill-java/0.10.0//chill-java-0.10.0.jar
chill_2.13/0.10.0//chill_2.13-0.10.0.jar
commons-cli/1.9.0//commons-cli-1.9.0.jar
Expand Down Expand Up @@ -62,12 +63,14 @@ derby/10.16.1.1//derby-10.16.1.1.jar
derbyshared/10.16.1.1//derbyshared-10.16.1.1.jar
derbytools/10.16.1.1//derbytools-10.16.1.1.jar
dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
error_prone_annotations/2.26.1//error_prone_annotations-2.26.1.jar
esdk-obs-java/3.20.4.2//esdk-obs-java-3.20.4.2.jar
failureaccess/1.0.2//failureaccess-1.0.2.jar
flatbuffers-java/24.3.25//flatbuffers-java-24.3.25.jar
gcs-connector/hadoop3-2.2.21/shaded/gcs-connector-hadoop3-2.2.21-shaded.jar
gmetric4j/1.0.10//gmetric4j-1.0.10.jar
gson/2.11.0//gson-2.11.0.jar
guava/14.0.1//guava-14.0.1.jar
guava/33.2.1-jre//guava-33.2.1-jre.jar
hadoop-aliyun/3.4.0//hadoop-aliyun-3.4.0.jar
hadoop-annotations/3.4.0//hadoop-annotations-3.4.0.jar
hadoop-aws/3.4.0//hadoop-aws-3.4.0.jar
Expand Down Expand Up @@ -101,6 +104,7 @@ icu4j/75.1//icu4j-75.1.jar
ini4j/0.5.4//ini4j-0.5.4.jar
istack-commons-runtime/3.0.8//istack-commons-runtime-3.0.8.jar
ivy/2.5.2//ivy-2.5.2.jar
j2objc-annotations/3.0.0//j2objc-annotations-3.0.0.jar
jackson-annotations/2.17.2//jackson-annotations-2.17.2.jar
jackson-core-asl/1.9.13//jackson-core-asl-1.9.13.jar
jackson-core/2.17.2//jackson-core-2.17.2.jar
Expand Down Expand Up @@ -184,6 +188,7 @@ lapack/3.0.3//lapack-3.0.3.jar
leveldbjni-all/1.8//leveldbjni-all-1.8.jar
libfb303/0.9.3//libfb303-0.9.3.jar
libthrift/0.16.0//libthrift-0.16.0.jar
listenablefuture/9999.0-empty-to-avoid-conflict-with-guava//listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
log4j-1.2-api/2.22.1//log4j-1.2-api-2.22.1.jar
log4j-api/2.22.1//log4j-api-2.22.1.jar
log4j-core/2.22.1//log4j-core-2.22.1.jar
Expand Down
3 changes: 2 additions & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@
<!-- org.apache.commons/commons-pool2/-->
<commons-pool2.version>2.12.0</commons-pool2.version>
<datanucleus-core.version>4.1.17</datanucleus-core.version>
<guava.version>14.0.1</guava.version>
<guava.version>33.2.1-jre</guava.version>
<gson.version>2.11.0</gson.version>
<janino.version>3.1.9</janino.version>
<jersey.version>3.0.12</jersey.version>
Expand Down Expand Up @@ -3420,6 +3420,7 @@
<includes>
<include>org.spark-project.spark:unused</include>
<include>com.google.guava:guava</include>
<include>com.google.guava:failureaccess</include>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I refer to the shading rules in the connect module to handle Guava dependencies, cc @LuciferYang, would be great if you have an opportunity to take a look.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fine to me

<include>org.jpmml:*</include>
</includes>
</artifactSet>
Expand Down
2 changes: 1 addition & 1 deletion project/SparkBuild.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1051,7 +1051,7 @@ object KubernetesIntegrationTests {
* Overrides to work around sbt's dependency resolution being different from Maven's.
*/
object DependencyOverrides {
lazy val guavaVersion = sys.props.get("guava.version").getOrElse("14.0.1")
lazy val guavaVersion = sys.props.get("guava.version").getOrElse("33.1.0-jre")
lazy val settings = Seq(
dependencyOverrides += "com.google.guava" % "guava" % guavaVersion,
dependencyOverrides += "xerces" % "xercesImpl" % "2.12.2",
Expand Down
2 changes: 1 addition & 1 deletion ...t/src/test/scala/org/apache/spark/sql/catalyst/expressions/IntervalExpressionsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -351,7 +351,7 @@ class IntervalExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {

Seq(
(Period.ofMonths(2), Int.MaxValue) -> "overflow",
(Period.ofMonths(Int.MinValue), 10d) -> "not in range",
(Period.ofMonths(Int.MinValue), 10d) -> "out of range",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, this error message is originated from Guava?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @cloud-fan do you think such error message change is acceptable?

(Period.ofMonths(-100), Float.NaN) -> "input is infinite or NaN",
(Period.ofMonths(200), Double.PositiveInfinity) -> "input is infinite or NaN",
(Period.ofMonths(-200), Float.NegativeInfinity) -> "input is infinite or NaN"
Expand Down
4 changes: 2 additions & 2 deletions sql/core/src/test/resources/sql-tests/results/ansi/interval.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2890,7 +2890,7 @@ SELECT (INTERVAL '-178956970-8' YEAR TO MONTH) / -1.0D
struct<>
-- !query output
java.lang.ArithmeticException
not in range
rounded value is out of range for input 2.147483648E9 and rounding mode HALF_UP


-- !query
Expand Down Expand Up @@ -2970,7 +2970,7 @@ SELECT (INTERVAL '-106751991 04:00:54.775808' DAY TO SECOND) / -1.0D
struct<>
-- !query output
java.lang.ArithmeticException
not in range
rounded value is out of range for input 9.223372036854776E18 and rounding mode HALF_UP


-- !query
Expand Down
4 changes: 2 additions & 2 deletions sql/core/src/test/resources/sql-tests/results/interval.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2713,7 +2713,7 @@ SELECT (INTERVAL '-178956970-8' YEAR TO MONTH) / -1.0D
struct<>
-- !query output
java.lang.ArithmeticException
not in range
rounded value is out of range for input 2.147483648E9 and rounding mode HALF_UP


-- !query
Expand Down Expand Up @@ -2793,7 +2793,7 @@ SELECT (INTERVAL '-106751991 04:00:54.775808' DAY TO SECOND) / -1.0D
struct<>
-- !query output
java.lang.ArithmeticException
not in range
rounded value is out of range for input 9.223372036854776E18 and rounding mode HALF_UP


-- !query
Expand Down