Skip to content

Commit

Permalink
[SPARK-49809][BUILD] Use sbt.IO in SparkBuild.scala to avoid nami…
Browse files Browse the repository at this point in the history
…ng conflicts with `java.io.IO` in Java 23

### What changes were proposed in this pull request?
This pr change to use `sbt.IO` in `SparkBuild.scala` to avoid naming conflicts with `java.io.IO` in Java 23, and after this PR, Spark can be built using sbt with Java 23(current pr does not focus on the results of `sbt/test` with Java 23)

### Why are the changes needed?
Make Spark be compiled using sbt with Java 23.

Because Java 23 has added `java.io.IO`, and `SparkBuild.scala` imports both `java.io._` and `sbt._`, this results in the following error when executing

```
build/sbt -Phadoop-3 -Phive-thriftserver -Pspark-ganglia-lgpl -Pdocker-integration-tests -Pyarn -Pvolcano -Pkubernetes -Pkinesis-asl -Phive -Phadoop-cloud Test/package streaming-kinesis-asl-assembly/assembly connect/assembly
```

with Java 23

```
build/sbt -Phadoop-3 -Phive-thriftserver -Pspark-ganglia-lgpl -Pdocker-integration-tests -Pyarn -Pvolcano -Pkubernetes -Pkinesis-asl -Phive -Phadoop-cloud Test/package streaming-kinesis-asl-assembly/assembly connect/assembly
Using /Users/yangjie01/Tools/zulu23 as default JAVA_HOME.
Note, this will be overridden by -java-home if it is set.
[info] welcome to sbt 1.9.3 (Azul Systems, Inc. Java 23)
[info] loading settings for project global-plugins from idea.sbt ...
[info] loading global plugins from /Users/yangjie01/.sbt/1.0/plugins
[info] loading settings for project spark-sbt-build from plugins.sbt ...
[info] loading project definition from /Users/yangjie01/SourceCode/git/spark-sbt/project
[info] compiling 3 Scala sources to /Users/yangjie01/SourceCode/git/spark-sbt/project/target/scala-2.12/sbt-1.0/classes ...
[error] /Users/yangjie01/SourceCode/git/spark-sbt/project/SparkBuild.scala:1209:7: reference to IO is ambiguous;
[error] it is imported twice in the same scope by
[error] import sbt._
[error] and import java.io._
[error]       IO.write(file, s"$hadoopProvidedProp = $isHadoopProvided")
[error]       ^
[error] one error found
[error] (Compile / compileIncremental) Compilation failed
```

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
- Pass Github Actions
- Manual check:

```
build/sbt -Phadoop-3 -Phive-thriftserver -Pspark-ganglia-lgpl -Pdocker-integration-tests -Pyarn -Pvolcano -Pkubernetes -Pkinesis-asl -Phive -Phadoop-cloud Test/package streaming-kinesis-asl-assembly/assembly connect/assembly
```

with Java 23, after this pr, the aforementioned command can be executed successfully.

### Was this patch authored or co-authored using generative AI tooling?
No

Closes apache#48280 from LuciferYang/build-with-java23.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
LuciferYang authored and dongjoon-hyun committed Sep 27, 2024
1 parent dd692e9 commit 6dc628c
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion project/SparkBuild.scala
Original file line number Diff line number Diff line change
Expand Up @@ -1206,7 +1206,7 @@ object YARN {
genConfigProperties := {
val file = (Compile / classDirectory).value / s"org/apache/spark/deploy/yarn/$propFileName"
val isHadoopProvided = SbtPomKeys.effectivePom.value.getProperties.get(hadoopProvidedProp)
IO.write(file, s"$hadoopProvidedProp = $isHadoopProvided")
sbt.IO.write(file, s"$hadoopProvidedProp = $isHadoopProvided")
},
Compile / copyResources := (Def.taskDyn {
val c = (Compile / copyResources).value
Expand Down

0 comments on commit 6dc628c

Please sign in to comment.