From fed9a8da3d4187794161e0be325aa96be8487783 Mon Sep 17 00:00:00 2001 From: yangjie01 Date: Thu, 10 Oct 2024 01:49:24 +0800 Subject: [PATCH] [SPARK-49569][BUILD][FOLLOWUP] Adds `scala-library` maven dependency to the `spark-connect-shims` module to fix Maven build errors ### What changes were proposed in this pull request? This PR adds `scala-library` maven dependency to the `spark-connect-shims` module to fix Maven build errors. ### Why are the changes needed? Maven daily test pipeline build failed: - https://github.com/apache/spark/actions/runs/11255598249 - https://github.com/apache/spark/actions/runs/11256610976 ``` scaladoc error: fatal error: object scala in compiler mirror not found. Error: Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.9.1:doc-jar (attach-scaladocs) on project spark-connect-shims_2.13: MavenReportException: Error while creating archive: wrap: Process exited with an error: 1 (Exit value: 1) -> [Help 1] Error: Error: To see the full stack trace of the errors, re-run Maven with the -e switch. Error: Re-run Maven using the -X switch to enable full debug logging. Error: Error: For more information about the errors and possible solutions, please read the following articles: Error: [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException Error: Error: After correcting the problems, you can resume the build with the command Error: mvn -rf :spark-connect-shims_2.13 Error: Process completed with exit code 1. ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GitHub Actions - locally test: ``` build/mvn clean install -DskipTests -Phive ``` **Before** ``` [INFO] --- scala:4.9.1:doc-jar (attach-scaladocs) spark-connect-shims_2.13 --- scaladoc error: fatal error: object scala in compiler mirror not found. [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for Spark Project Parent POM 4.0.0-SNAPSHOT: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [ 2.833 s] [INFO] Spark Project Tags ................................. SUCCESS [ 5.292 s] [INFO] Spark Project Sketch ............................... SUCCESS [ 5.675 s] [INFO] Spark Project Common Utils ......................... SUCCESS [ 16.762 s] [INFO] Spark Project Local DB ............................. SUCCESS [ 7.735 s] [INFO] Spark Project Networking ........................... SUCCESS [ 11.389 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 9.159 s] [INFO] Spark Project Variant .............................. SUCCESS [ 3.618 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 9.692 s] [INFO] Spark Project Connect Shims ........................ FAILURE [ 2.478 s] [INFO] Spark Project Launcher ............................. SKIPPED [INFO] Spark Project Core ................................. SKIPPED [INFO] Spark Project ML Local Library ..................... SKIPPED [INFO] Spark Project GraphX ............................... SKIPPED [INFO] Spark Project Streaming ............................ SKIPPED [INFO] Spark Project SQL API .............................. SKIPPED [INFO] Spark Project Catalyst ............................. SKIPPED [INFO] Spark Project SQL .................................. SKIPPED [INFO] Spark Project ML Library ........................... SKIPPED [INFO] Spark Project Tools ................................ SKIPPED [INFO] Spark Project Hive ................................. SKIPPED [INFO] Spark Project Connect Common ....................... SKIPPED [INFO] Spark Avro ......................................... SKIPPED [INFO] Spark Protobuf ..................................... SKIPPED [INFO] Spark Project REPL ................................. SKIPPED [INFO] Spark Project Connect Server ....................... SKIPPED [INFO] Spark Project Connect Client ....................... SKIPPED [INFO] Spark Project Assembly ............................. SKIPPED [INFO] Kafka 0.10+ Token Provider for Streaming ........... SKIPPED [INFO] Spark Integration for Kafka 0.10 ................... SKIPPED [INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED [INFO] Spark Project Examples ............................. SKIPPED [INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:15 min [INFO] Finished at: 2024-10-09T23:43:58+08:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.9.1:doc-jar (attach-scaladocs) on project spark-connect-shims_2.13: MavenReportException: Error while creating archive: wrap: Process exited with an error: 1 (Exit value: 1) -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :spark-connect-shims_2.13 ``` **After** ``` [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for Spark Project Parent POM 4.0.0-SNAPSHOT: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [ 2.766 s] [INFO] Spark Project Tags ................................. SUCCESS [ 5.398 s] [INFO] Spark Project Sketch ............................... SUCCESS [ 6.361 s] [INFO] Spark Project Common Utils ......................... SUCCESS [ 16.919 s] [INFO] Spark Project Local DB ............................. SUCCESS [ 8.083 s] [INFO] Spark Project Networking ........................... SUCCESS [ 11.240 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 9.438 s] [INFO] Spark Project Variant .............................. SUCCESS [ 3.697 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 9.939 s] [INFO] Spark Project Connect Shims ........................ SUCCESS [ 2.938 s] [INFO] Spark Project Launcher ............................. SUCCESS [ 6.502 s] [INFO] Spark Project Core ................................. SUCCESS [01:33 min] [INFO] Spark Project ML Local Library ..................... SUCCESS [ 18.220 s] [INFO] Spark Project GraphX ............................... SUCCESS [ 20.923 s] [INFO] Spark Project Streaming ............................ SUCCESS [ 29.949 s] [INFO] Spark Project SQL API .............................. SUCCESS [ 25.842 s] [INFO] Spark Project Catalyst ............................. SUCCESS [02:02 min] [INFO] Spark Project SQL .................................. SUCCESS [02:18 min] [INFO] Spark Project ML Library ........................... SUCCESS [01:38 min] [INFO] Spark Project Tools ................................ SUCCESS [ 3.365 s] [INFO] Spark Project Hive ................................. SUCCESS [ 45.357 s] [INFO] Spark Project Connect Common ....................... SUCCESS [ 33.636 s] [INFO] Spark Avro ......................................... SUCCESS [ 22.040 s] [INFO] Spark Protobuf ..................................... SUCCESS [ 24.557 s] [INFO] Spark Project REPL ................................. SUCCESS [ 13.843 s] [INFO] Spark Project Connect Server ....................... SUCCESS [ 35.587 s] [INFO] Spark Project Connect Client ....................... SUCCESS [ 33.929 s] [INFO] Spark Project Assembly ............................. SUCCESS [ 5.121 s] [INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [ 12.623 s] [INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 16.908 s] [INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [ 23.664 s] [INFO] Spark Project Examples ............................. SUCCESS [ 30.777 s] [INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 6.997 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 15:40 min [INFO] Finished at: 2024-10-09T23:27:20+08:00 [INFO] ------------------------------------------------------------------------ ``` ### Was this patch authored or co-authored using generative AI tooling? No Closes #48399 from LuciferYang/SPARK-49569-FOLLOWUP. Authored-by: yangjie01 Signed-off-by: yangjie01 --- sql/connect/shims/pom.xml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/sql/connect/shims/pom.xml b/sql/connect/shims/pom.xml index 6bb12a927738c..d177b4a9971f5 100644 --- a/sql/connect/shims/pom.xml +++ b/sql/connect/shims/pom.xml @@ -34,6 +34,13 @@ connect-shims + + + org.scala-lang + scala-library + + + target/scala-${scala.binary.version}/classes target/scala-${scala.binary.version}/test-classes