[SPARK-43595][BUILD] Update some maven plugins to newest version by panbingkun · Pull Request #41228 · apache/spark

panbingkun · 2023-05-19T03:33:12Z

What changes were proposed in this pull request?

The pr aims to update some maven plugins to newest version. include:

exec-maven-plugin from 1.6.0 to 3.1.0
scala-maven-plugin from 4.8.0 to 4.8.1
maven-antrun-plugin from 1.8 to 3.1.0
maven-enforcer-plugin from 3.2.1 to 3.3.0
build-helper-maven-plugin from 3.3.0 to 3.4.0
maven-surefire-plugin from 3.0.0 to 3.1.0
maven-assembly-plugin from 3.1.0 to 3.6.0
maven-install-plugin from 3.1.0 to 3.1.1
maven-deploy-plugin from 3.1.0 to 3.1.1
maven-checkstyle-plugin from 3.2.1 to 3.2.2

Why are the changes needed?

Routine upgrade.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GA.

srowen · 2023-05-20T13:50:23Z

Merged to master

bjornjorgensen · 2023-05-22T12:25:13Z

Using ./build/mvn with openjdk 17 I get this error with scala-maven-plugin.version 4.8.1

#14 238.4 [INFO] --- scala-maven-plugin:4.8.1:compile (scala-compile-first) @ spark-core_2.12 ---
#14 238.4 [INFO] Compiler bridge file: /root/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.8.0-bin_2.12.17__61.0-1.8.0_20221110T195421.jar
#14 238.4 [INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.17,1.7.10,null)
#14 238.4 [INFO] compiling 602 Scala sources and 101 Java sources to /tmp/spark/core/target/scala-2.12/classes ...
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:71: not found: value sun
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:26: not found: object sun
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27: not found: object sun
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:206: not found: type DirectBuffer
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:210: not found: type Unsafe
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:212: not found: type Unsafe
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:213: not found: type DirectBuffer
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:216: not found: type DirectBuffer
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:236: not found: type DirectBuffer
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala:452: not found: value sun
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26: not found: object sun
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:99: not found: type SignalHandler
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:99: not found: type Signal
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:83: not found: type Signal
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:108: not found: type SignalHandler
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:108: not found: value Signal
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:114: not found: type Signal
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:116: not found: value Signal
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:128: not found: value Signal
#14 238.4 [ERROR] 19 errors found
#14 238.4 [INFO] ------------------------------------------------------------------------
#14 238.4 [INFO] Reactor Summary for Spark Project Parent POM 3.5.0-SNAPSHOT:
#14 238.4 [INFO] 
#14 238.4 [INFO] Spark Project Parent POM ........................... SUCCESS [ 34.963 s]
#14 238.4 [INFO] Spark Project Tags ................................. SUCCESS [ 23.148 s]
#14 238.4 [INFO] Spark Project Sketch ............................... SUCCESS [ 12.645 s]
#14 238.4 [INFO] Spark Project Local DB ............................. SUCCESS [ 17.202 s]
#14 238.4 [INFO] Spark Project Networking ........................... SUCCESS [ 20.815 s]
#14 238.4 [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 17.392 s]
#14 238.4 [INFO] Spark Project Unsafe ............................... SUCCESS [ 16.924 s]
#14 238.4 [INFO] Spark Project Common Utils ......................... SUCCESS [ 11.795 s]
#14 238.4 [INFO] Spark Project Launcher ............................. SUCCESS [ 13.309 s]
#14 238.4 [INFO] Spark Project Core ................................. FAILURE [ 40.546 s]
#14 238.4 [INFO] Spark Project ML Local Library ..................... SKIPPED
#14 238.4 [INFO] Spark Project GraphX ............................... SKIPPED
#14 238.4 [INFO] Spark Project Streaming ............................ SKIPPED
#14 238.4 [INFO] Spark Project Catalyst ............................. SKIPPED
#14 238.4 [INFO] Spark Project SQL .................................. SKIPPED
#14 238.4 [INFO] Spark Project ML Library ........................... SKIPPED
#14 238.4 [INFO] Spark Project Tools ................................ SKIPPED
#14 238.4 [INFO] Spark Project Hive ................................. SKIPPED
#14 238.4 [INFO] Spark Project REPL ................................. SKIPPED
#14 238.4 [INFO] Spark Project Assembly ............................. SKIPPED
#14 238.4 [INFO] Kafka 0.10+ Token Provider for Streaming ........... SKIPPED
#14 238.4 [INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
#14 238.4 [INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED
#14 238.4 [INFO] Spark Project Examples ............................. SKIPPED
#14 238.4 [INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
#14 238.4 [INFO] Spark Avro ......................................... SKIPPED
#14 238.4 [INFO] Spark Project Connect Common ....................... SKIPPED
#14 238.4 [INFO] Spark Protobuf ..................................... SKIPPED
#14 238.4 [INFO] Spark Project Connect Server ....................... SKIPPED
#14 238.4 [INFO] Spark Project Connect Client ....................... SKIPPED
#14 238.4 [INFO] ------------------------------------------------------------------------
#14 238.4 [INFO] BUILD FAILURE
#14 238.4 [INFO] ------------------------------------------------------------------------
#14 238.4 [INFO] Total time:  03:29 min
#14 238.4 [INFO] Finished at: 2023-05-22T10:27:14Z
#14 238.4 [INFO] ------------------------------------------------------------------------
#14 238.4 [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.8.1:compile (scala-compile-first) on project spark-core_2.12: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:4.8.1:compile failed: org.apache.commons.exec.ExecuteException: Process exited with an error: 255 (Exit value: 255) -> [Help 1]
#14 238.4 [ERROR] 
#14 238.4 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
#14 238.4 [ERROR] Re-run Maven using the -X switch to enable full debug logging.
#14 238.4 [ERROR] 
#14 238.4 [ERROR] For more information about the errors and possible solutions, please read the following articles:
#14 238.4 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

When I change scala-maven-plugin.version to 4.8.0 it works

bjornjorgensen · 2023-05-22T13:43:24Z

@panbingkun dident you upgrade scala-maven-plugin from 4.8.0 to 4.8.1 in #40442
and @LuciferYang did have to revere it in 1530e8d
CC @srowen

srowen · 2023-05-22T13:54:33Z

Let's revert that change and leave a note that this plugin cannot be updated further. It seems to be missing the JDK 'internal' classes, maybe for good reasons but I don't see why

### What changes were proposed in this pull request? Downgrade scala-maven-plugin.version from 4.8.1 to 4.8.0 and add a note ### Why are the changes needed? see #41228 #40442 1530e8d As mentioned in #40442, there are some regression with the 4.8.1： - Run `./build/mvn -DskipTests clean package` with Java 17 will failed ``` [INFO] --- maven-compiler-plugin:3.11.0:compile (default-compile) spark-core_2.12 --- [INFO] Not compiling main sources [INFO] [INFO] --- scala-maven-plugin:4.8.1:compile (scala-compile-first) spark-core_2.12 --- [INFO] Compiler bridge file: /home/bjorn/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.8.0-bin_2.12.17__55.0-1.8.0_20221110T195421.jar [INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.17,1.7.10,null) [INFO] compiling 597 Scala sources and 103 Java sources to /home/bjorn/github/spark/core/target/scala-2.12/classes ... [ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:71: not found: value sun [ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:26: not found: object sun [ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27: not found: object sun [ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:206: not found: type DirectBuffer [ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:210: not found: type Unsafe [ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:212: not found: type Unsafe [ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:213: not found: type DirectBuffer [ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:216: not found: type DirectBuffer [ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:236: not found: type DirectBuffer [ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala:452: not found: value sun [ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26: not found: object sun [ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:99: not found: type SignalHandler [ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:99: not found: type Signal [ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:83: not found: type Signal [ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:108: not found: type SignalHandler [ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:108: not found: value Signal [ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:114: not found: type Signal [ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:116: not found: value Signal [ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:128: not found: value Signal [ERROR] 19 errors found [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for Spark Project Parent POM 3.5.0-SNAPSHOT: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [ 3.848 s] [INFO] Spark Project Tags ................................. SUCCESS [ 12.106 s] [INFO] Spark Project Sketch ............................... SUCCESS [ 10.685 s] [INFO] Spark Project Local DB ............................. SUCCESS [ 8.743 s] [INFO] Spark Project Networking ........................... SUCCESS [ 9.362 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 7.828 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 9.071 s] [INFO] Spark Project Launcher ............................. SUCCESS [ 4.776 s] [INFO] Spark Project Core ................................. FAILURE [ 17.228 s] ``` - Run `build/mvn clean install -DskipTests -Pscala-2.13` with Java8 + Scala 2.13 There are compilation errors as `ERROR] -release is only supported on Java 9 and higher` although it does not cause compilation failures. More, I saw davidB/scala-maven-plugin#686, So it seems that 4.8.1 and Java 8 are not compatible well. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA Closes #41261 from bjornjorgensen/revers-scala-maven-plugin. Authored-by: bjornjorgensen <bjornjorgensen@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

test

1ca8487

github-actions bot added the BUILD label May 19, 2023

Merge branch 'master' into maven_plugin_upgrade

41ead64

panbingkun changed the title ~~[WIP] Upgrade maven plugins~~ [SPARK-43595][BUILD] Update some maven plugins to newest version May 19, 2023

srowen approved these changes May 20, 2023

View reviewed changes

srowen closed this in afc5087 May 20, 2023

bjornjorgensen mentioned this pull request May 22, 2023

[SPARK-43716][BUILD] Revert scala-maven-plugin to 4.8.0 #41261

Closed

dongjoon-hyun mentioned this pull request Jun 24, 2025

[SPARK-52568][BUILD] Fix exec-maven-plugin version used by dev/test-dependencies.sh #51273

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-43595][BUILD] Update some maven plugins to newest version#41228

[SPARK-43595][BUILD] Update some maven plugins to newest version#41228
panbingkun wants to merge 2 commits intoapache:masterfrom
panbingkun:maven_plugin_upgrade

panbingkun commented May 19, 2023 •

edited

Loading

Uh oh!

srowen commented May 20, 2023

Uh oh!

bjornjorgensen commented May 22, 2023 •

edited

Loading

Uh oh!

bjornjorgensen commented May 22, 2023

Uh oh!

srowen commented May 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

panbingkun commented May 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

srowen commented May 20, 2023

Uh oh!

bjornjorgensen commented May 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bjornjorgensen commented May 22, 2023

Uh oh!

srowen commented May 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

panbingkun commented May 19, 2023 •

edited

Loading

bjornjorgensen commented May 22, 2023 •

edited

Loading