Skip to content

[SPARK-43595][BUILD] Update some maven plugins to newest version #41228

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

[SPARK-43595][BUILD] Update some maven plugins to newest version #41228

wants to merge 2 commits into from

Conversation

panbingkun
Copy link
Contributor

@panbingkun panbingkun commented May 19, 2023 •
edited
Loading

What changes were proposed in this pull request?

The pr aims to update some maven plugins to newest version. include:

  • exec-maven-plugin from 1.6.0 to 3.1.0
  • scala-maven-plugin from 4.8.0 to 4.8.1
  • maven-antrun-plugin from 1.8 to 3.1.0
  • maven-enforcer-plugin from 3.2.1 to 3.3.0
  • build-helper-maven-plugin from 3.3.0 to 3.4.0
  • maven-surefire-plugin from 3.0.0 to 3.1.0
  • maven-assembly-plugin from 3.1.0 to 3.6.0
  • maven-install-plugin from 3.1.0 to 3.1.1
  • maven-deploy-plugin from 3.1.0 to 3.1.1
  • maven-checkstyle-plugin from 3.2.1 to 3.2.2

Why are the changes needed?

Routine upgrade.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GA.

@github-actions github-actions bot added the BUILD label May 19, 2023
@panbingkun panbingkun changed the title [WIP] Upgrade maven plugins [SPARK-43595][BUILD] Update some maven plugins to newest version May 19, 2023
@srowen
Copy link
Member

srowen commented May 20, 2023

Merged to master

@srowen srowen closed this in afc5087 May 20, 2023
@bjornjorgensen
Copy link
Contributor

bjornjorgensen commented May 22, 2023 •
edited
Loading

Using ./build/mvn with openjdk 17 I get this error with scala-maven-plugin.version 4.8.1

#14 238.4 [INFO] --- scala-maven-plugin:4.8.1:compile (scala-compile-first) @ spark-core_2.12 ---
#14 238.4 [INFO] Compiler bridge file: /root/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.8.0-bin_2.12.17__61.0-1.8.0_20221110T195421.jar
#14 238.4 [INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.17,1.7.10,null)
#14 238.4 [INFO] compiling 602 Scala sources and 101 Java sources to /tmp/spark/core/target/scala-2.12/classes ...
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:71: not found: value sun
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:26: not found: object sun
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27: not found: object sun
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:206: not found: type DirectBuffer
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:210: not found: type Unsafe
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:212: not found: type Unsafe
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:213: not found: type DirectBuffer
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:216: not found: type DirectBuffer
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:236: not found: type DirectBuffer
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala:452: not found: value sun
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26: not found: object sun
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:99: not found: type SignalHandler
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:99: not found: type Signal
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:83: not found: type Signal
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:108: not found: type SignalHandler
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:108: not found: value Signal
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:114: not found: type Signal
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:116: not found: value Signal
Error: .4 [ERROR] [Error] /tmp/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:128: not found: value Signal
#14 238.4 [ERROR] 19 errors found
#14 238.4 [INFO] ------------------------------------------------------------------------
#14 238.4 [INFO] Reactor Summary for Spark Project Parent POM 3.5.0-SNAPSHOT:
#14 238.4 [INFO] 
#14 238.4 [INFO] Spark Project Parent POM ........................... SUCCESS [ 34.963 s]
#14 238.4 [INFO] Spark Project Tags ................................. SUCCESS [ 23.148 s]
#14 238.4 [INFO] Spark Project Sketch ............................... SUCCESS [ 12.645 s]
#14 238.4 [INFO] Spark Project Local DB ............................. SUCCESS [ 17.202 s]
#14 238.4 [INFO] Spark Project Networking ........................... SUCCESS [ 20.815 s]
#14 238.4 [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 17.392 s]
#14 238.4 [INFO] Spark Project Unsafe ............................... SUCCESS [ 16.924 s]
#14 238.4 [INFO] Spark Project Common Utils ......................... SUCCESS [ 11.795 s]
#14 238.4 [INFO] Spark Project Launcher ............................. SUCCESS [ 13.309 s]
#14 238.4 [INFO] Spark Project Core ................................. FAILURE [ 40.546 s]
#14 238.4 [INFO] Spark Project ML Local Library ..................... SKIPPED
#14 238.4 [INFO] Spark Project GraphX ............................... SKIPPED
#14 238.4 [INFO] Spark Project Streaming ............................ SKIPPED
#14 238.4 [INFO] Spark Project Catalyst ............................. SKIPPED
#14 238.4 [INFO] Spark Project SQL .................................. SKIPPED
#14 238.4 [INFO] Spark Project ML Library ........................... SKIPPED
#14 238.4 [INFO] Spark Project Tools ................................ SKIPPED
#14 238.4 [INFO] Spark Project Hive ................................. SKIPPED
#14 238.4 [INFO] Spark Project REPL ................................. SKIPPED
#14 238.4 [INFO] Spark Project Assembly ............................. SKIPPED
#14 238.4 [INFO] Kafka 0.10+ Token Provider for Streaming ........... SKIPPED
#14 238.4 [INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
#14 238.4 [INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED
#14 238.4 [INFO] Spark Project Examples ............................. SKIPPED
#14 238.4 [INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
#14 238.4 [INFO] Spark Avro ......................................... SKIPPED
#14 238.4 [INFO] Spark Project Connect Common ....................... SKIPPED
#14 238.4 [INFO] Spark Protobuf ..................................... SKIPPED
#14 238.4 [INFO] Spark Project Connect Server ....................... SKIPPED
#14 238.4 [INFO] Spark Project Connect Client ....................... SKIPPED
#14 238.4 [INFO] ------------------------------------------------------------------------
#14 238.4 [INFO] BUILD FAILURE
#14 238.4 [INFO] ------------------------------------------------------------------------
#14 238.4 [INFO] Total time:  03:29 min
#14 238.4 [INFO] Finished at: 2023-05-22T10:27:14Z
#14 238.4 [INFO] ------------------------------------------------------------------------
#14 238.4 [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.8.1:compile (scala-compile-first) on project spark-core_2.12: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:4.8.1:compile failed: org.apache.commons.exec.ExecuteException: Process exited with an error: 255 (Exit value: 255) -> [Help 1]
#14 238.4 [ERROR] 
#14 238.4 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
#14 238.4 [ERROR] Re-run Maven using the -X switch to enable full debug logging.
#14 238.4 [ERROR] 
#14 238.4 [ERROR] For more information about the errors and possible solutions, please read the following articles:
#14 238.4 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

When I change scala-maven-plugin.version to 4.8.0 it works

@bjornjorgensen
Copy link
Contributor

@panbingkun dident you upgrade scala-maven-plugin from 4.8.0 to 4.8.1 in #40442
and @LuciferYang did have to revere it in 1530e8d
CC @srowen

@srowen
Copy link
Member

srowen commented May 22, 2023

Let's revert that change and leave a note that this plugin cannot be updated further. It seems to be missing the JDK 'internal' classes, maybe for good reasons but I don't see why

@bjornjorgensen bjornjorgensen mentioned this pull request May 22, 2023
Closed
dongjoon-hyun pushed a commit that referenced this pull request May 22, 2023
@bjornjorgensen @dongjoon-hyun
[SPARK-43716][BUILD] Revert scala-maven-plugin to 4.8.0
23800cb 
### What changes were proposed in this pull request?
Downgrade scala-maven-plugin.version from 4.8.1 to 4.8.0
and add a note

### Why are the changes needed?
see
#41228
#40442
1530e8d

As mentioned in #40442,  there are some regression with the 4.8.1：

- Run `./build/mvn -DskipTests clean package` with Java 17 will failed
```
[INFO] --- maven-compiler-plugin:3.11.0:compile (default-compile)  spark-core_2.12 ---
[INFO] Not compiling main sources
[INFO]
[INFO] --- scala-maven-plugin:4.8.1:compile (scala-compile-first)  spark-core_2.12 ---
[INFO] Compiler bridge file: /home/bjorn/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.8.0-bin_2.12.17__55.0-1.8.0_20221110T195421.jar
[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.17,1.7.10,null)
[INFO] compiling 597 Scala sources and 103 Java sources to /home/bjorn/github/spark/core/target/scala-2.12/classes ...
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:71: not found: value sun
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:26: not found: object sun
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27: not found: object sun
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:206: not found: type DirectBuffer
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:210: not found: type Unsafe
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:212: not found: type Unsafe
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:213: not found: type DirectBuffer
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:216: not found: type DirectBuffer
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:236: not found: type DirectBuffer
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala:452: not found: value sun
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26: not found: object sun
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:99: not found: type SignalHandler
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:99: not found: type Signal
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:83: not found: type Signal
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:108: not found: type SignalHandler
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:108: not found: value Signal
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:114: not found: type Signal
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:116: not found: value Signal
[ERROR] [Error] /home/bjorn/github/spark/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:128: not found: value Signal
[ERROR] 19 errors found
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 3.5.0-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 3.848 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 12.106 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 10.685 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 8.743 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 9.362 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 7.828 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 9.071 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 4.776 s]
[INFO] Spark Project Core ................................. FAILURE [ 17.228 s]
```
- Run `build/mvn clean install  -DskipTests -Pscala-2.13` with Java8 + Scala 2.13

There are compilation errors as `ERROR] -release is only supported on Java 9 and higher`  although it does not cause compilation failures. More, I saw davidB/scala-maven-plugin#686, So  it seems that 4.8.1 and Java 8 are not compatible well.
### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Pass GA

Closes #41261 from bjornjorgensen/revers-scala-maven-plugin.

Authored-by: bjornjorgensen <bjornjorgensen@gmail.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@srowen srowen srowen approved these changes

Assignees
No one assigned
Labels
BUILD
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@panbingkun @srowen @bjornjorgensen