[INFRA] Introduce `mvnd` to speed up CI jobs of Dependency, Licence and Style Check #4274

bowenliang123 · 2023-02-08T10:27:44Z

Why are the changes needed?

Motivation:

speed up maven building and testing with mvnd (https://github.com/apache/maven-mvnd). mvnd itself embeds exactly the same distribution of maven with more improvements and features on the client side and compilation server in daemon mode.
also inspired by custom github action for mvnd installing in Apache Camel (https://github.com/apache/camel/blob/main/.github/actions/install-mvnd/action.yml)

Changes in this PR:

introducing mvnd by adding build/mvnd script
use mvnd version 0.9.0 which embeds maven 3.8.7. The maven version embedded in mvnd is also check and guaranteed the same as maven version required in pom by build/mvnd script.
use mvnd in CI jobs of Depedency Check and Style Check

Comparision

CI jobs
(both with maven dependencies cached in local repo)

Job	with `build/mvn`	with `build/mvnd`
Dependency Check	9min1sec;see log	6min 46 sec ; see log
Style Check	10min32sec ; see log	7min50s ; see log
Licence Check	32s ; see log	21s ; see log

building entire maven project on local machine (skipping test running and style checking with fast profile)

build/mvnd clean install -Pfast

build/mvn clean install -Pfast

How was this patch tested?

Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before make a pull request

codecov-commenter · 2023-02-09T00:57:21Z

Codecov Report

Merging #4274 (97becc0) into master (3f5121a) will decrease coverage by 0.06%.
The diff coverage is n/a.

❗ Current head 97becc0 differs from pull request most recent head bc8da13. Consider uploading reports for the commit bc8da13 to get more accurate results

@@             Coverage Diff              @@
##             master    #4274      +/-   ##
============================================
- Coverage     53.42%   53.36%   -0.06%     
  Complexity       13       13              
============================================
  Files           560      560              
  Lines         30571    30571              
  Branches       4140     4140              
============================================
- Hits          16333    16315      -18     
- Misses        12705    12721      +16     
- Partials       1533     1535       +2

Impacted Files	Coverage Δ
.../kyuubi/server/mysql/constant/MySQLErrorCode.scala	`13.84% <0.00%> (-6.16%)`	⬇️
...ache/kyuubi/server/mysql/MySQLCommandHandler.scala	`77.77% <0.00%> (-4.05%)`	⬇️
.../apache/kyuubi/server/api/v1/BatchesResource.scala	`70.09% <0.00%> (-2.81%)`	⬇️
...in/spark/authz/ranger/SparkRangerAdminPlugin.scala	`64.47% <0.00%> (-2.64%)`	⬇️
...ache/kyuubi/server/mysql/MySQLGenericPackets.scala	`76.59% <0.00%> (-2.13%)`	⬇️
...rg/apache/kyuubi/ctl/cmd/log/LogBatchCommand.scala	`59.09% <0.00%> (-1.52%)`	⬇️
...n/scala/org/apache/kyuubi/engine/ProcBuilder.scala	`78.39% <0.00%> (-0.62%)`	⬇️
...g/apache/kyuubi/operation/BatchJobSubmission.scala	`75.27% <0.00%> (ø)`
...apache/kyuubi/engine/JpsApplicationOperation.scala	`80.64% <0.00%> (+3.22%)`	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

build/mvnd

cfmcgrady · 2023-02-09T03:27:23Z

build/mvnd

+# Preserve the calling directory
+_CALLING_DIR="$(pwd)"
+# Options used during compilation
+_COMPILE_JVM_OPTS="-Xms2g -Xmx2g -XX:ReservedCodeCacheSize=1g -Xss128m"


Github Action limit memory: 6947m, we may increase the max heap size?

build/mvnd is designed for the usage of both on local machine and also the GitHub action, so I would prefer to keep -Xmx here the same as in build/mvn. And we could increase the memory limit in CI jobs by passing opts.

https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L950
Marking the memory limit in Spark as above.

cfmcgrady · 2023-02-09T03:32:03Z

.github/workflows/dep.yml

+      # when pom changes
      - '**/pom.xml'
+      # when dependency workflow changes
+      - .github/workflows/dep.yaml


unnecessary changes?

To verify using mvnd in dependency workflows. Changes in dep.yaml should trigger the jobs defined from the file itself.

pan3793 · 2023-02-09T04:24:50Z

I suppose this target to 1.8?

bowenliang123 · 2023-02-09T05:11:45Z

I suppose this target to 1.8?

I think mvnd works fine with Java 8 or up. And I have tried to apply mvnd to Integration Test for Trino and JDBC with Java 8 and 11, it works fine. As for compiler target option, it uses the same options in pom.
There's also an mvnd specific option -Djava.home= to set Java home for starting the daemon.

$ mvnd -h
...
mvnd specific options:
...
 -Djava.home=<path>                     Java home for starting the daemon
                                        Env. variable: JAVA_HOME
...

pan3793 · 2023-02-09T05:30:21Z

Sorry for confusing, I mean target to Kyuubi 1.8.0

bowenliang123 · 2023-02-09T05:39:14Z

Sorry for confusing, I mean target to Kyuubi 1.8.0

OK, let's wait for the 1.8.0-SNAPSHOT on master branch.
This PR brings no changes in the source code and it is not part of the distribution.

pan3793 · 2023-02-10T08:19:53Z

Thanks, merged to master for 1.8

…mvnd in CI jobs ### _Why are the changes needed?_ - concurrent execution and smart builder is shipped with `mvnd`, but the CI log shows they are not activated - increase maximum degree of concurrency and utilize `SmartBuilder` feature in `mvnd`. `-Dmvnd.minThreads` is set as fallback and it will be ignored if `--threads` or `-Dmvnd.threads` is used. #### Before: (https://github.com/apache/kyuubi/actions/runs/4276652363/jobs/7444896450#step:6:64) ``` [INFO] Build maximum degree of concurrency is 1 ``` #### After: (https://github.com/apache/kyuubi/actions/runs/4279322563/jobs/7449912132#step:8:64) ``` [INFO] Using the SmartBuilder implementation with a thread count of 4 [INFO] Build maximum degree of concurrency is 4 ``` (https://github.com/apache/kyuubi/actions/runs/4279322563/jobs/7449912132#step:8:553) ``` [INFO] Segment walltime 13 s, segment projects service time 36 s, effective/maximum degree of concurrency 2.63/4 ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4420 from bowenliang123/mvnd-smart. Closes #4274 64f0cc5 [liangbowen] update 391da26 [liangbowen] quite in dep and style jobs c776356 [liangbowen] use -Dmvnd.minThreads d10bfb7 [liangbowen] use -Dmvnd.minThreads f793d2b [liangbowen] warn bbb929f [liangbowen] quiet a5e6d49 [liangbowen] logging warn level 3cfbaad [liangbowen] enable quite option and cancel redirect stderr 09442ff [liangbowen] update ca5f855 [liangbowen] try to redirect stderr 0ebdd18 [liangbowen] set completion and print MAVEN_CLI_OPTS d3b2c96 [liangbowen] revert explicitly setting `mvnd.noBuffering` e2eca01 [liangbowen] revert explicitly setting `mvnd.noBuffering` dba25e3 [liangbowen] explicitly set `mvnd.noBuffering` to false prevent displaying events continuously b7583f9 [liangbowen] shell d60ebd0 [liangbowen] change mvnd cache key 2cc576e [liangbowen] dep bbd6414 [liangbowen] move multithread setting to mvnd's MAVEN_CLI_OPTS 347b58c [liangbowen] increase concurrency for spotless check 15c5519 [liangbowen] increase concurrency Authored-by: liangbowen <liangbowen@gf.com.cn> Signed-off-by: liangbowen <liangbowen@gf.com.cn>

…ncy for mvnd in CI jobs" This reverts commit 5fab9b7.

…f Dependency, Licence and Style Check" This reverts commit d862272.

aaime · 2025-11-30T07:56:52Z

Hi, I'm considering using mvnd in github actions too. I see mvnd usage was eventually reverted in this commit, and I'm wondering what was the reason? I would like to find out about drawbacks of mvnd adoption in github actions before I have to learn the hard way 🤣

github-actions bot added kind:infra license, community building, project builds, asf infra related, etc. kind:build labels Feb 8, 2023

bowenliang123 changed the title ~~[DRAFT-DO-NOT-MERGE] introduce and use mvnd in CI~~ [DRAFT-DO-NOT-MERGE] Introduce and Use mvnd in Dependency Check CI job Feb 8, 2023

bowenliang123 force-pushed the ci-mvnd branch from 373a699 to 494078f Compare February 8, 2023 14:41

bowenliang123 changed the title ~~[DRAFT-DO-NOT-MERGE] Introduce and Use mvnd in Dependency Check CI job~~ [DRAFT-DO-NOT-MERGE] Introduce and Use mvnd in Dependency and Style Checking CI job Feb 8, 2023

bowenliang123 changed the title ~~[DRAFT-DO-NOT-MERGE] Introduce and Use mvnd in Dependency and Style Checking CI job~~ [INFRA] Introduce and Use mvnd in Dependency and Style Checking CI job Feb 8, 2023

bowenliang123 marked this pull request as ready for review February 8, 2023 16:18

introduce mvnd

f6d0eb2

bowenliang123 force-pushed the ci-mvnd branch from 076ab8e to f6d0eb2 Compare February 9, 2023 00:49

update

c52861d

bowenliang123 changed the title ~~[INFRA] Introduce and Use mvnd in Dependency and Style Checking CI job~~ [INFRA] Introduce and use mvnd in Dependency and Style Check CI job Feb 9, 2023

bowenliang123 requested review from cfmcgrady and pan3793 February 9, 2023 01:16

yaooqinn reviewed Feb 9, 2023

View reviewed changes

build/mvnd Outdated Show resolved Hide resolved

typo in build/mvnd

7be181d

cfmcgrady reviewed Feb 9, 2023

View reviewed changes

cfmcgrady approved these changes Feb 9, 2023

View reviewed changes

bowenliang123 added 2 commits February 9, 2023 12:15

apply mvnd to license check

5b61f50

add build/maven-mvnd-*/** to rat-excludes

97becc0

bowenliang123 self-assigned this Feb 9, 2023

bowenliang123 and others added 3 commits February 9, 2023 14:17

Merge branch 'master' into ci-mvnd

bc8da13

style

a93bba7

update cache key

380c1c9

pan3793 added this to the v1.8.0 milestone Feb 9, 2023

bowenliang123 changed the title ~~[INFRA] Introduce and use mvnd in Dependency and Style Check CI job~~ [INFRA] Introduce mvnd to speed up CI jobs of Dependency, Licence and Style Check Feb 9, 2023

pan3793 approved these changes Feb 10, 2023

View reviewed changes

pan3793 closed this in d862272 Feb 10, 2023

bowenliang123 deleted the ci-mvnd branch February 10, 2023 16:25

pan3793 added a commit that referenced this pull request Apr 12, 2023

Revert "[KYUUBI #4274] [FOLLOWUP] Increase maximum degree of concurre…

1c69652

…ncy for mvnd in CI jobs" This reverts commit 5fab9b7.

pan3793 added a commit that referenced this pull request Apr 12, 2023

Revert "[KYUUBI #4274] [INFRA] Introduce mvnd to speed up CI jobs o…

6431225

…f Dependency, Licence and Style Check" This reverts commit d862272.

[INFRA] Introduce mvnd to speed up CI jobs of Dependency, Licence and Style Check #4274

[INFRA] Introduce mvnd to speed up CI jobs of Dependency, Licence and Style Check #4274

Uh oh!

Conversation

bowenliang123 commented Feb 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are the changes needed?

Motivation:

Changes in this PR:

Comparision

How was this patch tested?

Uh oh!

codecov-commenter commented Feb 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

cfmcgrady Feb 9, 2023

Choose a reason for hiding this comment

Uh oh!

bowenliang123 Feb 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bowenliang123 Feb 9, 2023

Choose a reason for hiding this comment

Uh oh!

cfmcgrady Feb 9, 2023

Choose a reason for hiding this comment

Uh oh!

bowenliang123 Feb 9, 2023

Choose a reason for hiding this comment

Uh oh!

pan3793 commented Feb 9, 2023

Uh oh!

bowenliang123 commented Feb 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pan3793 commented Feb 9, 2023

Uh oh!

bowenliang123 commented Feb 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pan3793 commented Feb 10, 2023

Uh oh!

aaime commented Nov 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

[INFRA] Introduce `mvnd` to speed up CI jobs of Dependency, Licence and Style Check #4274

[INFRA] Introduce `mvnd` to speed up CI jobs of Dependency, Licence and Style Check #4274

bowenliang123 commented Feb 8, 2023 •

edited

Loading

codecov-commenter commented Feb 9, 2023 •

edited

Loading

bowenliang123 Feb 9, 2023 •

edited

Loading

bowenliang123 commented Feb 9, 2023 •

edited

Loading

bowenliang123 commented Feb 9, 2023 •

edited

Loading

aaime commented Nov 30, 2025 •

edited

Loading