[KYUUBI #4274] [FOLLOWUP] Increase maximum degree of concurrency for mvnd in CI jobs

[bowenliang123]

Why are the changes needed?

• concurrent execution and smart builder is shipped with mvnd, but the CI log shows they are not activated
• increase maximum degree of concurrency and utilize SmartBuilder feature in mvnd. -Dmvnd.minThreads is set as fallback and it will be ignored if --threads or -Dmvnd.threads is used.

Before:

(https://github.com/apache/kyuubi/actions/runs/4276652363/jobs/7444896450#step:6:64)

[INFO] Build maximum degree of concurrency is 1

After:

(https://github.com/apache/kyuubi/actions/runs/4279322563/jobs/7449912132#step:8:64)

[INFO] Using the SmartBuilder implementation with a thread count of 4
[INFO] Build maximum degree of concurrency is 4

(https://github.com/apache/kyuubi/actions/runs/4279322563/jobs/7449912132#step:8:553)

[INFO] Segment walltime 13 s, segment projects service time 36 s, effective/maximum degree of concurrency 2.63/4

How was this patch tested?

• Add some test cases that check the changes thoroughly including negative and positive cases if possible

• Add screenshots for manual tests if appropriate

• Run test locally before make a pull request

[codecov-commenter]

Codecov Report

Merging #4420 (64f0cc5) into master (15a83e1) will increase coverage by 0.04%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #4420      +/-   ##
============================================
+ Coverage     53.14%   53.19%   +0.04%     
  Complexity       13       13              
============================================
  Files           564      569       +5     
  Lines         30978    31111     +133     
  Branches       4170     4203      +33     
============================================
+ Hits          16463    16548      +85     
- Misses        12972    12992      +20     
- Partials       1543     1571      +28     

Impacted Files	Coverage Δ
...ache/kyuubi/plugin/spark/authz/serde/package.scala	65.62% <0.00%> (-18.75%)	⬇️
.../authentication/EngineSecuritySecretProvider.scala	84.61% <0.00%> (-15.39%)	⬇️
...rg/apache/kyuubi/server/api/v1/AdminResource.scala	87.39% <0.00%> (-8.17%)	⬇️
...pache/kyuubi/server/metadata/MetadataManager.scala	80.95% <0.00%> (-3.33%)	⬇️
...in/spark/authz/ranger/SparkRangerAdminPlugin.scala	64.47% <0.00%> (-2.64%)	⬇️
...rg/apache/kyuubi/ctl/cmd/log/LogBatchCommand.scala	59.09% <0.00%> (-1.52%)	⬇️
...n/scala/org/apache/kyuubi/engine/ProcBuilder.scala	78.39% <0.00%> (-0.62%)	⬇️
...in/lineage/helper/SparkSQLLineageParseHelper.scala	62.64% <0.00%> (-0.40%)	⬇️
...in/scala/org/apache/kyuubi/config/KyuubiConf.scala	97.32% <0.00%> (-0.16%)	⬇️
...g/apache/kyuubi/operation/BatchJobSubmission.scala	75.27% <0.00%> (ø)
... and 14 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[bowenliang123]

Although mvnd does have output buffering for handling concurrent build in projects on multi-threads, the terminal of Github actions is still recognized as dumb type (see https://github.com/apache/maven-mvnd/blob/0.9.0/common/src/main/java/org/mvndaemon/mvnd/common/logging/TerminalOutput.java#L393) in mvnd and thenoBuffering option is forced to be enabled. Thus, it causes continuous mixed outputs from different projects and makes the logs less readable and readable.

Therefore, I would like to make mvnd in --quiet mode for less verbose in the jobs of dependency check and style checks.
(see: https://github.com/apache/kyuubi/actions/runs/4304980329/jobs/7506775519#step:5:1)

[bowenliang123]

Thanks, merged to master.