-
Notifications
You must be signed in to change notification settings - Fork 971
[INFRA] Introduce mvnd to speed up CI jobs of Dependency, Licence and Style Check
#4274
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
+185
−9
Closed
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
f6d0eb2
introduce mvnd
bowenliang123 c52861d
update
bowenliang123 7be181d
typo in build/mvnd
bowenliang123 5b61f50
apply mvnd to license check
bowenliang123 97becc0
add `build/maven-mvnd-*/**` to rat-excludes
bowenliang123 bc8da13
Merge branch 'master' into ci-mvnd
bowenliang123 a93bba7
style
bowenliang123 380c1c9
update cache key
bowenliang123 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| # | ||
| # Licensed to the Apache Software Foundation (ASF) under one or more | ||
| # contributor license agreements. See the NOTICE file distributed with | ||
| # this work for additional information regarding copyright ownership. | ||
| # The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| # (the "License"); you may not use this file except in compliance with | ||
| # the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| # | ||
|
|
||
| name: 'setup-mvnd' | ||
| description: 'Setup the maven daemon' | ||
| runs: | ||
| using: composite | ||
| steps: | ||
| - name: Cache Mvnd | ||
| uses: actions/cache@v3 | ||
| with: | ||
| path: | | ||
| build/maven-mvnd-* | ||
| build/apache-maven-* | ||
| key: setup-mvnd-${{ runner.os }}-mvnd | ||
| - name: Check Mvnd | ||
| run: build/mvnd -v | ||
| shell: bash |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,134 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| # | ||
| # Licensed to the Apache Software Foundation (ASF) under one or more | ||
| # contributor license agreements. See the NOTICE file distributed with | ||
| # this work for additional information regarding copyright ownership. | ||
| # The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| # (the "License"); you may not use this file except in compliance with | ||
| # the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| # | ||
|
|
||
| # Determine the current working directory | ||
| _DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
| # Preserve the calling directory | ||
| _CALLING_DIR="$(pwd)" | ||
| # Options used during compilation | ||
| _COMPILE_JVM_OPTS="-Xms2g -Xmx2g -XX:ReservedCodeCacheSize=1g -Xss128m" | ||
|
|
||
| if [ "$CI" ]; then | ||
| export MAVEN_CLI_OPTS="--no-transfer-progress --errors --fail-fast" | ||
| fi | ||
|
|
||
| # Installs any application tarball given a URL, the expected tarball name, | ||
| # and, optionally, a checkable binary path to determine if the binary has | ||
| # already been installed | ||
| ## Arg1 - URL | ||
| ## Arg2 - Tarball Name | ||
| ## Arg3 - Checkable Binary | ||
| install_app() { | ||
| local remote_tarball="$1/$2" | ||
| local local_tarball="${_DIR}/$2" | ||
| local binary="${_DIR}/$3" | ||
|
|
||
| # setup `curl` and `wget` silent options if we're running on Jenkins | ||
| local curl_opts="-L" | ||
| local wget_opts="" | ||
| curl_opts="--progress-bar ${curl_opts}" | ||
| wget_opts="--progress=bar:force ${wget_opts}" | ||
|
|
||
| if [ -z "$3" ] || [ ! -f "$binary" ]; then | ||
| # check if we already have the tarball | ||
| # check if we have curl installed | ||
| # download application | ||
| rm -f "$local_tarball" | ||
| [ ! -f "${local_tarball}" ] && [ "$(command -v curl)" ] && \ | ||
| echo "exec: curl ${curl_opts} ${remote_tarball}" 1>&2 && \ | ||
| curl ${curl_opts} "${remote_tarball}" > "${local_tarball}" | ||
| # if the file still doesn't exist, lets try `wget` and cross our fingers | ||
| [ ! -f "${local_tarball}" ] && [ "$(command -v wget)" ] && \ | ||
| echo "exec: wget ${wget_opts} ${remote_tarball}" 1>&2 && \ | ||
| wget ${wget_opts} -O "${local_tarball}" "${remote_tarball}" | ||
| # if both were unsuccessful, exit | ||
| [ ! -f "${local_tarball}" ] && \ | ||
| echo -n "ERROR: Cannot download $2 with cURL or wget; " && \ | ||
| echo "please install manually and try again." && \ | ||
| exit 2 | ||
| cd "${_DIR}" && tar -xzf "$2" | ||
| rm -rf "$local_tarball" | ||
| fi | ||
| } | ||
|
|
||
| function get_os_type() { | ||
| local unameOsOut=$(uname -s) | ||
| local osType | ||
| case "${unameOsOut}" in | ||
| Linux*) osType=linux ;; | ||
| Darwin*) osType=darwin ;; | ||
| CYGWIN*) osType=windows ;; | ||
| MINGW*) osType=windows ;; | ||
| *) osType="UNKNOWN:${unameOsOut}" ;; | ||
| esac | ||
| echo "$osType" | ||
| } | ||
|
|
||
| function get_os_arch() { | ||
| local unameArchOut="$(uname -m)" | ||
| local arch | ||
| case "${unameArchOut}" in | ||
| x86_64*) arch=amd64 ;; | ||
| arm64*) arch=aarch64 ;; | ||
| *) arch="UNKNOWN:${unameOsOut}" ;; | ||
| esac | ||
| echo "$arch" | ||
| } | ||
|
|
||
| # Determine the Mvnd version from the root pom.xml file and | ||
| # install mvnd under the build/ folder if needed. | ||
| function install_mvnd() { | ||
| local MVND_VERSION=$(grep "<mvnd.version>" "${_DIR}/../pom.xml" | head -n1 | awk -F '[<>]' '{print $3}') | ||
| local MVN_VERSION=$(grep "<maven.version>" "${_DIR}/../pom.xml" | head -n1 | awk -F '[<>]' '{print $3}') | ||
| MVND_BIN="$(command -v mvnd)" | ||
| if [ "$MVND_BIN" ]; then | ||
| local MVND_DETECTED_VERSION="$(mvnd -v 2>&1 | grep '(mvnd)' | awk '{print $5}')" | ||
| local MVN_DETECTED_VERSION="$(mvnd -v 2>&1 | grep 'Apache Maven' | awk 'NR==2 {print $3}')" | ||
| fi | ||
| # See simple version normalization: http://stackoverflow.com/questions/16989598/bash-comparing-version-numbers | ||
| function version { echo "$@" | awk -F. '{ printf("%03d%03d%03d\n", $1,$2,$3); }'; } | ||
|
|
||
| if [ $(version $MVND_DETECTED_VERSION) -ne $(version $MVND_VERSION) ]; then | ||
| local APACHE_MIRROR=${APACHE_MIRROR:-'https://downloads.apache.org'} | ||
| local OS_TYPE=$(get_os_type) | ||
| local ARCH=$(get_os_arch) | ||
|
|
||
| install_app \ | ||
| "${APACHE_MIRROR}/maven/mvnd/${MVND_VERSION}" \ | ||
| "maven-mvnd-${MVND_VERSION}-${OS_TYPE}-${ARCH}.tar.gz" \ | ||
| "maven-mvnd-${MVND_VERSION}-${OS_TYPE}-${ARCH}/bin/mvnd" | ||
|
|
||
| MVND_BIN="${_DIR}/maven-mvnd-${MVND_VERSION}-${OS_TYPE}-${ARCH}/bin/mvnd" | ||
| else | ||
| if [ "$(version $MVN_DETECTED_VERSION)" -ne "$(version $MVN_VERSION)" ]; then | ||
| echo "Mvnd $MVND_DETECTED_VERSION embedded maven version $MVN_DETECTED_VERSION is not equivalent to $MVN_VERSION required in pom." | ||
| exit 1 | ||
| fi | ||
| fi | ||
| } | ||
|
|
||
| install_mvnd | ||
|
|
||
| cd "${_CALLING_DIR}" | ||
|
|
||
| # Set any `mvn` options if not already present | ||
| export MAVEN_OPTS=${MAVEN_OPTS:-"$_COMPILE_JVM_OPTS"} | ||
|
|
||
| echo "Using \`mvnd\` from path: $MVND_BIN" 1>&2 | ||
| ${MVND_BIN} $MAVEN_CLI_OPTS "$@" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Github Action limit memory: 6947m, we may increase the max heap size?
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
build/mvndis designed for the usage of both on local machine and also the GitHub action, so I would prefer to keep -Xmx here the same as inbuild/mvn. And we could increase the memory limit in CI jobs by passing opts.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L950
Marking the memory limit in Spark as above.