Skip to content

Commit c992716

Browse files
committed
[SPARK-30572][BUILD] Add a fallback Maven repository
### What changes were proposed in this pull request? This PR aims to add a fallback Maven repository when a mirror to `central` fail. ### Why are the changes needed? We use `Google Maven Central` in GitHub Action as a mirror of `central`. However, `Google Maven Central` sometimes doesn't have newly published artifacts and there is no guarantee when we get the newly published artifacts. By duplicating `Maven Central` with a new ID, we can add a fallback Maven repository which is not mirrored by `Google Maven Central`. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Manually testing with the new `Twitter` chill artifacts by switching `chill.version` from `0.9.3` to `0.9.5`. ``` $ rm -rf ~/.m2/repository/com/twitter/chill* $ mvn compile | grep chill Downloading from google-maven-central: https://maven-central.storage-download.googleapis.com/repos/central/data/com/twitter/chill_2.12/0.9.5/chill_2.12-0.9.5.pom Downloading from central_without_mirror: https://repo.maven.apache.org/maven2/com/twitter/chill_2.12/0.9.5/chill_2.12-0.9.5.pom Downloaded from central_without_mirror: https://repo.maven.apache.org/maven2/com/twitter/chill_2.12/0.9.5/chill_2.12-0.9.5.pom (2.8 kB at 11 kB/s) ``` Closes #27281 from dongjoon-hyun/SPARK-30572. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
1 parent 3858e94 commit c992716

File tree

2 files changed

+23
-0
lines changed

2 files changed

+23
-0
lines changed

.github/workflows/master.yml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,13 @@ jobs:
6666
export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
6767
export MAVEN_CLI_OPTS="--no-transfer-progress"
6868
mkdir -p ~/.m2
69+
# `Maven Central` is too flaky in terms of downloading artifacts in `GitHub Action` environment.
70+
# `Google Maven Central Mirror` is too slow in terms of sycing upstream. To get the best combination,
71+
# 1) we set `Google Maven Central` as a mirror of `central` in `GitHub Action` environment only.
72+
# 2) we duplicates `Maven Central` in pom.xml with ID `central_without_mirror`.
73+
# In other words, in GitHub Action environment, `central` is mirrored by `Google Maven Central` first.
74+
# If `Google Maven Central` doesn't provide the artifact due to its slowness, `central_without_mirror` will be used.
75+
# Note that we aim to achieve the above while keeping the existing behavior of non-`GitHub Action` environment unchanged.
6976
echo "<settings><mirrors><mirror><id>google-maven-central</id><name>GCS Maven Central mirror</name><url>https://maven-central.storage-download.googleapis.com/repos/central/data/</url><mirrorOf>central</mirrorOf></mirror></mirrors></settings>" > ~/.m2/settings.xml
7077
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -P${{ matrix.hive }} -Phive-thriftserver -P${{ matrix.hadoop }} -Phadoop-cloud -Djava.version=${{ matrix.java }} install
7178
rm -rf ~/.m2/repository/org/apache/spark

pom.xml

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -257,6 +257,22 @@
257257
<enabled>false</enabled>
258258
</snapshots>
259259
</repository>
260+
<repository>
261+
<id>central_without_mirror</id>
262+
<!--
263+
This is used as a fallback when a mirror to `central` fail.
264+
For example, when we use Google Maven Central in GitHub Action as a mirror of `central`,
265+
this will be used when Google Maven Central is out of sync due to its late sync cycle.
266+
-->
267+
<name>Maven Repository</name>
268+
<url>https://repo.maven.apache.org/maven2</url>
269+
<releases>
270+
<enabled>true</enabled>
271+
</releases>
272+
<snapshots>
273+
<enabled>false</enabled>
274+
</snapshots>
275+
</repository>
260276
</repositories>
261277
<pluginRepositories>
262278
<pluginRepository>

0 commit comments

Comments
 (0)