[SPARK-30572][BUILD] Add a fallback Maven repository #27281

dongjoon-hyun · 2020-01-19T21:24:23Z

What changes were proposed in this pull request?

This PR aims to add a fallback Maven repository when a mirror to central fail.

Why are the changes needed?

We use Google Maven Central in GitHub Action as a mirror of central.
However, Google Maven Central sometimes doesn't have newly published artifacts
and there is no guarantee when we get the newly published artifacts.

By duplicating Maven Central with a new ID, we can add a fallback Maven repository
which is not mirrored by Google Maven Central.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manually testing with the new Twitter chill artifacts by switching chill.version from 0.9.3 to 0.9.5.

$ rm -rf ~/.m2/repository/com/twitter/chill*
$ mvn compile | grep chill
Downloading from google-maven-central: https://maven-central.storage-download.googleapis.com/repos/central/data/com/twitter/chill_2.12/0.9.5/chill_2.12-0.9.5.pom
Downloading from central_without_mirror: https://repo.maven.apache.org/maven2/com/twitter/chill_2.12/0.9.5/chill_2.12-0.9.5.pom
Downloaded from central_without_mirror: https://repo.maven.apache.org/maven2/com/twitter/chill_2.12/0.9.5/chill_2.12-0.9.5.pom (2.8 kB at 11 kB/s)

This reverts commit 234bb2e.

dongjoon-hyun · 2020-01-19T21:48:51Z

Note that I tested with the commit Test chill in this PR. Please ignore the dependency job failure in that because I didn't regenerate the dependency files. All the other build jobs succeed in the same commit. (In case of download failure, the build job's mvn command fails in 1 minutes.

srowen · 2020-01-19T23:19:56Z

pom.xml

@@ -257,6 +257,22 @@
        <enabled>false</enabled>
      </snapshots>
    </repository>
+    <repository>


Yeah I get it. At this point though, would it just be easier to declare the Google repo above central above, and not use the mirror settings?

Ur, I'm not sure that we can declare Google Maven Central as Apache Spark default repo because not everyone welcome Google repo. In Apache Spark, we only use this in GitHub Action. The other build environment, we don't use Google Maven Central at all.

Are you sure, @srowen ?

OK, it could be listed second. I don't see a problem with including a mirror as a repository here, as it's just a mirror of central. (We have had in the past, even, vendor-specific repos.) I hope it would have the same effect in improving robustness as anyone running the build would have it look at both repos before failing.

SparkQA · 2020-01-19T23:33:10Z

Test build #117023 has finished for PR 27281 at commit b77fd9b.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-01-19T23:58:01Z

retest this please

SparkQA · 2020-01-20T00:00:22Z

Test build #117024 has finished for PR 27281 at commit 81b11a4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-01-20T00:21:56Z

pom.xml

+        This is used as a fallback when a mirror to `central` fail.
+        For example, when we use Google Maven Central in GitHub Action as a mirror of `central`,
+        this will be used when Google Maven Central is out of sync due to its late sync cycle.
+      -->


Oh, I mistakenly thought the maven automatically would do the fallback in the mirror failure case.

How about leaving some comments, too, about this behaivour in the script of GitHub Actions?

spark/.github/workflows/master.yml

Line 69 in 19a1059

echo "<settings><mirrors><mirror><id>google-maven-central</id><name>GCS Maven Central mirror</name><url>https://maven-central.storage-download.googleapis.com/repos/central/data/</url><mirrorOf>central</mirrorOf></mirror></mirrors></settings>" > ~/.m2/settings.xml

Sure, @maropu .

dongjoon-hyun · 2020-01-20T00:45:23Z

Hi, @srowen and @maropu . I added the following comment to the GitHub Action script. As I wrote in the comment, this PR aims to fix the issue while keeping the existing Jenkins (and downstream) environment unaffected.

# `Maven Central` is too flaky in terms of downloading artifacts in `GitHub Action` environment.
# `Google Maven Central Mirror` is too slow in terms of sycing upstream. To get the best combination,
#   1) we set `Google Maven Central` as a mirror of `central` in `GitHub Action` environment only.
#   2) we duplicates `Maven Central` in pom.xml with ID `central_without_mirror`.
# In other words, in GitHub Action environment, `central` is mirrored by `Google Maven Central` first.
# If `Google Maven Central` doesn't provide the artifact due to its slowness, `central_without_mirror` will be used.
# Note that we aim to achieve the above while keeping the existing behavior of non-`GitHub Action` environment unchanged.

maropu

Looks fine to me, thanks, @dongjoon-hyun !

dongjoon-hyun · 2020-01-20T01:42:01Z

Thank you, @maropu and @srowen .
I'll merge this to unblock the other PRs.
Merged to master and branch-2.4

### What changes were proposed in this pull request? This PR aims to add a fallback Maven repository when a mirror to `central` fail. ### Why are the changes needed? We use `Google Maven Central` in GitHub Action as a mirror of `central`. However, `Google Maven Central` sometimes doesn't have newly published artifacts and there is no guarantee when we get the newly published artifacts. By duplicating `Maven Central` with a new ID, we can add a fallback Maven repository which is not mirrored by `Google Maven Central`. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Manually testing with the new `Twitter` chill artifacts by switching `chill.version` from `0.9.3` to `0.9.5`. ``` $ rm -rf ~/.m2/repository/com/twitter/chill* $ mvn compile | grep chill Downloading from google-maven-central: https://maven-central.storage-download.googleapis.com/repos/central/data/com/twitter/chill_2.12/0.9.5/chill_2.12-0.9.5.pom Downloading from central_without_mirror: https://repo.maven.apache.org/maven2/com/twitter/chill_2.12/0.9.5/chill_2.12-0.9.5.pom Downloaded from central_without_mirror: https://repo.maven.apache.org/maven2/com/twitter/chill_2.12/0.9.5/chill_2.12-0.9.5.pom (2.8 kB at 11 kB/s) ``` Closes #27281 from dongjoon-hyun/SPARK-30572. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit c992716) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

srowen · 2020-01-20T01:53:16Z

OK but this ends up kind of weird, no? the same repo is listed twice. It only helps the original problem you're trying to solve if you also declare a mirror. Why not just list both as repos? I think it achieves what you want here, which is to actually have the build check with two repos if needed.

SparkQA · 2020-01-20T02:41:40Z

Test build #117035 has finished for PR 27281 at commit 81b11a4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-01-20T03:09:27Z

Does the community and downstream agree to use Google Maven Mirror in their environment, @srowen ?

Why not just list both as repos?

SparkQA · 2020-01-20T03:44:35Z

Test build #117039 has finished for PR 27281 at commit f043826.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2020-01-20T15:30:21Z

Why would the mirror be a problem? I'm missing that. Right now you're kind of saying it's required anyway to work around sbt hard-coding an http: ref to the main central repo.

Oh, or is the point that that problem isn't fixable with a new repository and can only be resolved with a mirror, so this is how it has to work out?

Well it ends up a little weird but OK if so.

dongjoon-hyun · 2020-01-21T04:04:05Z

Hi, @srowen . Sorry, but I cannot understand your context.

First of all, this PR is irrelevant to the following. And, I didn't mention like that.

Right now you're kind of saying it's required anyway to work around sbt hard-coding an http: ref to the main central repo.

Retry doesn't change the HTTP protocol, does it?

This PR aims to help your PR ([SPARK-29290][CORE] Update to chill 0.9.5) which is blocked by GitHub Action failure due to the late sync period of Google Maven Central mirror.

Second, for GitHub Action linter failure, I just update lint-scala to use mvn instead of sbt (3848999) to unblock the community. SBT still has the issue like I wrote in the dev mailing list (https://lists.apache.org/thread.html/r74787faf95f94052f134b8bece48d44f7154cad5bde1dab62119db4e%40%3Cdev.spark.apache.org%3E)

I guess Jenkins SBT job will fail if we clean up the Maven local cache.

Technically, I'm not working on SBT issue itself at all.

srowen · 2020-01-21T14:23:18Z

@dongjoon-hyun we have three issues:

Maven Central flaky at times?
SBT trying to use http:// for Maven Central
Some artifacts not mirrored to GCS repo

To fix 1, we need a second repo like GCS. But to fix 3, we also still need Maven Central. So far, that just means adding both as repositories to the build.

But to fix 2, it seems like we have to try to change what SBT thinks 'central' is, which requires specifying a mirror of central that uses https. It isn't sufficient to just define it in the POM / SBT build. Right?

If so then I get this change. Let me comment on #27307

dongjoon-hyun · 2020-01-21T18:00:57Z

Yes. It does. Sure~
BTW, according to @HyukjinKwon investigation, it turns out that this additional redundant repository provides a workaround for the hard-coded SBT pom reader issue. Please see
#27242 (comment)

### What changes were proposed in this pull request? This PR proposes to address four things. Three issues and fixes were a bit mixed so this PR sorts it out. See also http://apache-spark-developers-list.1001551.n3.nabble.com/Adding-Maven-Central-mirror-from-Google-to-the-build-td28728.html for the discussion in the mailing list. 1. Add the Google Maven Central mirror (GCS) as a primary repository. This will not only help development more stable but also in order to make Github Actions build (where it is always required to download jars) stable. In case of Jenkins PR builder, it wouldn't be affected too much as it uses the pre-downloaded jars under `.m2`. - Google Maven Central seems stable for heavy workload but not synced very quickly (e.g., new release is missing) - Maven Central (default) seems less stable but synced quickly. We already added this GCS mirror as a default additional remote repository at SPARK-29175. So I don't see an issue to add it as a repo. https://github.com/apache/spark/blob/abf759a91e01497586b8bb6b7a314dd28fd6cff1/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L2111-L2118 2. Currently, we have the hard-corded repository in [`sbt-pom-reader`](https://github.com/JoshRosen/sbt-pom-reader/blob/v1.0.0-spark/src/main/scala/com/typesafe/sbt/pom/MavenPomResolver.scala#L32) and this seems overwriting Maven's existing resolver by the same ID `central` with `http://` when initially the pom file is ported into SBT instance. This uses `http://` which latently Maven Central disallowed (see #27242) My speculation is that we just need to be able to load plugin and let it convert POM to SBT instance with another fallback repo. After that, it _seems_ using `central` with `https` properly. See also #27307 (comment). I double checked that we use `https` properly from the SBT build as well: ``` [debug] downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom ... [debug] public: downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom [debug] public: downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom.sha1 ``` This was fixed by adding the same repo (#27281), `central_without_mirror`, which is a bit awkward. Instead, this PR adds GCS as a main repo, and community Maven central as a fallback repo. So, presumably the community Maven central repo is used when the plugin is loaded as a fallback. 3. While I am here, I fix another issue. Github Action at #27279 is being failed. The reason seems to be scalafmt 1.0.3 is in Maven central but not in GCS. ``` org.apache.maven.plugin.PluginResolutionException: Plugin org.antipathy:mvn-scalafmt_2.12:1.0.3 or one of its dependencies could not be resolved: Could not find artifact org.antipathy:mvn-scalafmt_2.12:jar:1.0.3 in google-maven-central (https://maven-central.storage-download.googleapis.com/repos/central/data/) at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve (DefaultPluginDependenciesResolver.java:131) ``` `mvn-scalafmt` exists in Maven central: ```bash $ curl https://repo.maven.apache.org/maven2/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom ``` ```xml <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> ... ``` whereas not in GCS mirror: ```bash $ curl https://maven-central.storage-download.googleapis.com/repos/central/data/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom ``` ```xml <?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: maven-central/repos/central/data/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom</Details></Error>% ``` In this PR, simply make both repos accessible by adding to `pluginRepositories`. 4. Remove the workarounds in Github Actions to switch mirrors because now we have same repos in the same order (Google Maven Central first, and Maven Central second) ### Why are the changes needed? To make the build and Github Action more stable. ### Does this PR introduce any user-facing change? No, dev only change. ### How was this patch tested? I roughly checked local and PR against my fork (HyukjinKwon#2 and HyukjinKwon#3). Closes #27307 from HyukjinKwon/SPARK-30572. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: HyukjinKwon <gurwls223@apache.org>

This PR proposes to address four things. Three issues and fixes were a bit mixed so this PR sorts it out. See also http://apache-spark-developers-list.1001551.n3.nabble.com/Adding-Maven-Central-mirror-from-Google-to-the-build-td28728.html for the discussion in the mailing list. 1. Add the Google Maven Central mirror (GCS) as a primary repository. This will not only help development more stable but also in order to make Github Actions build (where it is always required to download jars) stable. In case of Jenkins PR builder, it wouldn't be affected too much as it uses the pre-downloaded jars under `.m2`. - Google Maven Central seems stable for heavy workload but not synced very quickly (e.g., new release is missing) - Maven Central (default) seems less stable but synced quickly. We already added this GCS mirror as a default additional remote repository at SPARK-29175. So I don't see an issue to add it as a repo. https://github.com/apache/spark/blob/abf759a91e01497586b8bb6b7a314dd28fd6cff1/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L2111-L2118 2. Currently, we have the hard-corded repository in [`sbt-pom-reader`](https://github.com/JoshRosen/sbt-pom-reader/blob/v1.0.0-spark/src/main/scala/com/typesafe/sbt/pom/MavenPomResolver.scala#L32) and this seems overwriting Maven's existing resolver by the same ID `central` with `http://` when initially the pom file is ported into SBT instance. This uses `http://` which latently Maven Central disallowed (see apache#27242) My speculation is that we just need to be able to load plugin and let it convert POM to SBT instance with another fallback repo. After that, it _seems_ using `central` with `https` properly. See also apache#27307 (comment). I double checked that we use `https` properly from the SBT build as well: ``` [debug] downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom ... [debug] public: downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom [debug] public: downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom.sha1 ``` This was fixed by adding the same repo (apache#27281), `central_without_mirror`, which is a bit awkward. Instead, this PR adds GCS as a main repo, and community Maven central as a fallback repo. So, presumably the community Maven central repo is used when the plugin is loaded as a fallback. 3. While I am here, I fix another issue. Github Action at apache#27279 is being failed. The reason seems to be scalafmt 1.0.3 is in Maven central but not in GCS. ``` org.apache.maven.plugin.PluginResolutionException: Plugin org.antipathy:mvn-scalafmt_2.12:1.0.3 or one of its dependencies could not be resolved: Could not find artifact org.antipathy:mvn-scalafmt_2.12:jar:1.0.3 in google-maven-central (https://maven-central.storage-download.googleapis.com/repos/central/data/) at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve (DefaultPluginDependenciesResolver.java:131) ``` `mvn-scalafmt` exists in Maven central: ```bash $ curl https://repo.maven.apache.org/maven2/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom ``` ```xml <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> ... ``` whereas not in GCS mirror: ```bash $ curl https://maven-central.storage-download.googleapis.com/repos/central/data/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom ``` ```xml <?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: maven-central/repos/central/data/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom</Details></Error>% ``` In this PR, simply make both repos accessible by adding to `pluginRepositories`. 4. Remove the workarounds in Github Actions to switch mirrors because now we have same repos in the same order (Google Maven Central first, and Maven Central second) To make the build and Github Action more stable. No, dev only change. I roughly checked local and PR against my fork (#2 and #3). Closes apache#27307 from HyukjinKwon/SPARK-30572. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: HyukjinKwon <gurwls223@apache.org>

…ository ### What changes were proposed in this pull request? This PR backports #27307 This PR proposes to address four things. Three issues and fixes were a bit mixed so this PR sorts it out. See also http://apache-spark-developers-list.1001551.n3.nabble.com/Adding-Maven-Central-mirror-from-Google-to-the-build-td28728.html for the discussion in the mailing list. 1. Add the Google Maven Central mirror (GCS) as a primary repository. This will not only help development more stable but also in order to make Github Actions build (where it is always required to download jars) stable. In case of Jenkins PR builder, it wouldn't be affected too much as it uses the pre-downloaded jars under `.m2`. - Google Maven Central seems stable for heavy workload but not synced very quickly (e.g., new release is missing) - Maven Central (default) seems less stable but synced quickly. We already added this GCS mirror as a default additional remote repository at SPARK-29175. So I don't see an issue to add it as a repo. https://github.com/apache/spark/blob/abf759a91e01497586b8bb6b7a314dd28fd6cff1/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L2111-L2118 2. Currently, we have the hard-corded repository in [`sbt-pom-reader`](https://github.com/JoshRosen/sbt-pom-reader/blob/v1.0.0-spark/src/main/scala/com/typesafe/sbt/pom/MavenPomResolver.scala#L32) and this seems overwriting Maven's existing resolver by the same ID `central` with `http://` when initially the pom file is ported into SBT instance. This uses `http://` which latently Maven Central disallowed (see #27242) My speculation is that we just need to be able to load plugin and let it convert POM to SBT instance with another fallback repo. After that, it _seems_ using `central` with `https` properly. See also #27307 (comment). I double checked that we use `https` properly from the SBT build as well: ``` [debug] downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom ... [debug] public: downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom [debug] public: downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom.sha1 ``` This was fixed by adding the same repo (#27281), `central_without_mirror`, which is a bit awkward. Instead, this PR adds GCS as a main repo, and community Maven central as a fallback repo. So, presumably the community Maven central repo is used when the plugin is loaded as a fallback. 3. While I am here, I fix another issue. Github Action at #27279 is being failed. The reason seems to be scalafmt 1.0.3 is in Maven central but not in GCS. ``` org.apache.maven.plugin.PluginResolutionException: Plugin org.antipathy:mvn-scalafmt_2.12:1.0.3 or one of its dependencies could not be resolved: Could not find artifact org.antipathy:mvn-scalafmt_2.12:jar:1.0.3 in google-maven-central (https://maven-central.storage-download.googleapis.com/repos/central/data/) at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve (DefaultPluginDependenciesResolver.java:131) ``` `mvn-scalafmt` exists in Maven central: ```bash $ curl https://repo.maven.apache.org/maven2/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom ``` ```xml <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> ... ``` whereas not in GCS mirror: ```bash $ curl https://maven-central.storage-download.googleapis.com/repos/central/data/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom ``` ```xml <?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: maven-central/repos/central/data/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom</Details></Error>% ``` In this PR, simply make both repos accessible by adding to `pluginRepositories`. 4. Remove the workarounds in Github Actions to switch mirrors because now we have same repos in the same order (Google Maven Central first, and Maven Central second) ### Why are the changes needed? To make the build and Github Action more stable. ### Does this PR introduce any user-facing change? No, dev only change. ### How was this patch tested? I roughly checked local and PR against my fork (HyukjinKwon#2 and HyukjinKwon#3). Closes #27335 from HyukjinKwon/tmp. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

This PR aims to add a fallback Maven repository when a mirror to `central` fail. We use `Google Maven Central` in GitHub Action as a mirror of `central`. However, `Google Maven Central` sometimes doesn't have newly published artifacts and there is no guarantee when we get the newly published artifacts. By duplicating `Maven Central` with a new ID, we can add a fallback Maven repository which is not mirrored by `Google Maven Central`. No. Manually testing with the new `Twitter` chill artifacts by switching `chill.version` from `0.9.3` to `0.9.5`. ``` $ rm -rf ~/.m2/repository/com/twitter/chill* $ mvn compile | grep chill Downloading from google-maven-central: https://maven-central.storage-download.googleapis.com/repos/central/data/com/twitter/chill_2.12/0.9.5/chill_2.12-0.9.5.pom Downloading from central_without_mirror: https://repo.maven.apache.org/maven2/com/twitter/chill_2.12/0.9.5/chill_2.12-0.9.5.pom Downloaded from central_without_mirror: https://repo.maven.apache.org/maven2/com/twitter/chill_2.12/0.9.5/chill_2.12-0.9.5.pom (2.8 kB at 11 kB/s) ``` Closes apache#27281 from dongjoon-hyun/SPARK-30572. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

### What changes were proposed in this pull request? This PR aims to add a fallback Maven repository when a mirror to `central` fail. ### Why are the changes needed? We use `Google Maven Central` in GitHub Action as a mirror of `central`. However, `Google Maven Central` sometimes doesn't have newly published artifacts and there is no guarantee when we get the newly published artifacts. By duplicating `Maven Central` with a new ID, we can add a fallback Maven repository which is not mirrored by `Google Maven Central`. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Manually testing with the new `Twitter` chill artifacts by switching `chill.version` from `0.9.3` to `0.9.5`. ``` $ rm -rf ~/.m2/repository/com/twitter/chill* $ mvn compile | grep chill Downloading from google-maven-central: https://maven-central.storage-download.googleapis.com/repos/central/data/com/twitter/chill_2.12/0.9.5/chill_2.12-0.9.5.pom Downloading from central_without_mirror: https://repo.maven.apache.org/maven2/com/twitter/chill_2.12/0.9.5/chill_2.12-0.9.5.pom Downloaded from central_without_mirror: https://repo.maven.apache.org/maven2/com/twitter/chill_2.12/0.9.5/chill_2.12-0.9.5.pom (2.8 kB at 11 kB/s) ``` Closes apache#27281 from dongjoon-hyun/SPARK-30572. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit c992716) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

dongjoon-hyun added 4 commits January 19, 2020 13:23

[SPARK-30572][BUILD] Add a fallback Maven repository

31a64f8

a

b77fd9b

Test chill

234bb2e

Revert "Test chill"

81b11a4

This reverts commit 234bb2e.

This was referenced Jan 19, 2020

[SPARK-29290][CORE] Update to chill 0.9.5 #27227

Closed

[SPARK-30486][BUILD] Bump lz4-java version to 1.7.1 #27271

Closed

dongjoon-hyun added the BUILD label Jan 19, 2020

dongjoon-hyun mentioned this pull request Jan 19, 2020

[SPARK-30570][BUILD] Update scalafmt plugin to 1.0.3 with onlyChangedFiles feature #27279

Closed

srowen reviewed Jan 19, 2020

View reviewed changes

maropu reviewed Jan 20, 2020

View reviewed changes

Add comment

f043826

maropu approved these changes Jan 20, 2020

View reviewed changes

dongjoon-hyun closed this in c992716 Jan 20, 2020

dongjoon-hyun deleted the SPARK-30572 branch January 20, 2020 01:43

HyukjinKwon mentioned this pull request Jan 22, 2020

[SPARK-30601][BUILD] Add a Google Maven Central as a primary repository #27307

Closed

HyukjinKwon mentioned this pull request Jan 23, 2020

[SPARK-30601][BUILD][2.4] Add a Google Maven Central as a primary repository #27335

Closed

yifeih mentioned this pull request Jan 27, 2020

[SPARK-30572][BUILD] Add a fallback Maven repository palantir/spark#636

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-30572][BUILD] Add a fallback Maven repository #27281

[SPARK-30572][BUILD] Add a fallback Maven repository #27281

dongjoon-hyun commented Jan 19, 2020 •

edited

Loading

dongjoon-hyun commented Jan 19, 2020 •

edited

Loading

srowen Jan 19, 2020

dongjoon-hyun Jan 20, 2020

dongjoon-hyun Jan 20, 2020

srowen Jan 20, 2020

SparkQA commented Jan 19, 2020

maropu commented Jan 19, 2020

SparkQA commented Jan 20, 2020

maropu Jan 20, 2020

dongjoon-hyun Jan 20, 2020

maropu Jan 20, 2020

dongjoon-hyun commented Jan 20, 2020 •

edited

Loading

maropu left a comment

dongjoon-hyun commented Jan 20, 2020 •

edited by HyukjinKwon

Loading

srowen commented Jan 20, 2020

SparkQA commented Jan 20, 2020

dongjoon-hyun commented Jan 20, 2020 •

edited

Loading

SparkQA commented Jan 20, 2020

srowen commented Jan 20, 2020

dongjoon-hyun commented Jan 21, 2020 •

edited

Loading

srowen commented Jan 21, 2020

dongjoon-hyun commented Jan 21, 2020

[SPARK-30572][BUILD] Add a fallback Maven repository #27281

[SPARK-30572][BUILD] Add a fallback Maven repository #27281

Conversation

dongjoon-hyun commented Jan 19, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

dongjoon-hyun commented Jan 19, 2020 • edited Loading

srowen Jan 19, 2020

Choose a reason for hiding this comment

dongjoon-hyun Jan 20, 2020

Choose a reason for hiding this comment

dongjoon-hyun Jan 20, 2020

Choose a reason for hiding this comment

srowen Jan 20, 2020

Choose a reason for hiding this comment

SparkQA commented Jan 19, 2020

maropu commented Jan 19, 2020

SparkQA commented Jan 20, 2020

maropu Jan 20, 2020

Choose a reason for hiding this comment

dongjoon-hyun Jan 20, 2020

Choose a reason for hiding this comment

maropu Jan 20, 2020

Choose a reason for hiding this comment

dongjoon-hyun commented Jan 20, 2020 • edited Loading

maropu left a comment

Choose a reason for hiding this comment

dongjoon-hyun commented Jan 20, 2020 • edited by HyukjinKwon Loading

srowen commented Jan 20, 2020

SparkQA commented Jan 20, 2020

dongjoon-hyun commented Jan 20, 2020 • edited Loading

SparkQA commented Jan 20, 2020

srowen commented Jan 20, 2020

dongjoon-hyun commented Jan 21, 2020 • edited Loading

srowen commented Jan 21, 2020

dongjoon-hyun commented Jan 21, 2020

dongjoon-hyun commented Jan 19, 2020 •

edited

Loading

dongjoon-hyun commented Jan 19, 2020 •

edited

Loading

dongjoon-hyun commented Jan 20, 2020 •

edited

Loading

dongjoon-hyun commented Jan 20, 2020 •

edited by HyukjinKwon

Loading

dongjoon-hyun commented Jan 20, 2020 •

edited

Loading

dongjoon-hyun commented Jan 21, 2020 •

edited

Loading