2.11 build can't resolve Scala

[SethTisue]

https://scala-ci.typesafe.com/view/scala-2.11.x/job/scala-2.11.x-integrate-community-build/2000/

[scala-library] Extracting dependencies for: scala-library
[scala-library] Resolving artifact org.scala-lang:scala-library:pom:2.11.12-bin-2e2f65a
[scala-library] Could not resolve artifact!
[scala-library:error] The artifact could not be resolved from any of:
[scala-library:error] dbuild (https://scala-ci.typesafe.com/artifactory/dbuild/, default, releases+snapshots)
[scala:error] java.lang.RuntimeException: The artifact could not be resolved
[scala:error] 	at scala.sys.package$.error(package.scala:27)

this isn't intermittent, it's happening every time now, but only on the JDK 8 build (though perhaps if I zapped the workspace and/or the dbuild cache in the JDK 11 build it would happen there too?)

[SethTisue]

reproducible outside of dbuild with scala-runners:

% scala --scala-version 2.11.12-bin-2e2f65a
Resolution error: Error downloading org.scala-lang:scala-compiler:2.11.12-bin-2e2f65a
  not found: https://repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.11.12-bin-2e2f65a/scala-compiler-2.11.12-bin-2e2f65a.pom
  not found: https://scala-ci.typesafe.com/artifactory/scala-integration/org/scala-lang/scala-compiler/2.11.12-bin-2e2f65a/scala-compiler-2.11.12-bin-2e2f65a.pom

perhaps this is a consequence of scala/scala-dev#720 (Artifactory config changes) that just took time to rear its head

[SethTisue]

puzzling: 2.11.12-bin-3fd7aec does resolve — that's a July 2019 commit

whereas 2e2f65a is a December 2019 commit — it's the merge commit for the one PR we merged to 2.11.x in that time

but doesn't really make this any less mysterious. how/why was this working before?

[SethTisue]

aha, found a crucial clue

over at https://scala-ci.typesafe.com/view/scala-2.11.x/job/scala-2.11.x-integrate-bootstrap/ we've been endlessly rebuilding 2.11.12-bin-2e2f65a

the last green run was https://scala-ci.typesafe.com/view/scala-2.11.x/job/scala-2.11.x-integrate-bootstrap/1705/, 7 days ago

and then the runs from https://scala-ci.typesafe.com/view/scala-2.11.x/job/scala-2.11.x-integrate-bootstrap/1706/ on are red

the bad runs end with:

### Building locker
+++ ant -Dmaven.version.number=2.11.12-bin-2e2f65a -Dremote.snapshot.repository=NOPE -Dremote.release.repository=https://scala-ci.typesafe.com/artifactory/scala-integration/ -Drepository.credentials.id=private-repo -Dscalac.args.optimise=-optimise -Ddocs.skip=1 -Dlocker.skip=1 publish
Build step 'Execute shell' marked build as failure

[SethTisue]

ha, but the real error is in https://scala-ci.typesafe.com/view/scala-2.11.x/job/scala-2.11.x-integrate-bootstrap/1706/artifact/logs/builds/*view*/

BUILD FAILED
/home/jenkins/workspace/scala-2.11.x-integrate-bootstrap/build.xml:274: Unable to resolve artifact: Missing:
----------
1) biz.aQute:bnd:jar:1.50.0

  Try downloading the file manually from the project website.

  Then, install it using the command: 
      mvn install:install-file -DgroupId=biz.aQute -DartifactId=bnd -Dversion=1.50.0 -Dpackaging=jar -Dfile=/path/to/file

  Alternatively, if you host your own repository you can deploy the file there: 
      mvn deploy:deploy-file -DgroupId=biz.aQute -DartifactId=bnd -Dversion=1.50.0 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]

  Path to dependency: 
  	1) org.apache.maven:super-pom:jar:2.0
  	2) biz.aQute:bnd:jar:1.50.0

----------
1 required artifact is missing.

for artifact: 
  org.apache.maven:super-pom:jar:2.0

from the specified remote repositories:
  central-cache (https://scala-ci.typesafe.com/artifactory/central/)

so apparently I removed a resolver that the 2.11 build needs — just need to figure out which one it was and re-add it 👍

in the last green build:

[artifact:dependencies] Downloading: biz/aQute/bnd/1.50.0/bnd-1.50.0.pom from repository central-cache at https://scala-ci.typesafe.com/artifactory/central/
[artifact:dependencies] Transferring 1K from central-cache
[artifact:dependencies] Downloading: biz/aQute/bnd/1.50.0/bnd-1.50.0.jar from repository central-cache at https://scala-ci.typesafe.com/artifactory/central/
[artifact:dependencies] Transferring 809K from central-cache

[SethTisue]

huh, but Maven Central has it: https://search.maven.org/artifact/biz.aQute/bnd/1.50.0/jar

and Maven Central is definitely still part of our Artifactory config! or is it? it's there, but I guess it's conceivable that it's misconfigured, but due to caching this is the first time we noticed. hmm

[SethTisue]

and we have the files here: https://scala-ci.typesafe.com/artifactory/maven-central/biz/aQute/bnd/1.50.0/

but note the maven-central .whereas the error says from the specified remote repositories: central-cache (https://scala-ci.typesafe.com/artifactory/central/). there is a maven-central vs central mismatch

[SethTisue]

on 2.11.x in scripts/jobs/integrate/bootstrap we use a custom resolvers config:

[repositories]
  script-repo: $1
  jcenter-cache: $jcenterCacheUrl
  typesafe-ivy-releases: https://dl.bintray.com/typesafe/ivy-releases/, [organisation]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
  sbt-plugin-releases: https://dl.bintray.com/sbt/sbt-plugin-releases/, [organisation]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
  maven-central
  local

you would think that the maven-central would cause the needed artifacts to be retrieved directly from Maven Central without the scala-ci Artifactory config even mattering. but perhaps since the 2.11 build is ant-based, everything is ancient and maven-central tries to use http access, and that stopped working January 2020.

but why was it working until 7 days ago? well, I don't know, but I guess the needed artifacts were found anyway through some quirk of the old config that I don't understand. (we don't need to sink too much mental energy into the why-did-it-work-before question, unless doing so helps us fix it now!)

note the explicit jcenter-cache entry. it seems to me that this might be fixable by adding a similar maven-central-cache entry — or, equivalently, by replacing the maven-central line with a line with an explicit https URL

having it depend on our Artifactory's caching setup seems unnecessarily fragile, so I would like to try the explicit https URL solution path first

[SethTisue]

I would like to try the explicit https URL solution path first

trying it: scala/scala#9201

UPDATE: dammit, it didn't help

[SethTisue]

it seems to me that this might be fixable by adding a similar maven-central-cache entry

dammit, I tried this too and it also didn't help: https://scala-ci.typesafe.com/view/scala-2.11.x/job/scala-2.11.x-integrate-bootstrap/1714/artifact/logs/builds/*view*/

now I'm stumped again...

[SethTisue]

why is it looking at /central/? it isn't anywhere in scala/scala that I can find, so maybe it's set up that way on our Jenkins

so I looked over in scala/scala-jenkins-infra, even though since we moved off Chef it isn't current anymore, but could still have clues... and lo and behold, at https://github.com/scala/scala-jenkins-infra/blob/master/roles/worker/vars/main.yml we see:

repos_caching_proxy_central_url: "https://{{server_name}}/artifactory/central/"

and then over on jenkins-worker-publish we find that URL in ~/.m2/settings.xml 😎