pr-scala-integrate-partest now actually uses the distrib being tested

[xeno-by]

Apparently, it's not enough to set scalaVersion to point to the synthetic
version of the distrib that's just been built and published to maven.

It looks that regardless of the value of scalaVersion that's being set,
sbt is going to use the compiler provided in build.sbt of scala-partest.

I was quite surprised to see that, but I made sure that it's the stale
compiler that causes the problem. I put a random println in the new
compiler, build it, published it and ran a local equivalent of
pr-scala-integrate-partest. Unfortunately, I never saw the funny println
I introduced.

Therefore I propose to switch gears and use scalaHome instead of scalaVersion.

[xeno-by]

review @adriaanm @retronym

[retronym]

What did show scala-instance (the command after set scalaVersion := output before/after your change?

I'd like to characterize the underlying issue because we probably use the same facility in other builds.

[xeno-by]

Before:

[info] Scala instance{version label 2.11.0-06a9f7f-SNAPSHOT, 
actual version 2.11.0-20131004-094157-06a9f7f3f4, 
library jar: /Users/xeno_by/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.11.0-06a9f7f-SNAPSHOT.jar, 
compiler jar: /Users/xeno_by/.ivy2/cache/org.scala-lang/scala-compiler/jars/scala-compiler-2.11.0-06a9f7f-SNAPSHOT.jar}

After:

[info] Scala instance{version label 2.11.0-20131004-094157-06a9f7f3f4, 
actual version 2.11.0-20131004-094157-06a9f7f3f4, 
library jar: /Users/xeno_by/Projects/blackbox/build/pack/lib/scala-library.jar, 
compiler jar: /Users/xeno_by/Projects/blackbox/build/pack/lib/scala-compiler.jar}

[retronym]

I can fix this in a different way with:

/code/scala-partest git diff:

diff --git a/build.sbt b/build.sbt
index 7d149a8..eaae0b6 100644
--- a/build.sbt
+++ b/build.sbt
@@ -12,12 +12,15 @@ libraryDependencies += "com.googlecode.java-diff-utils" % "diffutils"      % "1.

 libraryDependencies += "org.scala-sbt"                  % "test-interface" % "1.0"

-libraryDependencies += "org.scalacheck"                %% "scalacheck"     % "1.10.1"
+libraryDependencies += "org.scalacheck"                %% "scalacheck"     % "1.10.1" intransitive()

-libraryDependencies += "org.scala-lang.modules"        %% "scala-xml"      % "1.0-RC2"
-
-libraryDependencies += "org.scala-lang"                 % "scalap"         % "2.11.0-M4"
+libraryDependencies += "org.scala-lang.modules"        %% "scala-xml"      % "1.0-RC2" intransitive()

+libraryDependencies ++= (
+   Seq("scalap", "scala-parser-combinators", "scala-compiler", "scala-reflect").map { artifact => 
+     "org.scala-lang" % artifact % scalaVersion.value intransitive()
+   }
+)

 // standard stuff follows:
 scalaVersion := "2.11.0-M4"

[retronym]

scala-partest is bringing in a transitive dependency on scala-compiler 2.11.0-M4 via scalap. SBT seems to be using that one in preference to the on in scalaInstance; I repeated your tests with adding ??? to Global and witnessing that the downstream build of scala-partest did not throw the expected exception.

It seems the scalaHome facility picks the right compiler.

But I'm really not sure about all of the forces in play. Need some advice from @adriaanm @gkossakowski

What filtering makes sense for the transitive dependencies that we use when building scala-partest?

What filtering/rewriting should we perform when publishing the POM of scala-partest.

[adriaanm]

@harrah, this looks like a real problem for us. I heard it may have been an ivy/maven caching issue instead of the original diagnosis. What's the final verdict?

[harrah]

@adriaanm, being a -SNAPSHOT published to a local Maven repository, it is probably sbt/sbt#321

[adriaanm]

ok, thanks for closing the loop. I'll close this PR until we hear differently. We really need to use the scalaVersion, as these artifacts are published to maven, there's not always a scala home available in PR validation

[xeno-by]

@adriaanm Then what do we do to fix the blocker for the blackbox pull request?

[adriaanm]

It's probably easier if you summarize the state of the PR there, I'm not sure which blocker you're referring to.

[xeno-by]

scala/scala#3001 (comment)

[retronym]

This script is actually about to be phased out in place of #39

[xeno-by]

@retronym Not sure if I understand. This PR is about job/pr-scala-integrate-partest. That one concerns job/pr-scala-integrate-ide-unified.

[retronym]

My bad.

[retronym]

We should be able to test the theory this bug by setting HOME=/tmp/home-$VERSION for the duration of this script so that we work with totally fresh Maven and Ivy caches.

[retronym]

I'll try that out and report back in a few hours.

[retronym]

-- six hours ago --
I'll try that out and report back in a few hours.

@xeno-by I found a few problems in our ant scripts. Most importantly, ant osgi.done did not propagate new JARs from build/pack automatically. It did however after you edited the build script (build.xml was included in the uptodate checks), which contributes an air of heisenbug to the problem.

I've fixed that problem, as well as finished what I started with docs.skip in scala/scala#3050.

With that fix in place, I consistently see my changes (e.g adding/removing ??? in Global) propagate to SBT with your integrate script, even without cleaning dists, .m2/repository, or .ivy2/cache.

So I don't know what to blame for the test failure in Jenkins. This should have been immune to these uptodate changes as it uses a clean checkout.

[xeno-by]

That's indeed strange. Maybe we should then go for #36 (comment)?

[adriaanm]

@harrah, we seem to have ruled out a caching issue, and are back to the original hypothesis that scala-compiler shows up as a transitive dependency, and is used to build instead of the scala-compiler requested by scalaVersion.

[harrah]

It is unlikely that would happen in sbt 0.13.0 because scalaVersion is used for the "scala-tool" configuration, which is used as the classpath to run the compiler. A transitive dependency in "compile" won't affect "scala-tool" because they are independent. It will determine the classpath passed to scalac, however.

How can I reproduce this issue to try to debug it?

[xeno-by]

@harrah 1) Clone scala/scala, change it to crash in some obvious place, compile.
2) Get my script from https://github.com/xeno-by/dotx/blob/bc7624209c245b684937f8048cee8728d0230227/integrate and adjust the first couple lines to point to the root of the clone you just compiled and to a clone of scala/scala-partest.
3) Run the script and observe the crash.
4) Uncomment the second-to-last line of the script and comment the last line.
5) Run the script and observe no crash.

[harrah]

I spent some time trying to make this work, but it needs to be trivial for me to reproduce. For example, how do I make it crash but still pass doc.

[xeno-by]

@harrah You can also try inserting a printline in one of the popular typer's methods. I went that route.

[gkossakowski]

@retronym: sorry for delayed response. I've been behind some of my e-mail.

scala-partest is bringing in a transitive dependency on scala-compiler 2.11.0-M4 via scalap. SBT seems to be using that one in preference to the on in scalaInstance; I repeated your tests with adding ??? to Global and witnessing that the downstream build of scala-partest did not throw the expected exception.

It seems the scalaHome facility picks the right compiler.

But I'm really not sure about all of the forces in play. Need some advice from @adriaanm @gkossakowski

Do I understand correctly that you are saying that libraryDependency on a compiler is used for instantiating the compiler that sbt uses to compile scala-partest? If so, then I cant reproduce it.

I tried:

$ cat build.sbt 
version := scalaVersion.value

crossVersion := CrossVersion.Disabled

name := "scala-compiler"

organization := "fake"

scalaVersion := "2.11.0-M5"

and:

cat Global.scala 
package scala.tools.nsc

class Global {
    sys.error("fake global")
}

I added this as dependency to scala-partest:

git diff
diff --git a/build.sbt b/build.sbt
index d74f48e..8daddf9 100644
--- a/build.sbt
+++ b/build.sbt
@@ -25,6 +25,8 @@ libraryDependencies += "org.scala-lang.modules"        %% "scala-xml"      % sca

 libraryDependencies += "org.scala-lang"                 % "scalap"         % scalaVersion.value

+libraryDependencies += "fake" % "scala-compiler" % scalaVersion.value
+

and I tried to compile scala-partest. It worked just fine. Any ideas?

What filtering makes sense for the transitive dependencies that we use when building scala-partest?

I believe none is needed. We should just mark all dependencies as provided. That won't help the problem we are discussing here: wrong scala-compiler classes being used. I believe this problem is not really related to transitive dependencies. It just happens to be triggered by them.

What filtering/rewriting should we perform when publishing the POM of scala-partest.

If we mark all partest dependencies as provided I think we don't need to do anything special.

[xeno-by]

@ALL Ping.

[gkossakowski]

@xeno-by: I believe our current conclusion is that we don't know what's going on but it's not an issue with SNAPSHOT dependencies.

@retronym: does your work-around fixes reliably the problem with wrong compiler classes being used?