provide guidance to project maintainers on including “latest Scala nightly” in their CI matrices

[SethTisue]

we should figure out how to do this and provide a sample .travis.yml that does it. the Scala modules could be a good place to start

[SethTisue]

https://stackoverflow.com/questions/40622878/how-do-i-tell-sbt-to-use-a-nightly-build-of-scala-2-11-2-12-or-2-13 shows how to get the latest nightly in sbt, but doing it in .travis.yml is a different matter

[dwijnand]

I had a little look at how to do this. So I looked for the simplest project I could find that use Travis CI and picked https://github.com/lihaoyi/sourcecode

However it depends on sbt-scalajs, which depends on a fully cross-built version of scalajs-compiler... See scala-js/scala-js#3140 for my conversation with the Scala.js project about this issue.

I expect Scala Native to be similarly difficult (if not worst...).

[sjrd]

In general, for compiler plugins, the only safe(-ish) way to set it up is to clone the source of the relevant compiler plugin and compile/publish it locally, built againt the particular Scala nightly.

This is necessary because scala-compiler.jar does not guarantee any kind of binary compatibility. In fact, it does not even provide any kind of source compat', so even this can fail, but in that case you want to know anyway.

You could try using using the published version of the compiler plugin that is built against the most recent Scala release, but you are open to binary incompatibilities.

[dwijnand]

Using Scala 2.12.4-bin-38628d1 with scalajs-compiler_2.12.3 😢

java.lang.NoSuchMethodError: scala.tools.nsc.backend.jvm.BTypesFromSymbols.initializeCoreBTypes()V
	at org.scalajs.core.compiler.Compat210Component.initializeCoreBTypesCompat(Compat210Component.scala:146)
	at org.scalajs.core.compiler.Compat210Component.initializeCoreBTypesCompat$(Compat210Component.scala:130)
	at org.scalajs.core.compiler.GenJSCode.initializeCoreBTypesCompat(GenJSCode.scala:31)
	at org.scalajs.core.compiler.GenJSCode$JSCodePhase.run(GenJSCode.scala:215)
	at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1432)
	at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1417)
	at scala.tools.nsc.Global$Run.compileSources(Global.scala:1413)
	at scala.tools.nsc.Global$Run.compile(Global.scala:1516)
	at xsbt.CachedCompiler0.run(CompilerInterface.scala:116)
	at xsbt.CachedCompiler0.run(CompilerInterface.scala:95)
	at xsbt.CompilerInterface.run(CompilerInterface.scala:26)

[dwijnand]

My code snippet:

// When using a nightly build of Scala (i.e when scalaVersion has a "-bin-.." suffix)
// Replace the scalajs-compiler compiler plugin injected by sbt-scalajs
// With the one built for the _previous_ version of Scala
//
// Example:
// When building with Scala 2.12.4-bin-38628d1, which is a nightly build for future Scala 2.12.4
// Replace scalajs-compiler_2.12.4-bin-38628d1 (which doesn't exist)
// with scalajs-compiler_2.12.3
libraryDependencies := {
  /** If scalaVersion has a "-bin-.." suffix, return the previous version.
   *
   *  | scalaVersion       | crossSv        | baseSv         | prevSv         |
   *  |--------------------+----------------+----------------+----------------|
   *  | 2.12.4             | Some("2.12.4") | None           | None           |
   *  | 2.12.4-bin-38628d1 | Some("2.12.4") | Some("2.12.4") | Some("2.12.3") |
   */
  def prevScalaVersion(scalaVersion: String, scalaBinaryVersion: String): Option[String] = {
    val crossVersion = CrossVersion(CrossVersion.patch, scalaVersion, scalaBinaryVersion)
    val crossSv = crossVersion map (fn => fn("") stripPrefix "_")
    val baseSv = crossSv filter (_ != scalaVersion)
    val prevSv = baseSv collect {
      case VersionNumber(Seq(x, y, z), Seq(), Seq()) if z > 0 => s"$x.$y.${z - 1}"
    }
    prevSv
  }

  val prevSv = prevScalaVersion(scalaVersion.value, scalaBinaryVersion.value)

  val libDeps = libraryDependencies.value
  prevSv match {
    case None         => libDeps
    case Some(prevSv) =>
      val scalaJsCompiler =
        compilerPlugin("org.scala-js" % s"scalajs-compiler_$prevSv" % scalaJSVersion)
      libDeps map (m => if (m.name == "scalajs-compiler") scalaJsCompiler else m)
  }
}

[sjrd]

Well, guess what? You found an actual binary incompatibility between 2.12.3 and 2.12.4-bin-whatever, introduced by scala/scala@5f5d525#diff-96d929f42cdd21128555c2ea328fbf95

Not sure how the community build passes, btw, because it's also not source compatible. Apparently calling that initialize() method is not as critical as was calling initializeCoreBTypes(). @lrytz, any clue?

See also:

• Commit where we start calling that method: scala-js/scala-js@58d9c45
• Issue that commit fixed: Community build broken: need to call bTypes.initializeCoreBTypes() scala-js/scala-js#2027
• The original issue in scala-dev: 2.12 community build: regression in scala-js #64

[lrytz]

Maybe I misunderstand, but I'm surprised this hasn't come up before: the scala-compiler jar is not binary compatible across minor versions.

[sjrd]

The new thing here is that @dwijnand is putting himself in harm's way from bin incompatibility. The community build always builds from source, so only source incompatibilities are a problem. But @dwijnand is playing with fire by linking a compiler plugin built for a different version of the compiler. I had warned against this in my first comment in this thread.

(The reason I pinged you is only mildly related to the main topic of the discussion; I was surprised that the renaming of initializeCoreBTypes() did not cause a source incompatibility, which would have angered the community build.)

[lrytz]

OK, I read the rest of this ticket now, so there's no misunderstanding about binary / source compatibility for scala-compiler :)

Indeed, calling initialize is no longer necessary for things to work, give me a minute to explain that in detail.

[lrytz]

So the backend has quite a bit of state that is derived from the symbol table. Since types of symbols might change between runs, this has to be re-computed in each compiler run. This used to be done in the initializeCoreBTypes method (and some others).

Now we have a bit of an abstraction, state is created through perRunLazy, which creates a LazyVar, which caches the value and keeps a reference to the initializer, so it can be re-invoked. The new initialize method now re-initializes all LazyVars. So calling initialize is no longer necessary to get the initial values computed.

If you're running multiple compiler runs, you should probably call GenBCode.initialize in each run (or parts of it).

[sjrd]

Hum. So IIUC, now Scala.js with 2.12.4 is vulnerable to crashes or inconsistent behavior in the PC?

[dwijnand]

I'm just experimenting the viability of people testing Scala nightly builds in their projects. They would be configured with allow_failures anyways.

[dwijnand]

@sjrd Would the Scala.js project be willing to setup nightly releases of Scala.js against Scala nightlies?