Jardiff for local bootstrap and artifact on scala-integration

[lrytz]

➜  scala git:(2.12.x) git log --oneline -n 1 | cat
38628d11fc Merge pull request #6077 from retronym/topic/unbreak-windows

Ivy cache for that version is empty

➜  scala git:(2.12.x) find ~/.ivy2 -path '*38628d1*'
➜  scala git:(2.12.x)

Fetch bootstrapped version from https://scala-ci.typesafe.com/artifactory/scala-integration/ (built by https://scala-ci.typesafe.com/view/scala-bench/job/bootstrap-benchmark/321/consoleFull)

➜  scala git:(2.12.x) sbt -Dstarr.version=2.12.4-bin-38628d1 update
➜  scala git:(2.12.x) find ~/.ivy2 -path '*38628d1*.jar'
/Users/luc/.ivy2/cache/org.scala-lang/scala-compiler/jars/scala-compiler-2.12.4-bin-38628d1.jar
/Users/luc/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.4-bin-38628d1.jar
/Users/luc/.ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.12.4-bin-38628d1.jar

Just to make sure that the inliner worked when compiling this version (getOrElse was inlined into orNull)

➜  scala git:(2.12.x) unzip /Users/luc/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.4-bin-38628d1.jar scala/Option.class -d sandbox
➜  scala git:(2.12.x) cfr-decompiler sandbox/scala/Option.class
...
    public final <A1> A1 orNull(Predef..less.colon.less<Null., A1> ev) {
        if (this.isEmpty()) {
            return (A1)ev.apply((Object)null);
        }
        return (A1)this.get();
    }
...

Local bootstrap

➜  scala git:(2.12.x) sbt -Dstarr.version=2.12.4-bin-38628d1
> setupBootstrapQuick "no-publish-repo" "2.12.4-bin-38628d1-local"
> version
[info] library/*:version
[info] 	2.12.4-bin-38628d1-local

> show scalacOptions
[info] library/compile:scalacOptions
[info] 	List(-opt:l:inline, -opt-inline-from:scala/**, -sourcepath, /Users/luc/scala/scala/src/library)

> publishLocal

[info] 	published scala-library to /Users/luc/.ivy2/local/org.scala-lang/scala-library/2.12.4-bin-38628d1-local/jars/scala-library.jar
...
[info] 	published scala-reflect to /Users/luc/.ivy2/local/org.scala-lang/scala-reflect/2.12.4-bin-38628d1-local/jars/scala-reflect.jar
...
[info] 	published scala-compiler to /Users/luc/.ivy2/local/org.scala-lang/scala-compiler/2.12.4-bin-38628d1-local/jars/scala-compiler.jar

Check if the inliner worked

➜  scala git:(2.12.x) rm -rf sandbox/scala
➜  scala git:(2.12.x) unzip /Users/luc/.ivy2/local/org.scala-lang/scala-library/2.12.4-bin-38628d1-local/jars/scala-library.jar  scala/Option.class -d sandbox
➜  scala git:(2.12.x) cfr-decompiler sandbox/scala/Option.class
    public final <A1> A1 orNull(Predef..less.colon.less<Null., A1> ev) {
        if (this.isEmpty()) {
            return (A1)ev.apply((Object)null);
        }
        return (A1)this.get();
    }

Jardiff library

➜  scala git:(2.12.x) find ~/.ivy2 -path '*38628d1*.jar'
/Users/luc/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.4-bin-38628d1.jar
...
/Users/luc/.ivy2/local/org.scala-lang/scala-library/2.12.4-bin-38628d1-local/jars/scala-library.jar

➜  scala git:(2.12.x) jardiff /Users/luc/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.12.4-bin-38628d1.jar /Users/luc/.ivy2/local/org.scala-lang/scala-library/2.12.4-bin-38628d1-local/jars/scala-library.jar

https://gist.github.com/lrytz/2d582b3cc03f4d8bfed9800658c35844

[lrytz]

So one example is Option.WithFilter.map. The version compiled on jenkins by the bootstrap script has only map inlined, but invokes filter. The other version (and also when compiling Option.scala separately with the inliner enabled) has both filter and map inlined.

[lrytz]

Started a build with -opt-warnings:_ (https://github.com/scala/scala/compare/2.12.x...lrytz:log-inline?expand=1) here https://scala-ci.typesafe.com/job/bootstrap-benchmark/322/parameters/

[lrytz]

Another example I looked at is in scala/util/matching/Regex.class, there is a number of

-    INVOKESTATIC scala/util/matching/Regex.$anonfun$unapplySeq$1$adapted (Ljava/util/regex/Matcher;Ljava/lang/Object;)Ljava/lang/String;
+    INVOKESTATIC scala/runtime/BoxesRunTime.unboxToInt (Ljava/lang/Object;)I
+    INVOKESTATIC scala/util/matching/Regex.$anonfun$unapplySeq$1 (Ljava/util/regex/Matcher;I)Ljava/lang/String;

Again, when compiled on jenkins the method $anonfun$unapplySeq$1$adapted was not inlined, while it is when compiled locally.

[lrytz]

This reminds me very much of the situation in #424, where inlining fails because of mixed compilation and there's no bytecode available for Java-defined classes. But why is it different when compiling in sbt locally? Need to figure that out.

[lrytz]

Indeed:

[warn] /home/jenkins/workspace/bootstrap-benchmark/src/library/scala/Option.scala:207: scala/Option::filter(Lscala/Function1;)Lscala/Option; is annotated @inline but could not be inlined:
[warn] Failed to check if scala/Option::filter(Lscala/Function1;)Lscala/Option; can be safely inlined to scala/Option$WithFilter without causing an IllegalAccessError. Checking instruction INVOKESTATIC scala/runtime/BoxesRunTime.unboxToBoolean (Ljava/lang/Object;)Z failed:
[warn] The method unboxToBoolean(Ljava/lang/Object;)Z could not be found in the class scala/runtime/BoxesRunTime or any of its parents.
[warn] Note that class scala/runtime/BoxesRunTime is defined in a Java source (mixed compilation), no bytecode is available.
[warn]     def map[B](f: A => B): Option[B] = self filter p map f
[warn]                                             ^

[lrytz]

Now I'm getting very curious what will happen to compiler performance if we go from mixed compilation to JavaThenScala

[lrytz]

Aha, mystery solved. The reason inlining worked on my machine is that I didn't run sbt clean before, so the classfiles corresponding to Java sources were still in the output directory, which is on the classpath.

After a clean, the classfiles of a local bootstrap are identical.

Follow-up:

• Disable mixed compilation
• Add some integration test that ensures inlining in the standard library happened as expected

[jvican]

Hey Lukas, I assume you want to disable mixed compilation only for the Scala library. Is that correct?

[lrytz]

Currently we only have mixed compilation enabled for the library, the others are JavaThenScala

[jvican]

Alright, I was asking because my statistics PR enabled Mixed for Reflect, but I guess that with the pending change I have to make that will go away.

[dwijnand]

@lrytz what's the desired outcome for this, in your eyes? A local command/script? A nightly CI process? Just docs somewhere?