Regression with Scala 2.13: cannot compile GraalVM native-image

[plokhotnyuk]

An example which successfully compiles to native binaries from Scala 2.11 & 2.12 uber-jars cannot be compiled from Scala 2.13 uber-jar due some internal changes in Scala collection implementation.

Error output from the native-image tool is:

Error: Unsupported features in 5 methods
Detailed message:
Error: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved method during parsing: java.lang.Object[].clone(). To diagnose the issue you can use the --allow-incomplete-classpath option. The missing method is then reported at run time when it is accessed the first time.
Trace: 
	at parsing scala.collection.immutable.VectorPointer.gotoPosWritable1(Vector.scala:1243)
Call path from entry point to scala.collection.immutable.VectorPointer.gotoPosWritable1(int, int, int, Object[]): 
	at scala.collection.immutable.VectorPointer.gotoPosWritable1(Vector.scala:1242)
	at scala.collection.immutable.VectorPointer.gotoPosWritable1$(Vector.scala:1241)
	at scala.collection.immutable.VectorBuilder.addAll(Vector.scala:840)
	at scala.collection.immutable.Vector$.from(Vector.scala:840)
	at scala.collection.immutable.Vector$.from(Vector.scala:27)
	at scala.collection.IterableFactory.apply(Factory.scala:104)
	at scala.collection.IterableFactory.apply$(Factory.scala:104)
	at scala.collection.immutable.List$.apply(List.scala:618)
	at scala.collection.SeqFactory$Delegate.apply(Factory.scala:305)
	at com.github.plokhotnyuk.jsoniter_scala.examples.Example01$.main(Example01.scala:18)
	at com.github.plokhotnyuk.jsoniter_scala.examples.Example01.main(Example01.scala)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:147)
	at com.oracle.svm.core.code.IsolateEnterStub.JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b(generated:0)
Error: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved method during parsing: java.lang.Object[][].clone(). To diagnose the issue you can use the --allow-incomplete-classpath option. The missing method is then reported at run time when it is accessed the first time.
Trace: 
	at parsing scala.collection.immutable.VectorPointer.stabilize(Vector.scala:1197)
Call path from entry point to scala.collection.immutable.VectorPointer.stabilize(int): 
	at scala.collection.immutable.VectorPointer.stabilize(Vector.scala:1163)
	at scala.collection.immutable.VectorPointer.stabilize$(Vector.scala:1163)
	at scala.collection.immutable.VectorBuilder.addAll(Vector.scala:840)
	at scala.collection.immutable.Vector$.from(Vector.scala:840)
	at scala.collection.immutable.Vector$.from(Vector.scala:27)
	at scala.collection.IterableFactory.apply(Factory.scala:104)
	at scala.collection.IterableFactory.apply$(Factory.scala:104)
	at scala.collection.immutable.List$.apply(List.scala:618)
	at scala.collection.SeqFactory$Delegate.apply(Factory.scala:305)
	at com.github.plokhotnyuk.jsoniter_scala.examples.Example01$.main(Example01.scala:18)
	at com.github.plokhotnyuk.jsoniter_scala.examples.Example01.main(Example01.scala)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:147)
	at com.oracle.svm.core.code.IsolateEnterStub.JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b(generated:0)
Error: com.oracle.svm.hosted.substitute.DeletedElementException: Unsupported type java.lang.invoke.MemberName is reachable: All methods from java.lang.invoke should have been replaced during image building.
To diagnose the issue, you can add the option --report-unsupported-elements-at-runtime. The unsupported element is then reported at run time when it is accessed the first time.
Trace: 
	at parsing java.lang.invoke.MethodHandles$Lookup.findVirtual(MethodHandles.java:861)
Call path from entry point to java.lang.invoke.MethodHandles$Lookup.findVirtual(Class, String, MethodType): 
	no path found from entry point to target method

Error: type is not available in this platform: com.oracle.svm.hosted.ClassValueFeature
Trace: 	object java.lang.Class[]
	object java.lang.invoke.MethodType
	object java.lang.invoke.MethodType$ConcurrentWeakInternSet$WeakEntry
	object java.util.concurrent.ConcurrentHashMap$Node
	object java.util.concurrent.ConcurrentHashMap$Node[]
	object java.util.concurrent.ConcurrentHashMap
	object java.lang.invoke.MethodType$ConcurrentWeakInternSet
	method java.lang.invoke.MethodType.makeImpl(Class, Class[], boolean)
Call path from entry point to java.lang.invoke.MethodType.makeImpl(Class, Class[], boolean): 
	no path found from entry point to target method

Error: type is not available in this platform: com.oracle.svm.hosted.substitute.ComputedValueField
Trace: 	object java.lang.invoke.MethodType
	object java.lang.invoke.MethodType$ConcurrentWeakInternSet$WeakEntry
	object java.util.concurrent.ConcurrentHashMap$Node
	object java.util.concurrent.ConcurrentHashMap$Node[]
	object java.util.concurrent.ConcurrentHashMap
	object java.lang.invoke.MethodType$ConcurrentWeakInternSet
	method java.lang.invoke.MethodType.makeImpl(Class, Class[], boolean)
Call path from entry point to java.lang.invoke.MethodType.makeImpl(Class, Class[], boolean): 
	no path found from entry point to target method

Steps to reproduce:

1. Install or make sure all those things are installed: Sbt, GraalVM CE/EE and its native-image tool
2. Clone the following repo: https://github.com/plokhotnyuk/jsoniter-scala
3. Checkout the switch-examples-to-scala-2.13 branch
4. Run the following commands:

cd jsoniter-scala-examples
sbt clean +assembly
/usr/lib/jvm/graalvm-ce-19/bin/native-image --no-server --no-fallback -H:UnsafeAutomaticSubstitutionsLogLevel=3 -jar target/scala-2.13/jsoniter-scala-examples-assembly-0.1.0-SNAPSHOT.jar

[SethTisue]

is this a bug in Scala?

[plokhotnyuk]

IMHO, yes.

Some of these errors appear due to releaseFence calls, as reporred here. Others also could be prevented in case of opening some kind of community build for tools and libs which going to support AOT compilation mode, starting from scalac and core libraries.

[SethTisue]

are you saying that in order for this to work, we can't use releaseFence at all?

[plokhotnyuk]

There are implementation options to make Unsafe usage recognizable for the Graal compiler.

And for edge cases some cooperation with the GraalVM team will be required for sure.

I have managed to fix compilation error for releaseFence calls by the following commit: plokhotnyuk/jsoniter-scala@e089f06

Others calls to java.lang.Object[].clone() and java.lang.Object[][].clone() are still not substituted, and if they are not called in runtime then possible W/A is to use the --allow-incomplete-classpath option of the GraalVM native-image compiler.

[SethTisue]

Tentatively milestoned for 2.13.2, since it should certainly at least be assessed further.

[adriaanm]

Great that you managed to override the call to releaseFence. This (allowing other platforms to customize low level stuff) is why we provide that hook through scala.runtime.Statics. Maybe we can do something similar for the array clone methods?

[sjrd]

What's the problem with Array.clone()? They're not low-level at all; they're straightforward reflection-free user code:
https://github.com/scala/scala/blob/4ec48ff29755fc01fdde880dbb4548085420c601/src/library/scala/runtime/ScalaRunTime.scala#L94-L105

[adriaanm]

I meant java.lang.Object[].clone()

[adriaanm]

Anyway, that still seems a little too common for graal not to support natively? Quick google led to a similar, resolved issue: oracle/graal#877

[plokhotnyuk]

@adriaanm checked with GraalVM CE 19.2.0.1 and still got the same error...

now I'm waiting for the next release: 19.2.1 or 19.3.0 to check again...