GC crash in native executable with graalvm 20.2 and 20.3

[skoptelov]

Describe the issue

I have a Quarkus app that, when compiled to native linux x86_64 and run in docker, crashes in GC. Crash timing is immediately on container start, or a bit later, depending on memory settings (faster crash with lower heap memory). Rolling back to 20.1 solved the issue.

Steps to reproduce the issue
It's a proprietary app, with a bunch of dependencies, so isolating the minimal repro is difficult. Hopefully crash info will provide some insight into what's going on.

Describe GraalVM and your environment:

• GraalVM CE 20.2, 20.3
• JDK major version: 11
• OS: linux
• Architecture: x86_64

More details
Here is the example of crash info in container logs:
https://pastebin.com/THUHuDUd

All crashes follow the same pattern - stack trace points to GC logic (OldGeneration.promoteObject and up), which is called from somewhere in user code (GC is triggered from different location every crash).

Here is the command line used to make native image:

/home/gitlab-runner/.sdkman/candidates/java/20.3.0.r11-grl/bin/native-image -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-DCoordinatorEnvironmentBean.transactionStatusManagerEnable=false -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=1 -J-Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true -J-Duser.language=en -J-Dfile.encoding=UTF-8 -J-Dfile.encoding=UTF-8 -H:ResourceConfigurationFiles=resources-config.json -H:ReflectionConfigurationFiles=reflection-config.json,reflection-poi.json --report-unsupported-elements-at-runtime --allow-incomplete-classpath --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy\$BySpaceAndTime -H:+JNI -jar equeue-0.2.26-runner.jar -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http,https --enable-all-security-services --no-server -H:-UseServiceLoaderFeature -H:+StackTrace equeue-0.2.26-runner

[munishchouhan]

@skoptelov thanks for reporting the issue
We will check it out and update you

[christianhaeubl]

@skoptelov thanks for reporting this issue. To track down this error, we will need a bit more information. As this is a proprietary app, can you try the following with GraalVM 20.3?

• Specify -ea -esa as extra arguments when building the native image. This embeds assertions into the native image, so it should print a better error message and possibly also a different stack trace when crashing.
• Specify -H:-ImageHeapCardMarking as an extra argument when building the native image. This disables a GC feature that was added in GraalVM 20.2 and might be a workaround for the issue that you are seeing.

[skoptelov]

@christianhaeubl Followed both of your suggestions, here is the result: https://pastebin.com/uRNfLtQ4
Updated native-image command line:

/home/gitlab-runner/.sdkman/candidates/java/20.3.0.r11-grl/bin/native-image -J-DCoordinatorEnvironmentBean.transactionStatusManagerEnable=false -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=1 -J-Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true -J-Duser.language=en -J-Dfile.encoding=UTF-8 -J-Dfile.encoding=UTF-8 -H:ResourceConfigurationFiles=resources-config.json -H:ReflectionConfigurationFiles=reflection-config.json,reflection-poi.json --report-unsupported-elements-at-runtime --allow-incomplete-classpath -ea -esa -H:-ImageHeapCardMarking --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -H:+JNI -jar equeue-0.2.27-1-gb51c159-runner.jar -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http,https --enable-all-security-services --no-server -H:-UseServiceLoaderFeature -H:+StackTrace equeue-0.2.27-1-gb51c159-runner

[christianhaeubl]

@skoptelov thanks for giving that a try. Unfortunately, the new crash log does not provide any new insights. There is one last option that you can try:

• Specify -H:+VerifyHeap as an extra argument when building the native image. This embeds more GC-related verification code into the native image but it will significantly slow down the execution of the native image.

If that doesn't give any more information either, then we probably won't be able to fix that issue without a debuggable reproducer. GraalVM 21.0 contains a few smaller GC-related fixes and will be released next month, so you could give that a try once it is out. In the meanwhile, I will improve the GC-related checks & verifications so that they collect a bit more information upon a crash.

[skoptelov]

@christianhaeubl Added the flag per your suggestion, here's the new output: https://pastebin.com/fLb7ZEGU

If there's anything else I can do on my side, please let me know, e.g. I could try to build with native-image from master after you push GC checks there.

[christianhaeubl]

@skoptelov thanks, the heap verification printed some extra information:

• An object or an object array contains an invalid object reference that has the value 1 (0 would be the null pointer). Unfortunately, the validation does not print the class of the object that contains this invalid reference (we only know that its size is less than 128 kb as it was allocated in the TLAB).
• As the heap verification fails before garbage collection, the application most likely corrupted the heap between garbage collections (i.e., the garbage collector only crashes as a side effect of the corrupted heap). This could still be a native image bug but it could also be caused by the application itself (e.g., via JNI or via sun.misc.Unsafe).

Do you also use a more recent Quarkus version when you use GraalVM 20.3 instead of GraalVM 20.1?

[skoptelov]

@christianhaeubl It seems that you're right, Unsafe was indeed the immediate culprit. Hazelcast and POI dependencies were my main suspects since native-image barked at them at compilation time. So I substituted out some hazelcast logic that uses it: forced SafeSwCounter usage, and removed ICMPHelper JNI usage just in case, and also removed POI Unsafe.invokeCleaner usage, and now the crash is gone (even without -H:-ImageHeapCardMarking).

I could submit a PR to quarkus-hazelcast-client with these substitutions but I'm not sure if that's a proper way to address this crash. Hazelcast on JVM is a stable and widely used software, and even in native-image 20.1 it worked just fine (presumably because 20.2+ added some memory checks that were not there before). That is to say, it seems unlikely that there's a glaring crash-inducing bug in its usage of Unsafe. So is it that Unsafe is not fully supported in native-image? I'm not sure how it's supposed to work - does native-image provide (or aims to provide) full Unsafe implementation, or are we supposed to substitute out its usage (which may or may not be a reasonable ask in any particular case, since people sometimes use Unsafe for good reasons)?

[christianhaeubl]

@skoptelov Native image supports most use cases of Unsafe.

During the image build, the HotSpot JVM is used. Field offsets may get cached during the image build and may end up in the native image. As native image layouts Java objects differently from the HotSpot JVM, we must ensure that the native image offset of a field (and not the HotSpot JVM offset of a field) is used at native image run-time. This works for most use cases of Unsafe but in fairly complex use cases, it can happen that our detection fails and a HotSpot JVM offset is used at native image run-time. This can result in all kinds of problems, such as overwriting arbitrary data.

You mentioned that the image build printed a warning regarding Hazelcast? Can you post that warning?