JDK11 Segmentation Error vmState=0x0002000f

[connglli]

Java version

openjdk version "11.0.19-internal" 2023-04-18
OpenJDK Runtime Environment (build 11.0.19-internal+0-adhoc..openj9-openjdk-jdk11)
Eclipse OpenJ9 VM (build master-f14707f, JRE 11 Linux amd64-64-Bit Compressed References 20230303_000000 (JIT enabled, AOT enabled)
OpenJ9   - f14707f
OMR      - 342b647
JCL      - a8cc681 based on jdk-11.0.19+4)

Javac version

javac 11.0.19-internal

Code and summary of the problem

The following test makes OpenJ9 to crash in GC. This should be a JIT bug since there aren't any crash if -Xint is enabled.

import java.io.IOException;
import java.io.OutputStream;
import java.io.PrintStream;

public class T {
  public void foo(String[] g) {}

  public static void main(String[] g) {
    T t = new T();
    int[] b = new int[8];
    for (int q = -787; ; ) {
      for (int k = 0; k < 10000; k++) {}
      C.z();
      t.foo(new String[] {});
      try {
        for (int y = 0; ; y++)
          for (int x = 0; x <= 3; x++) b[y * x] += 1;
      } catch (Throwable ax$18) {
      } finally {
      }
    }
  }

  public static final class C {
    final PrintStream q =
        new PrintStream(
            new OutputStream() {
              @Override
              public void write(int i) throws IOException {}
            });
    private static final PrintStream o = System.out;

    public static void z() {
      System.setOut(C.o);
    }
  }
}

Segmentation error:

Unhandled exception
Type=Segmentation error vmState=0x0002000f
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000002
Handler1=00007FAE97439DE0 Handler2=00007FAE96C3B1F0 InaccessibleAddress=000000000B000A1A
RDI=00007FAE9003BD38 RSI=00007FAE30001D08 RAX=00007FAE90015788 RBX=00007FAE900474E0
RCX=00007FAE4A607778 RDX=00007FAE4A607950 R8=00007FAE4A607780 R9=00007FAE4A607788
R10=000000000000001C R11=00000000FFF7CC36 R12=000000000B000A00 R13=00007FAE4A607950
R14=00007FAE4A607778 R15=00007FAE9003BCA0
RIP=00007FAE94A239E4 GS=0000 FS=0000 RSP=00007FAE4A6076D0
EFlags=0000000000010206 CS=0033 RBP=020000000B000A00 ERR=0000000000000004
TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=000000000B000A1A
xmm0 000000000000022f (f: 559.000000, d: 2.761827e-321)
xmm1 00650072002e0067 (f: 3014759.000000, d: 9.346084e-307)
xmm2 000000000000022e (f: 558.000000, d: 2.756886e-321)
xmm3 006600650072002e (f: 7471150.000000, d: 9.791011e-307)
xmm4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm5 fff7d0b000000000 (f: 0.000000, d: -nan)
xmm6 fff7d15800000000 (f: 0.000000, d: -nan)
xmm7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/home/simon/JVMs/openj9/openj9-openjdk-jdk11/build/linux-x86_64-normal-server-release/images/jdk/lib/default/libj9gc29.so
Module_base_address=00007FAE94870000
Target=2_90_20230303_000000 (Linux 5.15.0-56-generic)
CPU=amd64 (8 logical CPUs) (0x7cda7d000 RAM)
----------- Stack Backtrace -----------
_ZN22GC_ObjectModelDelegate29calculateObjectDetailsForCopyEP18MM_EnvironmentBaseP18MM_ForwardedHeaderPmS4_S4_+0x34 (0x00007FAE94A239E4 [libj9gc29.so+0x1b39e4])
_ZN12MM_Scavenger14copyForVariantILb0EEEP8J9ObjectP22MM_EnvironmentStandardP18MM_ForwardedHeader+0x61 (0x00007FAE949FD5E1 [libj9gc29.so+0x18d5e1])
_ZN12MM_Scavenger26incrementalScanCacheBySlotEP22MM_EnvironmentStandardP24MM_CopyScanCacheStandard+0x78d (0x00007FAE949FA72D [libj9gc29.so+0x18a72d])
_ZN12MM_Scavenger12completeScanEP22MM_EnvironmentStandard+0x76 (0x00007FAE949FAEE6 [libj9gc29.so+0x18aee6])
_ZN12MM_Scavenger24workThreadGarbageCollectEP22MM_EnvironmentStandard+0x41c (0x00007FAE949FB57C [libj9gc29.so+0x18b57c])
_ZN21MM_ParallelDispatcher16workerEntryPointEP18MM_EnvironmentBase+0x1f7 (0x00007FAE949A5797 [libj9gc29.so+0x135797])
_Z23dispatcher_thread_proc2P14OMRPortLibraryPv+0x109 (0x00007FAE949A4FC9 [libj9gc29.so+0x134fc9])
omrsig_protect+0x1e3 (0x00007FAE96C3BF53 [libj9prt29.so+0x29f53])
dispatcher_thread_proc+0x3f (0x00007FAE949A414F [libj9gc29.so+0x13414f])
thread_wrapper+0x162 (0x00007FAE971ED322 [libj9thr29.so+0xe322])
start_thread+0xd9 (0x00007FAE97EB6609 [libpthread.so.0+0x8609])
clone+0x43 (0x00007FAE98012133 [libc.so.6+0x11f133])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2023/03/28 19:11:07 - please wait.
JVMDUMP032I JVM requested System dump using '/home/simon/Desktop/test-jitcomp/ax-eval/bugs2/10.openj9/mutant/red/red/red/ttt/core.20230328.191107.2199507.0001.dmp' in response to an event
JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.2199532.  Review the manual for the external program to find where the core dump is written and ensure the program does not truncate it.

JVMPORT049I The core file created by child process with pid = 2199532 was not found. Review the documentation for the /proc/sys/kernel/core_pattern program "|/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" to find where the core file is written and ensure that program does not truncate it.

JVMDUMP012E Error in System dump: /home/simon/Desktop/test-jitcomp/ax-eval/bugs2/10.openj9/mutant/red/red/red/ttt/core.20230328.191107.2199507.0001.dmp
JVMDUMP032I JVM requested Java dump using '/home/simon/Desktop/test-jitcomp/ax-eval/bugs2/10.openj9/mutant/red/red/red/ttt/javacore.20230328.191107.2199507.0002.txt' in response to an event
JVMDUMP012E Error in Java dump: /home/simon/Desktop/test-jitcomp/ax-eval/bugs2/10.openj9/mutant/red/red/red/ttt/javacore.20230328.191107.2199507.0002.txt
JVMDUMP032I JVM requested Snap dump using '/home/simon/Desktop/test-jitcomp/ax-eval/bugs2/10.openj9/mutant/red/red/red/ttt/Snap.20230328.191107.2199507.0003.trc' in response to an event
JVMDUMP010I Snap dump written to /home/simon/Desktop/test-jitcomp/ax-eval/bugs2/10.openj9/mutant/red/red/red/ttt/Snap.20230328.191107.2199507.0003.trc
JVMDUMP032I JVM requested JIT dump using '/home/simon/Desktop/test-jitcomp/ax-eval/bugs2/10.openj9/mutant/red/red/red/ttt/jitdump.20230328.191107.2199507.0004.dmp' in response to an event
JVMDUMP051I JIT dump occurred in 'GC Worker' thread 0x0000000000157900
JVMDUMP010I JIT dump written to /home/simon/Desktop/test-jitcomp/ax-eval/bugs2/10.openj9/mutant/red/red/red/ttt/jitdump.20230328.191107.2199507.0004.dmp
JVMDUMP013I Processed dump event "gpf", detail "".

Diagnostic files

In my settings, the bug can be reproduced deterministically. See the logs (javacore, jitdump, Snap) in issue17045.tar.gz.

[connglli]

@pshipton By the way, I still have arount 12 tests that can cause GC to crash, which should be induced by JIT. However, all of them are non-deterministic. So it's hard for me to reduce them (and check their uniqueness preliminarily). Even though OpenJ9 crashes evey time you run them, they crash with a different symptom (different types like segfault, assertion failures, and different stacktraces).

Importantly, each test (of the 12 tests) can behave with at least 1 symptom that are different from all bugs that I've reported. I wanted to display to you the symptom that are unique, but I'm afraid to smash your issue tracker because they are non-deterministic. So which of the following you suggest me to do

1. Put them all in a single issue.
2. Put one test per-issue (I'll attach the logs of the unique symptom and other duplicate symptoms).

Thank you.

[pshipton]

@connglli I'll leave this to the JIT team since they tend to be JIT issues.

@0xdaryl @hzongaro FYI this new issue. Can you pls answer the question about how you would like the other 12.

[0xdaryl]

I'd say it is highly unlikely that we have an additional 12 unique issues with JIT/GC. Let's start with a single issue for the most reproducible problem you have and include the code for the others in that same issue. If they turn out to be unique problems we can create issues for them as we go. Thanks.

[connglli]

Sure @0xdaryl. As I said, I don't know the uniqueness, either. So let's start like that. Looks like a good way!!! (It's night at my time. I'll do it tomorrow.)

[dmitripivkine]

This kind of failure (stack) is very common in the case object heap is corrupted or there is incorrect (stale?) pointer to the object from root or another object. This is failure from the same group of assertions mentioned in #17052. BTW there are assertions in that list can be triggered by object heap corruption only, not an invalid object reference (failures in Memory Pool for instance)

[hzongaro]

No one has had the opportunity to investigate further yet. Moving this to 0.43 release.

[hzongaro]

Brad @BradleyWood , may I ask you to take a look at this problem? I think it's related to Dynamic Loop Transfer. It runs successfully (indefinitely) with either:

java -Xjit:limit={T.main*},optLevel=cold,verbose T
java -Xjit:limit={T.main*},optLevel=warm,verbose T

but crashes with either

java -Xjit:limit={T.main*},optLevel=hot,verbose T
java -Xjit:limit={T.main*},optLevel=scorching,verbose T

[hzongaro]

@BradleyWood, may I ask you to check whether this problem was fixed by either or both the OMR pull requests eclipse-omr/omr#7237 and eclipse-omr/omr#7238?

[BradleyWood]

@hzongaro Its fixed