JDK21 serviceability/jvmti/vthread/FollowReferences/VThreadStackRefTest.java Segmentation error vmState=0x00040000 #17712

JasonFengJ9 · 2023-06-29T17:35:32Z

Failure link

From an internal build(rhel8x86-rtp-rt2-1):

08:24:53  openjdk version "21-internal" 2023-09-19
08:24:53  OpenJDK Runtime Environment (build 21-internal-adhoc.jenkins.BuildJDK21x86-64linuxPersonal)
08:24:53  Eclipse OpenJ9 VM (build master-163a51495d5, JRE 21 Linux amd64-64-Bit Compressed References 20230624_7 (JIT enabled, AOT enabled)
08:24:53  OpenJ9   - 163a51495d5
08:24:53  OMR      - 59b55e30960
08:24:53  JCL      - e76b3946ed5 based on jdk-21+27)

Rerun in Grinder - Change TARGET to run only the failed test targets.

Optional info

Failure output (captured from console output)

08:27:16  variation: Mode150
08:27:16  JVM_OPTIONS:  -XX:+UseCompressedOops 

08:30:09  TEST: serviceability/jvmti/vthread/FollowReferences/VThreadStackRefTest.java#no-vmcontinuations

08:30:09  STDOUT:
08:30:09  created class VThreadStackRefTest$VThreadUnmountedReferenced
08:30:09  created class VThreadStackRefTest$VThreadUnmountedEnded
08:30:09  created class VThreadStackRefTest$VThreadMountedReferenced
08:30:09  created class VThreadStackRefTest$VThreadMountedJNIReferenced
08:30:09  created class VThreadStackRefTest$PThreadReferenced
08:30:09  threads:
08:30:09    - vthreadUnmounted: VirtualThread[#23]/waiting
08:30:09    - vthreadEnded: VirtualThread[#26]/terminated
08:30:09    - vthreadMounted: VirtualThread[#27]/runnable@ForkJoinPool-1-worker-1
08:30:09    - pthread: Thread[#28,Thread-1,5,MainThreadGroup]
08:30:09  test classes:
08:30:09    (0) class VThreadStackRefTest$VThreadUnmountedReferenced
08:30:09    (1) class VThreadStackRefTest$VThreadUnmountedJNIReferenced
08:30:09    (2) class VThreadStackRefTest$VThreadMountedReferenced
08:30:09    (3) class VThreadStackRefTest$VThreadMountedJNIReferenced
08:30:09    (4) class VThreadStackRefTest$PThreadReferenced
08:30:09    (5) class VThreadStackRefTest$VThreadUnmountedEnded
08:30:09  INFO: No mount/unmount checks
08:30:09  Stack local: index = 2, thread_id = 27
08:30:09  STDERR:
08:30:09  Unhandled exception
08:30:09  Type=Segmentation error vmState=0x00040000
08:30:09  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
08:30:09  Handler1=00007F4C46DB8650 Handler2=00007F4C483DE710 InaccessibleAddress=0000000000000010
08:30:09  RDI=0000000000000000 RSI=000000000000000C RAX=0000000000000000 RBX=00007F4C2441D280
08:30:09  RCX=00007F4BC0047350 RDX=00007F4C2441CDD0 R8=0000000000000030 R9=00007F4C40031080
08:30:09  R10=00007F4C453C25F0 R11=0000000000000001 R12=0000000000000000 R13=0000000000000000
08:30:09  R14=00007F4C2441C1E0 R15=00007F4C40031080
08:30:09  RIP=00007F4C453C33BA GS=0000 FS=0000 RSP=00007F4C2441C1B0
08:30:09  EFlags=0000000000010246 CS=0033 RBP=0000000000245D00 ERR=0000000000000004
08:30:09  TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=0000000000000010
08:30:09  xmm0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:30:09  xmm1 00000000ffef43c0 (f: 4293870592.000000, d: 2.121454e-314)
08:30:09  xmm2 00000000002533d9 (f: 2438105.000000, d: 1.204584e-317)
08:30:09  xmm3 00000000000b53d8 (f: 742360.000000, d: 3.667746e-318)
08:30:09  xmm4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:30:09  xmm5 0000003000000020 (f: 32.000000, d: 1.018558e-312)
08:30:09  xmm6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:30:09  xmm7 0000003000000020 (f: 32.000000, d: 1.018558e-312)
08:30:09  xmm8 0a6425203d206469 (f: 1025533056.000000, d: 1.310212e-258)
08:30:09  xmm9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:30:09  xmm10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:30:09  xmm11 0000015200000151 (f: 337.000000, d: 7.172346e-312)
08:30:09  xmm12 0000013d00000140 (f: 320.000000, d: 6.726727e-312)
08:30:09  xmm13 000001380000013f (f: 319.000000, d: 6.620627e-312)
08:30:09  xmm14 0000000008001800 (f: 134223872.000000, d: 6.631540e-316)
08:30:09  xmm15 000001420000013b (f: 315.000000, d: 6.832826e-312)
08:30:09  Module=/home/jenkins/workspace/Test_openjdk21_j9_extended.openjdk_x86-64_linux_Personal/openjdkbinary/j2sdk-image/lib/default/libj9jvmti29.so
08:30:09  Module_base_address=00007F4C453AC000
08:30:09  Target=2_90_20230624_7 (Linux 4.18.0-425.19.2.el8_7.x86_64)
08:30:09  CPU=amd64 (4 logical CPUs) (0x1e5342000 RAM)
08:30:09  ----------- Stack Backtrace -----------
08:30:09  followReferencesCallback+0xdca (0x00007F4C453C33BA [libj9jvmti29.so+0x173ba])
08:30:09  _ZN23MM_ReferenceChainWalker6doSlotEPP8J9ObjectllS1_.localalias+0x96 (0x00007F4C4559D246 [libj9gc29.so+0x38246])
08:30:09  walkStackFrames+0x11d0 (0x00007F4C46DFC020 [libj9vm29.so+0x87020])
08:30:09  walkContinuationStackFrames.part.0+0x56 (0x00007F4C46E137D6 [libj9vm29.so+0x9e7d6])
08:30:09  _ZN28GC_VMThreadStackSlotIterator9scanSlotsEP10J9VMThreadP16J9VMContinuationPvPFvP8J9JavaVMPP8J9ObjectS4_P16J9StackWalkStatePKvEbb+0x3e (0x00007F4C455A9CBE [libj9gc29.so+0x44cbe])
08:30:09  _ZN14MM_RootScanner13scanOneThreadEP18MM_EnvironmentBaseP10J9VMThreadPv+0x13a (0x00007F4C455A108A [libj9gc29.so+0x3c08a])
08:30:09  _ZN14MM_RootScanner11scanThreadsEP18MM_EnvironmentBase+0xcf (0x00007F4C4559F94F [libj9gc29.so+0x3a94f])
08:30:09  _ZN14MM_RootScanner12scanAllSlotsEP18MM_EnvironmentBase+0x3a (0x00007F4C455A30CA [libj9gc29.so+0x3e0ca])
08:30:09  j9gc_ext_reachable_objects_do+0x1b9 (0x00007F4C4559F009 [libj9gc29.so+0x3a009])
08:30:09  jvmtiFollowReferences+0x2ab (0x00007F4C453C3F1B [libj9jvmti29.so+0x17f1b])
08:30:09  Java_VThreadStackRefTest_test+0x163 (0x00007F4C4402A723 [libVThreadStackRefTest.so+0x3723])
08:30:09  ffi_call_unix64+0x52 (0x00007F4C46F88DAA [libj9vm29.so+0x213daa])
08:30:09  ffi_call_int+0x1a1 (0x00007F4C46F87F41 [libj9vm29.so+0x212f41])
08:30:09  _ZN32VM_BytecodeInterpreterCompressed3runEP10J9VMThread+0x11925 (0x00007F4C46E25635 [libj9vm29.so+0xb0635])
08:30:09  bytecodeLoopCompressed+0xca (0x00007F4C46E13CFA [libj9vm29.so+0x9ecfa])
08:30:09   (0x00007F4C46F16632 [libj9vm29.so+0x1a1632])
08:30:09  ---------------------------------------
08:30:09  JVMDUMP039I Processing dump event "gpf", detail "" at 2023/06/24 05:30:06 - please wait.

08:30:44  Test results: passed: 118; failed: 7
08:30:53  Report written to /home/jenkins/workspace/Test_openjdk21_j9_extended.openjdk_x86-64_linux_Personal/jvmtest/openjdk/report/html/report.html
08:30:53  Results written to /home/jenkins/workspace/Test_openjdk21_j9_extended.openjdk_x86-64_linux_Personal/aqa-tests/TKG/output_16876095876025/serviceability_jvmti_j9_0/work
08:30:53  Error: Some tests failed or other problems occurred.
08:30:53  -----------------------------------
08:30:53  serviceability_jvmti_j9_0_FAILED

Created from

JDK21 serviceability_jvmti_j9 a few failures #17671

FYI @babsingh

The text was updated successfully, but these errors were encountered:

Uber issue: eclipse-openj9/openj9#17671 Issues for individual failures: - eclipse-openj9/openj9#17711 - eclipse-openj9/openj9#17712 - eclipse-openj9/openj9#17713 - eclipse-openj9/openj9#17715 - eclipse-openj9/openj9#17716 - eclipse-openj9/openj9#17717 - eclipse-openj9/openj9#17718 Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>

babsingh · 2023-07-19T17:43:06Z

jvmtiFollowReferences relies upon j9gc_ext_reachable_objects_do, which receives followReferencesCallback as an input parameter.

There is a crash in followReferencesCallback:

openj9/runtime/jvmti/jvmtiHeap.c

Line 891 in a2c8e15

    
           threadID = J9VMJAVALANGTHREAD_TID(iteratorData->currentThread, walkState->walkThread->threadObject);

In the above code, walkState->walkThread->threadObject and walkState->walkThread->carrierThreadObject are NULL, probably, because a stack allocated J9VMThread is created using the fields from a J9VMContinuation. Even after disabling the above line of code, other issues are seen in tracking references and keeping their count.

Test code:

@amicic @dmitripivkine @LinHu2016 Would you have guidelines on how followReferencesCallback and the associated code should be updated to handle virtual threads and continuations?

[GDB] Stacktrace during the crash

...
#12 <signal handler called>
#13 0x00007f7a97f1a493 in jvmtiHeapFollowRefs_getStackData (walkState=<optimized out>, iteratorData=0x7f7a7c2530e0) at /root/openj9-openjdk-jdk21/openj9/runtime/jvmti/jvmtiHeap.c:891
#14 wrap_heapReferenceCallback (vm=0x7f7a98015ea0, iteratorData=0x7f7a7c2530e0) at /root/openj9-openjdk-jdk21/openj9/runtime/jvmti/jvmtiHeap.c:759
#15 followReferencesCallback (slotPtr=<optimized out>, referrer=<optimized out>, userData=0x7f7a7c2530e0, type=<optimized out>, referrerIndex=<optimized out>, wasReportedBefore=0)
    at /root/openj9-openjdk-jdk21/openj9/runtime/jvmti/jvmtiHeap.c:635
#16 0x00007f7a9c136d4a in MM_ReferenceChainWalker::doSlot (this=0x7f7a7c252fc0, slotPtr=<optimized out>, type=<optimized out>, index=<optimized out>, sourceObj=<optimized out>)
    at /root/openj9-openjdk-jdk21/openj9/runtime/gc_base/ReferenceChainWalker.cpp:203
#17 0x00007f7a9d52e560 in walkDescribedPushes (argCount=<optimized out>, descriptionSlots=0x7f7a7c2520f4, slotCount=1, highestSlot=0x2674f0, walkState=0x7f7a7c252c40)
    at /root/openj9-openjdk-jdk21/openj9/runtime/vm/swalk.c:684
#18 walkMethodFrame (walkState=0x7f7a7c252c40) at /root/openj9-openjdk-jdk21/openj9/runtime/vm/swalk.c:769
#19 walkStackFrames (currentThread=<optimized out>, walkState=0x7f7a7c252c40) at /root/openj9-openjdk-jdk21/openj9/runtime/vm/swalk.c:331
#20 0x00007f7a9d546216 in walkContinuationStackFrames (currentThread=0x1dc400, continuation=<optimized out>, walkState=0x7f7a7c252c40) at /root/openj9-openjdk-jdk21/openj9/runtime/vm/ContinuationHelpers.cpp:472
#21 0x00007f7a9c143dc2 in GC_VMThreadStackSlotIterator::scanSlots (vmThread=vmThread@entry=0x1dc400, continuation=0x7f7a0800c230, userData=userData@entry=0x7f7a7c252f60,
    oSlotIterator=oSlotIterator@entry=0x7f7a9c13b000 <stackSlotIterator(J9JavaVM*, J9Object**, void*, J9StackWalkState*, void const*)>, includeStackFrameClassReferences=<optimized out>,
    trackVisibleFrameDepth=<optimized out>) at /root/openj9-openjdk-jdk21/openj9/runtime/gc_structs/VMThreadStackSlotIterator.cpp:165
#22 0x00007f7a9c13afb2 in MM_RootScanner::scanOneThread (this=0x7f7a7c252fc0, env=0x7f7a98c1af58, walkThread=0x267900, localData=0x7f7a7c252f60) at /root/openj9-openjdk-jdk21/openj9/runtime/gc_base/RootScanner.cpp:549
#23 0x00007f7a9c13980f in MM_RootScanner::scanThreads (this=0x7f7a7c252fc0, env=0x7f7a98c1af58) at /root/openj9-openjdk-jdk21/openj9/runtime/gc_base/RootScanner.cpp:508
#24 0x00007f7a9c13d05d in MM_RootScanner::scanAllSlots (this=0x7f7a7c252fc0, env=0x7f7a98c1af58) at /root/openj9-openjdk-jdk21/openj9/runtime/gc_base/RootScanner.cpp:1085
#25 0x00007f7a9c138bb9 in MM_ReferenceChainWalker::scanReachableObjects (env=0x7f7a98c1af58, this=0x7f7a7c252fc0) at /root/openj9-openjdk-jdk21/openj9/runtime/gc_base/ReferenceChainWalker.hpp:250
#26 j9gc_ext_reachable_objects_do (vmThread=<optimized out>, userCallback=0x7f7a97f19890 <followReferencesCallback>, userData=<optimized out>, walkFlags=<optimized out>)
    at /root/openj9-openjdk-jdk21/openj9/runtime/gc_base/ReferenceChainWalker.cpp:91
#27 0x00007f7a97f1b1fb in jvmtiFollowReferences (env=<optimized out>, heap_filter=0, klass=0x0, initial_object=0x0, callbacks=0x7f7a7c253210, user_data=0x0)
    at /root/openj9-openjdk-jdk21/openj9/runtime/jvmti/jvmtiHeap.c:538
#28 0x00007f7a9d4806fb in _jvmtiEnv::FollowReferences (user_data=0x0, callbacks=0x7f7a7c253210, initial_object=0x0, klass=0x0, heap_filter=0, this=<optimized out>)
    at build/linux-x86_64-server-release/support/modules_include/java.base/jvmti.h:1292
#29 Java_VThreadStackRefTest_test (env=0x1dc400, clazz=<optimized out>, classes=<optimized out>) at test/hotspot/jtreg/serviceability/jvmti/vthread/FollowReferences/libVThreadStackRefTest.cpp:136
...

babsingh · 2023-07-27T21:23:37Z

@amicic @dmitripivkine @LinHu2016 Any updates or guidance on how to resolve this issue?

dmitripivkine · 2023-07-28T18:34:14Z

I don't know answer to this question, I don't know enough what VM is doing with VTs:

> !J9JVMTIHeapData 0x7F4C2441D280
J9JVMTIHeapData at 0x7f4c2441d280 {
  Fields for J9JVMTIHeapData:
	0x0: struct J9JVMTIEnv* env = !j9jvmtienv 0x00007F4C400AF6D8
	0x8: struct J9VMThread* currentThread = !j9vmthread 0x000000000021B200
	0x10: I32 filter = 0x00000000 (0)
	0x18: struct J9Class* classFilter = !j9class 0x0000000000000000
	0x20: void* userData = !j9x 0x0000000000000000
	0x28: struct J9Class* clazz = !j9class 0x0000000000245D00 // java/lang/VirtualThread$VThreadContinuation
	0x30: enum jvmtiError rc = 0x0 (0) //JVMTI_ERROR_NONE
	0x34: enum jvmtiIterationControl visitRc = 0x0 (0) //JVMTI_ITERATION_ABORT
	0x38: enum J9JVMTIHeapIterationFlags flags = 0x0 (0) <<Not matched to enum constant>>
	0x40: struct J9JVMTIHeapEvent event = !j9jvmtiheapevent 0x00007F4C2441D2C0
	0x98: struct J9Object* referrer = !j9object 0x00007F4C2441CDD0 // <FAULT>
	0xa0: struct J9Object* object = !j9object 0x00000000FFEF43C0 // java/lang/VirtualThread$VThreadContinuation <----- continuation object
	0xa8: I64 objectSize = 0x0000000000000030 (48)
	0xb0: struct jvmtiHeapTags tags = !jvmtiheaptags 0x00007F4C2441D330
	0xd0: const struct jvmtiHeapCallbacks* callbacks = !jvmtiheapcallbacks 0x00007F4C2441D3B0
}

> !j9object 0xFFEF43C0
!J9Object 0x00000000FFEF43C0 {
	struct J9Class* clazz = !j9class 0x245D00 // java/lang/VirtualThread$VThreadContinuation
	Object flags = 0x00000010;
	I lockword = 0x00000000 (offset = 0) (java/lang/Object) <hidden>
	J vmRef = 0x00007F4BC400DFF0 (offset = 4) (jdk/internal/vm/Continuation)
	Ljava/lang/Thread; vthread = !fj9object 0xffef4258 (offset = 20) (jdk/internal/vm/Continuation)
	Ljdk/internal/vm/ContinuationScope; scope = !fj9object 0xffef1258 (offset = 24) (jdk/internal/vm/Continuation)
	Ljava/lang/Runnable; runnable = !fj9object 0xffef83b8 (offset = 28) (jdk/internal/vm/Continuation)
	Ljdk/internal/vm/Continuation; parent = !fj9object 0x0 (offset = 32) (jdk/internal/vm/Continuation)
	J state = 0x0000000000253801 (offset = 12) (jdk/internal/vm/Continuation) <----- mounted?
	Z isAccessible = 0x00000000 (offset = 36) (jdk/internal/vm/Continuation)
	I continuationLink = 0x00000000 (offset = 40) (jdk/internal/vm/Continuation) <hidden>
}

> !stack 0x0000000000253800
<253800> 	!j9method 0x0000000000201FA8   VThreadStackRefTest.createObjAndWait(Ljava/lang/Class;)V
<253800> 	!j9method 0x0000000000202048   VThreadStackRefTest.lambda$main$2()V
<253800> 	!j9method 0x0000000000203138   VThreadStackRefTest$$Lambda/0x0000000024424b88.run()V
<253800> 	!j9method 0x00000000000923A8   java/lang/VirtualThread.runWith(Ljava/lang/Object;Ljava/lang/Runnable;)V
<253800> 	!j9method 0x0000000000092388   java/lang/VirtualThread.run(Ljava/lang/Runnable;)V
<253800> 	!j9method 0x0000000000244D90   java/lang/VirtualThread$VThreadContinuation$1.run()V
<253800> 	!j9method 0x00000000000A1740   jdk/internal/vm/Continuation.enter(Ljdk/internal/vm/Continuation;)V
<253800> 	                        JNI call-in frame
<253800> 	                        Native method frame

> !j9vmcontinuation 0x00007f4bc400dff0
J9VMContinuation at 0x7f4bc400dff0 {
  Fields for J9VMContinuation:
	0x0: UDATA* arg0EA = !j9x 0x00000000002533F0
	0x8: UDATA* bytecodes = !j9x 0x0000000000000000
	0x10: UDATA* sp = !j9x 0x00000000002533C8
	0x18: U8* pc = !j9x 0x0000000000000003
	0x20: struct J9Method* literals = !j9method 0x0000000000000000
	0x28: UDATA* stackOverflowMark = !j9x 0x0000000000252EC0
	0x30: UDATA* stackOverflowMark2 = !j9x 0x0000000000252EC0
	0x38: struct J9JavaStack* stackObject = !j9javastack 0x00000000002506B8
	0x40: struct J9JITDecompilationInfo* decompilationStack = !j9jitdecompilationinfo 0x0000000000000000
	0x48: UDATA* j2iFrame = !j9x 0x0000000000000000
	0x50: struct J9JITGPRSpillArea jitGPRs = !j9jitgprspillarea 0x00007F4BC400E040
	0xd0: struct J9I2JState i2jState = !j9i2jstate 0x00007F4BC400E0C0
	0xf0: struct J9VMEntryLocalStorage* oldEntryLocalStorage = !j9vmentrylocalstorage 0x0000000000000000
	0xf8: UDATA dropFlags = 0x0000000000000000 (0)
}

> !j9stackwalkstate 0x00007F4C2441CDD0
J9StackWalkState at 0x7f4c2441cdd0 {
  Fields for J9StackWalkState:
	0x0: struct J9StackWalkState* previous = !j9stackwalkstate 0x0000000000000004
	0x8: struct J9VMThread* walkThread = !j9vmthread 0x00007F4C2441C340 <-----
	0x10: struct J9JavaVM* javaVM = !j9javavm 0x00007F4C40031080
	0x18: UDATA flags = 0x0000000024600006 (610271238)
	0x20: UDATA* bp = !j9x 0x00000000002533E8
	0x28: UDATA* unwindSP = !j9x 0x00000000002533C8
	0x30: U8* pc = !j9x 0x0000000000000003
	0x38: U8* nextPC = !j9x 0x0000000000000000
	0x40: UDATA* sp = !j9x 0x00000000002533C8
	0x48: UDATA* arg0EA = !j9x 0x00000000002533F0
	0x50: struct J9Method* literals = !j9method 0x0000000000000000
	0x58: UDATA* walkSP = !j9x 0x00000000002533C8
	0x60: UDATA argCount = 0x0000000000000001 (1)
	0x68: struct J9ConstantPool* constantPool = !j9constantpool 0x00000000000A1030 (flags = 0x0)
	0x70: struct J9Method* method = !j9method 0x00000000000A18E0 // jdk/internal/vm/Continuation.enterImpl()Z
	0x78: struct J9JITExceptionTable* jitInfo = !j9jitexceptiontable 0x0000000000000000
	0x80: UDATA frameFlags = 0x0000000000000000 (0)
	0x88: UDATA resolveFrameFlags = 0x0000000000000000 (0)
	0x90: UDATA skipCount = 0x00007F4C45797BB7 (139965559831479)
	0x98: UDATA maxFrames = 0x000000000001F902 (129282)
	0xa0: void* userData1 = !j9x 0x00007F4C455A10E0
	0xa8: void* userData2 = !j9x 0x00007F4C40031080
	0xb0: void* userData3 = !j9x 0x00007F4C2441D100
	0xb8: void* userData4 = !j9x 0x00007F4C46727809
	0xc0: UDATA framesWalked = 0x0000000000000000 (0)
	0xc8: void* frameWalkFunction = !j9x 0x00007F4C455A9B20
	0xd0: void* objectSlotWalkFunction = !j9x 0x00007F4C455A9AF0
	0xd8: void* returnAddressWalkFunction = !j9x 0x0000000000000000
	0xe0: UDATA* cache = !j9x 0x0000000000000000
	0xe8: void* restartPoint = !j9x 0x00007F4C46FA2D85
	0xf0: void* restartException = !j9x 0x00007F4C00000000
	0xf8: void* inlinerMap = !j9x 0x0000000000000000
	0x100: UDATA inlineDepth = 0x0000000000000000 (0)
	0x108: UDATA* cacheCursor = !j9x 0x00007F4C2441CFE0
	0x110: struct J9JITDecompilationInfo* decompilationRecord = !j9jitdecompilationinfo 0x0000000000000000
	0x118: struct J9WalkStackFramesAndSlotsStorage registerEAs = !j9walkstackframesandslotsstorage 0x00007F4C2441CEE8
	0x198: struct J9VMEntryLocalStorage* walkedEntryLocalStorage = !j9vmentrylocalstorage 0x00007F4C2441C2E0
	0x1a0: struct J9I2JState* i2jState = !j9i2jstate 0x00007F4C2441C2F0
	0x1a8: struct J9JITDecompilationInfo* decompilationStack = !j9jitdecompilationinfo 0x0000000000000000
	0x1b0: void** pcAddress = !j9x 0x00007F4C2441C368
	0x1b8: UDATA outgoingArgCount = 0x0000000000000000 (0)
	0x1c0: U8* objectSlotBitVector = !j9x 0x0000000000801000 // ""
	0x1c8: UDATA elsBitVector = 0x0000003000000020 (206158430240)
	0x1d0: void* savedObjectSlotWalkFunction = !j9x 0x0000000000000000
	0x1d8: IDATA bytecodePCOffset = 0xFFFFFFFFFFFFFFFF (-1)
	0x1e0: void* dropToCurrentFrame = !j9x 0x00007F4C46DFA3A0
	0x1e8: UDATA* j2iFrame = !j9x 0x0000000000000000
	0x1f0: UDATA previousFrameFlags = 0x0000000000000000 (0)
	0x1f8: IDATA slotIndex = 0x0000000000000000 (0)
	0x200: UDATA slotType = 0x0000000000000001 (1)
	0x208: struct J9VMThread* currentThread = !j9vmthread 0x000000000021B200
	0x210: void* linearSlotWalker = !j9x 0x0000000000000000
	0x218: void* inlinedCallSite = !j9x 0x0000000000000000
	0x220: void* stackMap = !j9x 0x0000000000000000
	0x228: void* inlineMap = !j9x 0x0000000000000000
	0x230: UDATA loopBreaker = 0x0000000000000000 (0)
}

And yes, !j9vmthread 0x00007F4C2441C340 is stack allocated and has threadObject set to NULL.
I am not sure what rules should be applied for this case. I don't think this is GC question.

tajila · 2023-07-31T15:54:52Z

@babsingh the jvmti code expects the walkthread to be a carrier thread. See jvmtiHeapFollowRefs_getStackData.

I general I dont think continuations should ever need to scan JNI_LOCALS, since it its unmounted it cannot have a JNI frame. This eliminates one requirement for TID. The other is JVMTI_HEAP_REFERENCE_STACK_LOCAL, we may be able to just put a zero for TID there. But we should check what the RI does for this.

Unmounted virtual threads cannot have have a JNI frame. JNI_LOCAL requests should be ignored for unmounted virtual threads. STACK_LOCAL requests use 0 for TID since it is not currently feasible to derive this information for unmounted virtual threads. Related: eclipse-openj9#17712 Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>

babsingh · 2023-08-25T18:34:28Z

@babsingh the jvmti code expects the walkthread to be a carrier thread. See jvmtiHeapFollowRefs_getStackData.

I general I dont think continuations should ever need to scan JNI_LOCALS, since it its unmounted it cannot have a JNI frame. This eliminates one requirement for TID. The other is JVMTI_HEAP_REFERENCE_STACK_LOCAL, we may be able to just put a zero for TID there. But we should check what the RI does for this.

@tajila #18020 has the above changes. We avoid the segfault but the test still fails. We fail the following scenarios: VThreadUnmountedReferenced and VThreadMountedJNIReferenced. In the failing cases, we return 0 for TID whereas the RI returns the correct TID. More details are below.

Test source code

https://github.com/ibmruntimes/openj9-openjdk-jdk21/tree/openj9/test/hotspot/jtreg/serviceability/jvmti/vthread/FollowReferences

Test output with OpenJ9 + #18020

STDOUT:
created class VThreadStackRefTest$VThreadUnmountedReferenced
created class VThreadStackRefTest$VThreadUnmountedEnded
created class VThreadStackRefTest$VThreadMountedReferenced
created class VThreadStackRefTest$VThreadMountedJNIReferenced
created class VThreadStackRefTest$PThreadReferenced
threads:
  - vthreadUnmounted: VirtualThread[#27]/waiting
  - vthreadEnded: VirtualThread[#31]/terminated
  - vthreadMounted: VirtualThread[#32]/runnable@ForkJoinPool-1-worker-1
  - pthread: Thread[#33,Thread-1,5,MainThreadGroup]
test classes:
  (0) class VThreadStackRefTest$VThreadUnmountedReferenced
  (1) class VThreadStackRefTest$VThreadUnmountedJNIReferenced
  (2) class VThreadStackRefTest$VThreadMountedReferenced
  (3) class VThreadStackRefTest$VThreadMountedJNIReferenced
  (4) class VThreadStackRefTest$PThreadReferenced
  (5) class VThreadStackRefTest$VThreadUnmountedEnded
INFO: No mount/unmount checks
Stack local: index = 2, thread_id = 32
Stack local: index = 4, thread_id = 33
  (0) ERROR class VThreadStackRefTest$VThreadUnmountedReferenced: ref count = 0 (expected 1), thread id = 0 (expected 27)
  (1) OK class VThreadStackRefTest$VThreadUnmountedJNIReferenced: ref count = 0 (expected 0), thread id = 0 (expected 0)
  (2) OK class VThreadStackRefTest$VThreadMountedReferenced: ref count = 1 (expected 1), thread id = 32 (expected 32)
  (3) ERROR class VThreadStackRefTest$VThreadMountedJNIReferenced: ref count = 0 (expected 1), thread id = 0 (expected 32)
  (4) OK class VThreadStackRefTest$PThreadReferenced: ref count = 1 (expected 1), thread id = 33 (expected 33)
  (5) OK class VThreadStackRefTest$VThreadUnmountedEnded: ref count = 0 (expected 0), thread id = 0 (expected 0)

Test output with the RI

STDOUT:
created class VThreadStackRefTest$VThreadUnmountedReferenced
created class VThreadStackRefTest$VThreadUnmountedEnded
created class VThreadStackRefTest$VThreadMountedReferenced
created class VThreadStackRefTest$VThreadMountedJNIReferenced
created class VThreadStackRefTest$PThreadReferenced
threads:
  - vthreadUnmounted: VirtualThread[#21]/waiting
  - vthreadEnded: VirtualThread[#22]/terminated
  - vthreadMounted: VirtualThread[#23]/runnable
  - pthread: Thread[#24,Thread-1,5,MainThreadGroup]
test classes:
  (0) class VThreadStackRefTest$VThreadUnmountedReferenced
  (1) class VThreadStackRefTest$VThreadUnmountedJNIReferenced
  (2) class VThreadStackRefTest$VThreadMountedReferenced
  (3) class VThreadStackRefTest$VThreadMountedJNIReferenced
  (4) class VThreadStackRefTest$PThreadReferenced
  (5) class VThreadStackRefTest$VThreadUnmountedEnded
INFO: No mount/unmount checks
Stack local: index = 0, thread_id = 21
JNI local: index = 3, thread_id = 23
Stack local: index = 2, thread_id = 23
Stack local: index = 4, thread_id = 24
  (0) OK class VThreadStackRefTest$VThreadUnmountedReferenced: ref count = 1 (expected 1), thread id = 21 (expected 21)
  (1) OK class VThreadStackRefTest$VThreadUnmountedJNIReferenced: ref count = 0 (expected 0), thread id = 0 (expected 0)
  (2) OK class VThreadStackRefTest$VThreadMountedReferenced: ref count = 1 (expected 1), thread id = 23 (expected 23)
  (3) OK class VThreadStackRefTest$VThreadMountedJNIReferenced: ref count = 1 (expected 1), thread id = 23 (expected 23)
  (4) OK class VThreadStackRefTest$PThreadReferenced: ref count = 1 (expected 1), thread id = 24 (expected 24)
  (5) OK class VThreadStackRefTest$VThreadUnmountedEnded: ref count = 0 (expected 0), thread id = 0 (expected 0)

babsingh · 2023-08-30T21:25:41Z

The reported VThreadStackRefTest failure happens in both the test variants: default and no-vmcontinuations.

JasonFengJ9 · 2023-09-05T12:40:53Z

JDK21 aarch64_mac

08:17:04  variation: Mode650
08:17:04  JVM_OPTIONS:  -XX:-UseCompressedOops 

08:18:34  TEST: serviceability/jvmti/vthread/FollowReferences/VThreadStackRefTest.java#default

08:18:34  STDERR:
08:18:34  Unhandled exception
08:18:34  Type=Segmentation error vmState=0x00040000
08:18:34  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000002
08:18:34  Handler1=00000001005CCB80 Handler2=0000000100482000 InaccessibleAddress=0000000000000018
08:18:34  x0=0000000000000000 x1=0000000000000000 x2=00000001780CBF20 x3=000000010140B1E0
08:18:34  x4=FFFFFFFFFFFFFFFF x5=0000000000000000 x6=000000000000006D x7=0000000000000004
08:18:34  x8=0000000000000000 x9=0000000000000010 x10=0000000000000010 x11=0000000000000002
08:18:34  x12=0000000000000002 x13=0000000000000000 x14=000000013F800000 x15=0000000000000000
08:18:34  x16=00000001A0344AF0 x17=000000020085CDB0 x18=000000017098EC10 x19=000000017098E070
08:18:34  x20=000000014481C020 x21=0000000000000000 x22=000000010140B1B8 x23=000000017098E0B8
08:18:34  x24=00000002801D5500 x25=0000000000000000 x26=0000000000000000 x27=000000017098DB58
08:18:34  x28=0000000000000000 x29(FP)=000000017098CEB0 x30(LR)=00000001008649E0 x31(SP)=000000017098CE20
08:18:34  PC=00000001008649FC SP=000000017098CE20
08:18:34  v0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v1 00000001780cbda0 (f: 2014100864.000000, d: 3.117094e-314)
08:18:34  v2 000001be0000013e (f: 318.000000, d: 9.464101e-312)
08:18:34  v3 000001be000001be (f: 446.000000, d: 9.464101e-312)
08:18:34  v4 00000000000001be (f: 446.000000, d: 2.203533e-321)
08:18:34  v5 0000013e0000013e (f: 318.000000, d: 6.747947e-312)
08:18:34  v6 000000000000013e (f: 318.000000, d: 1.571129e-321)
08:18:34  v7 0000013e00000000 (f: 0.000000, d: 6.747947e-312)
08:18:34  v8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v16 bfd0000000000000 (f: 0.000000, d: -2.500000e-01)
08:18:34  v17 3fd559e2a6a41555 (f: 2795771136.000000, d: 3.336112e-01)
08:18:34  v18 bf5237dcca8226ac (f: 3397527296.000000, d: -1.111951e-03)
08:18:34  v19 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
08:18:34  v20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  Module=/Users/jenkins/workspace/Test_openjdk21_j9_extended.openjdk_aarch64_mac_Personal/openjdkbinary/j2sdk-image/lib/default/libj9jvmti29.dylib
08:18:34  Module_base_address=0000000100854000 Symbol=followReferencesCallback
08:18:34  Symbol_address=00000001008642FC
08:18:34  Target=2_90_20230903_56 (Mac OS X 13.0)
08:18:34  CPU=aarch64 (8 logical CPUs) (0x400000000 RAM)
08:18:34  ----------- Stack Backtrace -----------
08:18:34  ---------------------------------------
08:18:34  JVMDUMP039I Processing dump event "gpf", detail "" at 2023/09/03 08:18:10 - please wait.

08:18:34  TEST RESULT: Failed. Unexpected exit from test [exit code: 255]
08:18:34  --------------------------------------------------
08:18:34  TEST: serviceability/jvmti/vthread/FollowReferences/VThreadStackRefTest.java#no-vmcontinuations

08:18:34  STDERR:
08:18:34  Unhandled exception
08:18:34  Type=Segmentation error vmState=0x00040000
08:18:34  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000002
08:18:34  Handler1=0000000102BBCB80 Handler2=0000000102A72000 InaccessibleAddress=0000000000000018
08:18:34  x0=0000000000000000 x1=0000000000000000 x2=00000001780ABF20 x3=0000000103A09CD0
08:18:34  x4=FFFFFFFFFFFFFFFF x5=0000000000000000 x6=000000000000006D x7=0000000000000004
08:18:34  x8=0000000000000000 x9=0000000000000010 x10=0000000000000010 x11=0000000000000002
08:18:34  x12=0000000000000002 x13=0000000000000000 x14=0000000142800000 x15=0000000000000000
08:18:34  x16=00000001A0344AF0 x17=000000020085CDB0 x18=000000016E436C10 x19=000000016E436070
08:18:34  x20=000000014181A620 x21=0000000000000000 x22=0000000103A09CA8 x23=000000016E4360B8
08:18:34  x24=0000000170C95400 x25=0000000000000000 x26=0000000000000000 x27=000000016E435B58
08:18:34  x28=0000000000000000 x29(FP)=000000016E434EB0 x30(LR)=0000000102E549E0 x31(SP)=000000016E434E20
08:18:34  PC=0000000102E549FC SP=000000016E434E20
08:18:34  v0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v1 000000028000c360 (f: 2147533568.000000, d: 5.305014e-314)
08:18:34  v2 000001be0000013e (f: 318.000000, d: 9.464101e-312)
08:18:34  v3 000001be000001be (f: 446.000000, d: 9.464101e-312)
08:18:34  v4 00000000000001be (f: 446.000000, d: 2.203533e-321)
08:18:34  v5 0000013e0000013e (f: 318.000000, d: 6.747947e-312)
08:18:34  v6 000000000000013e (f: 318.000000, d: 1.571129e-321)
08:18:34  v7 0000013e00000000 (f: 0.000000, d: 6.747947e-312)
08:18:34  v8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v16 bfd0000000000000 (f: 0.000000, d: -2.500000e-01)
08:18:34  v17 3fd54864d12fd555 (f: 3509572864.000000, d: 3.325436e-01)
08:18:34  v18 3f69d697062c8233 (f: 103580208.000000, d: 3.154082e-03)
08:18:34  v19 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
08:18:34  v20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  v31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
08:18:34  Module=/Users/jenkins/workspace/Test_openjdk21_j9_extended.openjdk_aarch64_mac_Personal/openjdkbinary/j2sdk-image/lib/default/libj9jvmti29.dylib
08:18:34  Module_base_address=0000000102E44000 Symbol=followReferencesCallback
08:18:34  Symbol_address=0000000102E542FC
08:18:34  Target=2_90_20230903_56 (Mac OS X 13.0)
08:18:34  CPU=aarch64 (8 logical CPUs) (0x400000000 RAM)
08:18:34  ----------- Stack Backtrace -----------
08:18:34  ---------------------------------------
08:18:34  JVMDUMP039I Processing dump event "gpf", detail "" at 2023/09/03 08:18:10 - please wait.

08:18:34  TEST RESULT: Failed. Unexpected exit from test [exit code: 255]
08:18:34  --------------------------------------------------
08:18:58  Test results: passed: 152; failed: 2
08:19:01  Report written to /Users/jenkins/workspace/Test_openjdk21_j9_extended.openjdk_aarch64_mac_Personal/jvmtest/openjdk/report/html/report.html
08:19:01  Results written to /Users/jenkins/workspace/Test_openjdk21_j9_extended.openjdk_aarch64_mac_Personal/aqa-tests/TKG/output_16937434035787/serviceability_jvmti_j9_1/work
08:19:01  Error: Some tests failed or other problems occurred.
08:19:01  -----------------------------------
08:19:01  serviceability_jvmti_j9_1_FAILED

babsingh · 2023-09-05T17:15:34Z

@llxia In adoptium/aqa-tests#4738, we added an exclude for VThreadStackRefTest. The test variants still run. Do we need two separate excludes as below?

serviceability/jvmti/vthread/FollowReferences/VThreadStackRefTest.java#no-vmcontinuations https://github.com/eclipse-openj9/openj9/issues/17712 generic-all
serviceability/jvmti/vthread/FollowReferences/VThreadStackRefTest.java#default https://github.com/eclipse-openj9/openj9/issues/17712 generic-all

llxia · 2023-09-05T18:41:20Z

I think so.

babsingh · 2023-09-05T20:10:27Z

re #17712 (comment): adoptium/aqa-tests#4744 individually excludes each VThreadStackRefTest variant.

babsingh · 2023-09-29T14:30:42Z

@LinHu2016 Can you post in this issue so we can assign this issue to you?

LinHu2016 · 2023-09-29T17:52:13Z

I have tried two fixes:

* for the first fix, the associated PR and results are mentioned in [JDK21 serviceability/jvmti/vthread/FollowReferences/VThreadStackRefTest.java Segmentation error vmState=0x00040000 #17712 (comment)](https://github.com/eclipse-openj9/openj9/issues/17712#issuecomment-1693778164); and

* the second fix is [Pass threadObject to walkContinuationStackFrames #18180](https://github.com/eclipse-openj9/openj9/pull/18180).

Results from the second fix

The results are exactly identical to the first fix. The segfault is resolved but the references are not correctly calculated for the VThreadUnmountedReferenced and VThreadMountedJNIReferenced scenarios. @LinHu2016 @amicic @dmitripivkine Is there any support missing which will prevent us to correctly follow references for these two scenarios?

STDOUT:
created class VThreadStackRefTest$VThreadUnmountedReferenced
created class VThreadStackRefTest$VThreadUnmountedEnded
created class VThreadStackRefTest$VThreadMountedReferenced
created class VThreadStackRefTest$VThreadMountedJNIReferenced
created class VThreadStackRefTest$PThreadReferenced
threads:
  - vthreadUnmounted: VirtualThread[#27]/waiting
  - vthreadEnded: VirtualThread[#32]/terminated
  - vthreadMounted: VirtualThread[#33]/runnable@ForkJoinPool-1-worker-1
  - pthread: Thread[#34,Thread-1,5,MainThreadGroup]
test classes:
  (0) class VThreadStackRefTest$VThreadUnmountedReferenced
  (1) class VThreadStackRefTest$VThreadUnmountedJNIReferenced
  (2) class VThreadStackRefTest$VThreadMountedReferenced
  (3) class VThreadStackRefTest$VThreadMountedJNIReferenced
  (4) class VThreadStackRefTest$PThreadReferenced
  (5) class VThreadStackRefTest$VThreadUnmountedEnded
INFO: No mount/unmount checks
Stack local: index = 2, thread_id = 33
Stack local: index = 4, thread_id = 34
  (0) ERROR class VThreadStackRefTest$VThreadUnmountedReferenced: ref count = 0 (expected 1), thread id = 0 (expected 27)
  (1) OK class VThreadStackRefTest$VThreadUnmountedJNIReferenced: ref count = 0 (expected 0), thread id = 0 (expected 0)
  (2) OK class VThreadStackRefTest$VThreadMountedReferenced: ref count = 1 (expected 1), thread id = 33 (expected 33)
  (3) ERROR class VThreadStackRefTest$VThreadMountedJNIReferenced: ref count = 0 (expected 1), thread id = 0 (expected 33)
  (4) OK class VThreadStackRefTest$PThreadReferenced: ref count = 1 (expected 1), thread id = 34 (expected 34)
  (5) OK class VThreadStackRefTest$VThreadUnmountedEnded: ref count = 0 (expected 0), thread id = 0 (expected 0)

For (0) ERROR class VThreadStackRefTest$VThreadUnmountedReferenced: ref count = 0 (expected 1), thread id = 0 (expected 27) it has been identified that missing feature to handling continuation Object in MM_ReferenceChainWalker, the PR #18214 has been created for fixing the issue, but (3) ERROR class VThreadStackRefTest$VThreadMountedJNIReferenced: ref count = 0 (expected 1), thread id = 0 (expected 33) is different problem, the PR #18214 would not intend to handle the JNI issue.

the below is that test result of the latest local build with PR #18214

command: main -XX:+UnlockExperimentalVMOptions -XX:-VMContinuations -agentlib:VThreadStackRefTest VThreadStackRefTest NoMountCheck
reason: User specified action: run main/othervm/native -XX:+UnlockExperimentalVMOptions -XX:-VMContinuations -agentlib:VThreadStackRefTest VThreadStackRefTest NoMountCheck 
started: Thu Sep 28 14:54:33 PDT 2023
Mode: othervm [/othervm specified]
finished: Thu Sep 28 14:54:33 PDT 2023
elapsed time (seconds): 0.835
configuration:
STDOUT:
created class VThreadStackRefTest$VThreadUnmountedReferenced
created class VThreadStackRefTest$VThreadUnmountedEnded
created class VThreadStackRefTest$VThreadMountedReferenced
created class VThreadStackRefTest$VThreadMountedJNIReferenced
created class VThreadStackRefTest$PThreadReferenced
threads:
  - vthreadUnmounted: VirtualThread[#27]/waiting
  - vthreadEnded: VirtualThread[#31]/terminated
  - vthreadMounted: VirtualThread[#32]/runnable@ForkJoinPool-1-worker-1
  - pthread: Thread[#33,Thread-1,5,MainThreadGroup]
test classes:
  (0) class VThreadStackRefTest$VThreadUnmountedReferenced
  (1) class VThreadStackRefTest$VThreadUnmountedJNIReferenced
  (2) class VThreadStackRefTest$VThreadMountedReferenced
  (3) class VThreadStackRefTest$VThreadMountedJNIReferenced
  (4) class VThreadStackRefTest$PThreadReferenced
  (5) class VThreadStackRefTest$VThreadUnmountedEnded
INFO: No mount/unmount checks
Stack local: index = 2, thread_id = 32
Stack local: index = 4, thread_id = 33
Stack local: index = 0, thread_id = 27
  (0) OK class VThreadStackRefTest$VThreadUnmountedReferenced: ref count = 1 (expected 1), thread id = 27 (expected 27)
  (1) OK class VThreadStackRefTest$VThreadUnmountedJNIReferenced: ref count = 0 (expected 0), thread id = 0 (expected 0)
  (2) OK class VThreadStackRefTest$VThreadMountedReferenced: ref count = 1 (expected 1), thread id = 32 (expected 32)
  (3) ERROR class VThreadStackRefTest$VThreadMountedJNIReferenced: ref count = 0 (expected 1), thread id = 0 (expected 32)
  (4) OK class VThreadStackRefTest$PThreadReferenced: ref count = 1 (expected 1), thread id = 33 (expected 33)
  (5) OK class VThreadStackRefTest$VThreadUnmountedEnded: ref count = 0 (expected 0), thread id = 0 (expected 0)
STDERR:
java.lang.RuntimeException: Test failed
	at VThreadStackRefTest.main(VThreadStackRefTest.java:192)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
	at java.base/java.lang.Thread.run(Thread.java:1595)

JavaTest Message: Test threw exception: java.lang.RuntimeException: Test failed
JavaTest Message: shutting down test

threadObject is required in jvmtiFollowReferences to get thread specific details such as the TID. So, threadObject needs to be passed via walkContinuationStackFrames so that it is accessible in jvmtiFollowReferences. walkContinuationStackFrames is invoked either for a [1] J9VMContinuation in the Continuation object or [2] J9VMThread->currentContinuation In [1], if the Continuation is fully mounted, then it stores the state of the CarrierThread. Otherwise, it stores the state of the VirtualThread. In [2], if J9VMThread->currentContinuation is not NULL, then it stores the state of the CarrierThread because the VirtualThread is mounted on the J9VMThread. The CarrierThread is retrieved from J9VMThread->carrierThreadObject. We have assumed that the state of the Continuation or J9VMContinuation doesn't change while walkContinuationStackFrames is invoked i.e. the VirtualThread shouldn't be mounted or unmounted. Otherwise, the values of the VirtualThread and CarrierThread objects accessed via the Continuation object and J9VMThread->carrierThreadObject can vary and cause a crash during walkContinuationStackFrames. Related: eclipse-openj9#17712 Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>

tajila · 2023-10-05T19:28:15Z

@gacholio After looking at j9gc_ext_reachable_objects_do. It looks like it calls scanThreads which calls GC_VMThreadIterator and then calls GC_VMThreadJNISlotIterator. GC_VMThreadJNISlotIterator walks the vmthread->jniLocalReferences pool.

Can we actually rely on jniLocalReferences to find all jniLocals?

It looks like when we call j9jni_createLocalRef we try to allocate on the java stack, failing that we allocate it out of jniLocalReferences->references.

@LinHu2016 Is there a reason why we dont just do a stack walk in scanThreads?

gacholio · 2023-10-06T13:52:21Z

As you say, stack-based refs are just O slots reported by the stack walk.

LinHu2016 · 2023-10-10T14:48:32Z

@gacholio After looking at j9gc_ext_reachable_objects_do. It looks like it calls scanThreads which calls GC_VMThreadIterator and then calls GC_VMThreadJNISlotIterator. GC_VMThreadJNISlotIterator walks the vmthread->jniLocalReferences pool.

Can we actually rely on jniLocalReferences to find all jniLocals?

It looks like when we call j9jni_createLocalRef we try to allocate on the java stack, failing that we allocate it out of jniLocalReferences->references.

@LinHu2016 Is there a reason why we dont just do a stack walk in scanThreads?

@TobiAjila GC does two stack walks, one for normal thread java stack, another one is only for mounted virtual thread case - scan currentContinuation(carrierthread stack, we don't stackwalk for mounted continuation during GC heap walk).

tajila · 2023-10-10T18:16:53Z

we don't stackwalk for mounted continuation during GC heap walk

@LinHu2016 The test is failing because we are not finding objects on the stack. The RI is able to find them. We should try to match their behaviour for JVMTI.

LinHu2016 · 2023-10-11T13:57:09Z

we don't stackwalk for mounted continuation during GC heap walk

@LinHu2016 The test is failing because we are not finding objects on the stack. The RI is able to find them. We should try to match their behaviour for JVMTI.

Hi @tajila , for regular GC scan, we have scanned mounted continuation( inside is carrier thread java stack) during root scanning(threads scan), so scanning mounted continuation in heap scanning is duplicated( also for some GC processing, it doesn't allow scanning second time such as compaction), we can treat JVMTI heap walk(referencechainwalker) differently, but I need to confirm if JVMTI is ok for duplicated scan or we need to stop scan currentContinuation during root scanning(thread scanning) for JVMTI case.

tajila · 2023-10-12T17:31:10Z

Scan all slots

void
MM_RootScanner::scanAllSlots(MM_EnvironmentBase *env)
{
	if(!_nurseryReferencesOnly && !_nurseryReferencesPossibly) {
		scanClasses(env);
		scanVMClassSlots(env);
	}

	scanClassLoaders(env);

	scanThreads(env);
#if defined(J9VM_GC_FINALIZATION)
	scanFinalizableObjects(env);
#endif /* J9VM_GC_FINALIZATION */
	scanJNIGlobalReferences(env);

	if(!_nurseryReferencesOnly && !_nurseryReferencesPossibly) {
		scanStringTable(env);
	}

	scanWeakReferenceObjects(env);
	scanSoftReferenceObjects(env);
	scanPhantomReferenceObjects(env);

#if defined(J9VM_GC_FINALIZATION)
	scanUnfinalizedObjects(env);
#endif /* J9VM_GC_FINALIZATION */

	scanMonitorReferences(env);
	scanJNIWeakGlobalReferences(env);

#if defined(J9VM_GC_MODRON_SCAVENGER)
	if(_includeRememberedSetReferences && !_nurseryReferencesOnly && !_nurseryReferencesPossibly) {
		scanRememberedSet(env);
	}
#endif /* J9VM_GC_MODRON_SCAVENGER */
	
#if defined(J9VM_OPT_JVMTI)
	if(_includeJVMTIObjectTagTables) {
		scanJVMTIObjectTagTables(env);
	}
#endif /* J9VM_OPT_JVMTI */

#if defined(J9VM_GC_ENABLE_DOUBLE_MAP)
        if (_includeDoubleMap) {
                scanDoubleMappedObjects(env);
        }
#endif /* J9VM_GC_ENABLE_DOUBLE_MAP */

	scanOwnableSynchronizerObjects(env);
	scanContinuationObjects(env);
}

tajila · 2023-10-12T17:34:12Z

MM_HeapRootScanner::scanThreads()
{
	reportScanningStarted(RootScannerEntity_Threads);
	setReachability(RootScannerEntityReachability_Strong);
	
	GC_VMThreadListIterator vmThreadListIterator(_javaVM);
	
	while (J9VMThread *walkThread = vmThreadListIterator.nextVMThread()) {
		if (scanOneThread(walkThread)) {
			vmThreadListIterator.reset(_javaVM->mainThread);
		}
	}

	reportScanningEnded(RootScannerEntity_Threads);
}

bool
MM_HeapRootScanner::scanOneThread(J9VMThread* walkThread)
{
	GC_VMThreadIterator vmThreadIterator(walkThread);
	
	while (J9Object **slot = vmThreadIterator.nextSlot()) {
		doVMThreadSlot(slot, &vmThreadIterator);
	}
	
	return false;
}

public:
	GC_VMThreadIterator(J9VMThread *vmThread) :
		_vmThread(vmThread),
		_state(vmthreaditerator_state_start),
		_threadSlotIterator(_vmThread),
		_jniSlotIterator(vmThread)
#if defined(J9VM_INTERP_HOT_CODE_REPLACEMENT)
		, _monitorRecordSlotIterator(_vmThread)
#endif
		{};

tajila · 2023-10-12T17:36:18Z

j9object_t *
GC_VMThreadIterator::nextSlot()
{
	j9object_t *slotPtr;

	switch(_state) {
	case vmthreaditerator_state_start:
		_state += 1;

	case vmthreaditerator_state_slots:
		slotPtr = _threadSlotIterator.nextSlot();
		if(NULL != slotPtr) {
			return slotPtr;
		}
		_state += 1;
		
	case vmthreaditerator_state_jni_slots:
		slotPtr = _jniSlotIterator.nextSlot();
		if(NULL != slotPtr) {
			return slotPtr;
		}
#if defined(J9VM_INTERP_HOT_CODE_REPLACEMENT)
		_state += 1;
		
	case vmthreaditerator_state_monitor_records:
		slotPtr = _monitorRecordSlotIterator.nextSlot();
		if(NULL != slotPtr) {
			return slotPtr;
		}
#endif
		_state += 1;

	default:
		break;
	}

	return NULL;
}

j9object_t *
GC_VMThreadSlotIterator::nextSlot() 
{
	switch (_scanIndex++) {
	case 0:
		return &(_vmThread->jitException);
	case 1:
		return &(_vmThread->currentException);
	case 2:
		return &(_vmThread->threadObject);
	case 3:
		return &(_vmThread->stopThrowable);
	case 4:
		return &(_vmThread->outOfMemoryError);
	case 5:
		return &(_vmThread->blockingEnterObject);
	case 6:
		return &(_vmThread->forceEarlyReturnObjectSlot);
	case 7:
		return &(_vmThread->javaLangThreadLocalCache);
	case 8:
		return (j9object_t *)&(_vmThread->omrVMThread->_savedObject1);
	case 9:
		return (j9object_t *)&(_vmThread->omrVMThread->_savedObject2);
#if JAVA_SPEC_VERSION >= 19
	case 10:
		return &(_vmThread->carrierThreadObject);
	case 11:
		return &(_vmThread->scopedValueCache);
#endif /* JAVA_SPEC_VERSION >= 19 */
	default:
		break;
	}
	return NULL;
}

j9object_t *
GC_VMThreadJNISlotIterator::nextSlot()
{
	while(_jniFrame) {
		j9object_t *objectPtr;

		objectPtr = (j9object_t *)_poolIterator.nextSlot();
		if(objectPtr) {
			return objectPtr;
		}

		_jniFrame = _jniFrame->previous;
		_poolIterator.init(_jniFrame ? (J9Pool *)_jniFrame->references : (J9Pool *)NULL);
	}
	return NULL;
}

tajila · 2023-10-12T17:37:03Z

@LinHu2016 Based on the snippets above, I couldn't find where a stack walk was occurring.

dmitripivkine · 2023-10-12T17:39:48Z

Are you asking about stacks for regular (carrier) threads or unmounted VTs?

LinHu2016 · 2023-10-12T17:55:41Z

@tajila
class MM_ReferenceChainWalker : public MM_RootScanner not from MM_HeapRootScanner,
there is no scanOneThread() in MM_ReferenceChainWalker, it should use base class MM_RootScanner::scanOneThread(),

bool
MM_RootScanner::scanOneThread(MM_EnvironmentBase *env, J9VMThread* walkThread, void* localData)
{
	GC_VMThreadIterator vmThreadIterator(walkThread);

	while(J9Object **slot = vmThreadIterator.nextSlot()) {
		doVMThreadSlot(slot, &vmThreadIterator);
	}

	J9VMThread *currentThread = (J9VMThread *)env->getOmrVMThread()->_language_vmthread;
	/* In a case this thread is a carrier thread, and a virtual thread is mounted, we will scan virtual thread's stack.
	 * If virtual thread is not mounted, or this is just a regular thread, this will scan its own stack. */
	GC_VMThreadStackSlotIterator::scanSlots(currentThread, walkThread, localData, stackSlotIterator, isStackFrameClassWalkNeeded(), _trackVisibleStackFrameDepth);

#if JAVA_SPEC_VERSION >= 19
	if (NULL != walkThread->currentContinuation)
	{
		/* At this point we know that a virtual thread is mounted. We previously scanned its stack,
		 * and now we will scan carrier's stack, that continuation struct is currently pointing to. */
		GC_VMThreadStackSlotIterator::scanSlots(currentThread, walkThread, walkThread->currentContinuation, localData, stackSlotIterator, isStackFrameClassWalkNeeded(), _trackVisibleStackFrameDepth);
	}
#endif /* JAVA_SPEC_VERSION >= 19 */
	return false;
}

LinHu2016 · 2023-10-12T18:06:22Z

/**
 * @todo Provide function documentation
 */
void
MM_ReferenceChainWalker::doVMThreadSlot(J9Object **slotPtr, GC_VMThreadIterator *vmThreadIterator)
{
	J9Object *slotValue = *slotPtr;

	switch(vmThreadIterator->getState()) {
	case vmthreaditerator_state_slots:
		doSlot(slotPtr, J9GC_ROOT_TYPE_THREAD_SLOT, -1, NULL);
		break;
	case vmthreaditerator_state_jni_slots:
		doSlot(slotPtr, J9GC_ROOT_TYPE_JNI_LOCAL, -1, NULL);
		break;
#if defined(J9VM_INTERP_HOT_CODE_REPLACEMENT)
	case vmthreaditerator_state_monitor_records:
		if (isHeapObject(slotValue) && !_heap->objectIsInGap(slotValue)) {
			doSlot(slotPtr, J9GC_ROOT_TYPE_THREAD_MONITOR, -1, NULL);
		}
		break;
#endif
	default:
		doSlot(slotPtr, J9GC_ROOT_TYPE_UNKNOWN, -1, NULL);
		break;
	}
}

fengxue-IS · 2023-10-30T21:10:01Z

Per discussion with @LinHu2016 , the cause of this test failure is not the JNI local ref not being found by GC, rather GC have located the object via stack walking but returned the reference with an incorrect type, hence the test code wasn't able to find the JNI ref expected from API.

JasonFengJ9 added comp:vm test failure comp:jvmti jdk21 labels Jun 29, 2023

JasonFengJ9 added this to the Java 21 milestone Jun 29, 2023

JasonFengJ9 mentioned this issue Jun 29, 2023

JDK21 serviceability_jvmti_j9 a few failures #17671

Closed

babsingh self-assigned this Jun 30, 2023

babsingh mentioned this issue Jun 30, 2023

[JDK21] Disable failing JVMTI tests until they are fixed adoptium/aqa-tests#4655

Merged

babsingh added the test excluded label Jun 30, 2023

babsingh removed their assignment Jul 27, 2023

babsingh mentioned this issue Aug 25, 2023

Update jvmtiHeapFollowRefs_getStackData to handle virtual threads #18020

Closed

fengxue-IS mentioned this issue Aug 30, 2023

JTReg test failure serviceability/jvmti/vthread/FollowReferences/VThreadStackRefTest.java#default #18053

Closed

babsingh changed the title ~~JDK21 serviceability/jvmti/vthread/FollowReferences/VThreadStackRefTest.java#no-vmcontinuations Segmentation error vmState=0x00040000~~ JDK21 serviceability/jvmti/vthread/FollowReferences/VThreadStackRefTest.java Segmentation error vmState=0x00040000 Aug 30, 2023

babsingh mentioned this issue Sep 5, 2023

jdk19 OpenJDK java/lang/Thread/virtual/stress/Skynet.java hang / timeout #16728

Closed

babsingh self-assigned this Sep 7, 2023

fengxue-IS added the project:loom Used to track Project Loom related work label Sep 16, 2023

tajila modified the milestones: Java 21 (0.42), Release 0.43 (Java 8, 11, 17, 21) Jan refresh Oct 5, 2023

fengxue-IS mentioned this issue Oct 30, 2023

Recognize JNI local refs in JNINativeMethodFrames #18378

Merged

LinHu2016 mentioned this issue Oct 30, 2023

Fix JNI Local Reference reporting issue #18379

Merged

fengxue-IS mentioned this issue Nov 1, 2023

Enable VThreadStackRefTest.java adoptium/aqa-tests#4847

Merged

pshipton modified the milestones: Release 0.43 (Java 8, 11, 17, 21) Jan refresh, Java 21 (0.42) Nov 1, 2023

smlambert closed this as completed in adoptium/aqa-tests#4847 Nov 2, 2023

pshipton removed the test excluded label Nov 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JDK21 serviceability/jvmti/vthread/FollowReferences/VThreadStackRefTest.java Segmentation error vmState=0x00040000 #17712

JDK21 serviceability/jvmti/vthread/FollowReferences/VThreadStackRefTest.java Segmentation error vmState=0x00040000 #17712

JasonFengJ9 commented Jun 29, 2023 •

edited

Loading

babsingh commented Jul 19, 2023

babsingh commented Jul 27, 2023

dmitripivkine commented Jul 28, 2023 •

edited

Loading

tajila commented Jul 31, 2023

babsingh commented Aug 25, 2023

babsingh commented Aug 30, 2023

JasonFengJ9 commented Sep 5, 2023

babsingh commented Sep 5, 2023

llxia commented Sep 5, 2023

babsingh commented Sep 5, 2023

babsingh commented Sep 29, 2023

LinHu2016 commented Sep 29, 2023

Results from the second fix

tajila commented Oct 5, 2023 •

edited

Loading

gacholio commented Oct 6, 2023

LinHu2016 commented Oct 10, 2023

tajila commented Oct 10, 2023

LinHu2016 commented Oct 11, 2023

tajila commented Oct 12, 2023

tajila commented Oct 12, 2023

tajila commented Oct 12, 2023

tajila commented Oct 12, 2023

dmitripivkine commented Oct 12, 2023 •

edited

Loading

LinHu2016 commented Oct 12, 2023 •

edited

Loading

LinHu2016 commented Oct 12, 2023

fengxue-IS commented Oct 30, 2023

JDK21 serviceability/jvmti/vthread/FollowReferences/VThreadStackRefTest.java Segmentation error vmState=0x00040000 #17712

JDK21 serviceability/jvmti/vthread/FollowReferences/VThreadStackRefTest.java Segmentation error vmState=0x00040000 #17712

Comments

JasonFengJ9 commented Jun 29, 2023 • edited Loading

Failure link

Optional info

Failure output (captured from console output)

babsingh commented Jul 19, 2023

[GDB] Stacktrace during the crash

babsingh commented Jul 27, 2023

dmitripivkine commented Jul 28, 2023 • edited Loading

tajila commented Jul 31, 2023

babsingh commented Aug 25, 2023

Test source code

Test output with OpenJ9 + #18020

Test output with the RI

babsingh commented Aug 30, 2023

JasonFengJ9 commented Sep 5, 2023

babsingh commented Sep 5, 2023

llxia commented Sep 5, 2023

babsingh commented Sep 5, 2023

babsingh commented Sep 29, 2023

LinHu2016 commented Sep 29, 2023

Results from the second fix

tajila commented Oct 5, 2023 • edited Loading

gacholio commented Oct 6, 2023

LinHu2016 commented Oct 10, 2023

tajila commented Oct 10, 2023

LinHu2016 commented Oct 11, 2023

tajila commented Oct 12, 2023

tajila commented Oct 12, 2023

tajila commented Oct 12, 2023

tajila commented Oct 12, 2023

dmitripivkine commented Oct 12, 2023 • edited Loading

LinHu2016 commented Oct 12, 2023 • edited Loading

LinHu2016 commented Oct 12, 2023

fengxue-IS commented Oct 30, 2023

JasonFengJ9 commented Jun 29, 2023 •

edited

Loading

dmitripivkine commented Jul 28, 2023 •

edited

Loading

tajila commented Oct 5, 2023 •

edited

Loading

dmitripivkine commented Oct 12, 2023 •

edited

Loading

LinHu2016 commented Oct 12, 2023 •

edited

Loading