Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JDK21 serviceability_jvmti_j9_0 Fatal error: Error in ResumeThread & get_method_class_name: error in JVMTI GetMethodDeclaringClass #17865

Closed
JasonFengJ9 opened this issue Jul 27, 2023 · 7 comments
Closed

JDK21 serviceability_jvmti_j9_0 Fatal error: Error in ResumeThread & get_method_class_name: error in JVMTI GetMethodDeclaringClass #17865

JasonFengJ9 opened this issue Jul 27, 2023 · 7 comments
Assignees
@babsingh
Labels
comp:jvmti comp:vm jdk21 project:loom Used to track Project Loom related work test failure
Milestone
Java 21 (0.42)

Comments

@JasonFengJ9
Copy link
Member

JasonFengJ9 commented Jul 27, 2023 •
edited
Loading

Failure link

From an internal build(paix911):

08:08:05  openjdk version "21-internal" 2023-09-19
08:08:05  OpenJDK Runtime Environment (build 21-internal-adhoc.jenkins.BuildJDK21ppc64aixPersonal)
08:08:05  Eclipse OpenJ9 VM (build master-c9a98669642, JRE 21 AIX ppc64-64-Bit Compressed References 20230726_28 (JIT enabled, AOT enabled)
08:08:05  OpenJ9   - c9a98669642
08:08:05  OMR      - 436bad3c438
08:08:05  JCL      - 78015256cbd based on jdk-21+31)

Rerun in Grinder - Change TARGET to run only the failed test targets.

Optional info

Failure output (captured from console output)

08:12:32  variation: Mode150
08:12:32  JVM_OPTIONS:  -XX:+UseCompressedOops 

08:17:58  TEST: serviceability/jvmti/stress/StackTrace/Suspended/GetStackTraceSuspendedStressTest.java

08:17:58  STDERR:
08:17:58  
08:17:58  Fatal error: Error in ResumeThread
08:17:58  
08:17:58  Fatal error: get_method_class_name: error in JVMTI GetMethodDeclaringClass

08:17:58  TEST RESULT: Failed. Unexpected exit from test [exit code: 87]
08:17:58  --------------------------------------------------
08:22:08  Test results: passed: 122; failed: 1
08:22:08  Report written to /home/jenkins/workspace/Test_openjdk21_j9_extended.openjdk_ppc64_aix_Personal/jvmtest/openjdk/report/html/report.html
08:22:08  Results written to /home/jenkins/workspace/Test_openjdk21_j9_extended.openjdk_ppc64_aix_Personal/aqa-tests/TKG/output_16903735147351/serviceability_jvmti_j9_0/work
08:22:08  Error: Some tests failed or other problems occurred.
08:22:08  -----------------------------------
08:22:08  serviceability_jvmti_j9_0_FAILED

50x internal grinder - 11/50 failing rate

FYI @babsingh
@JasonFengJ9 JasonFengJ9 added this to the Java 21 milestone Jul 27, 2023
This was referenced Jul 27, 2023
Closed
Closed
Closed
@JasonFengJ9
Copy link
Member Author

JDK21 ppc64_aix(paix824)

15:50:15  variation: Mode150
15:50:15  JVM_OPTIONS:  -XX:+UseCompressedOops 

15:56:07  TEST: serviceability/jvmti/stress/StackTrace/Suspended/GetStackTraceSuspendedStressTest.java

15:56:07  STDERR:
15:56:07  
15:56:07  Fatal error: Error in ResumeThread
15:56:07  
15:56:07  Fatal error: get_method_class_name: error in JVMTI GetMethodDeclaringClass

16:00:32  serviceability_jvmti_j9_0_FAILED

@fengxue-IS fengxue-IS added the project:loom Used to track Project Loom related work label Sep 16, 2023
@JasonFengJ9
Copy link
Member Author

JDK21 ppc64_aix(paix826)

21:07:54  TEST: serviceability/jvmti/stress/StackTrace/Suspended/GetStackTraceSuspendedStressTest.java

21:07:54  STDERR:
21:07:54  
21:07:54  Fatal error: Error in ResumeThread
21:07:54  
21:07:54  Fatal error: get_method_class_name: error in JVMTI GetMethodDeclaringClass

21:13:18  serviceability_jvmti_j9_0_FAILED

@babsingh
Copy link
Contributor

Moving to the 0.44 release since the failure is highly intermittent.

@JasonFengJ9
Copy link
Member Author

JasonFengJ9 commented Oct 27, 2023 •
edited
Loading

JDK21 ppc64_aix(paix816)

java version "21.0.1-beta" 2023-10-17
IBM Semeru Runtime Certified Edition 21.0.1+12-202310261948 (build 21.0.1-beta+12-202310261948)
Eclipse OpenJ9 VM 21.0.1+12-202310261948 (build master-39fd615b3, JRE 21 AIX ppc64-64-Bit Compressed References 20231026_22 (JIT enabled, AOT enabled)
OpenJ9   - 39fd615b3
OMR      - 071c0c25b
JCL      - c06eaf638 based on jdk-21.0.1+12)

[2023-10-26T21:39:03.456Z] variation: Mode650
[2023-10-26T21:39:03.456Z] JVM_OPTIONS:  -XX:-UseCompressedOops 

[2023-10-26T21:45:25.093Z] TEST: serviceability/jvmti/stress/StackTrace/Suspended/GetStackTraceSuspendedStressTest.java

[2023-10-26T21:45:25.097Z] STDERR:
[2023-10-26T21:45:25.097Z] 
[2023-10-26T21:45:25.097Z] Fatal error: Error in ResumeThread
[2023-10-26T21:45:25.097Z] 
[2023-10-26T21:45:25.097Z] Fatal error: get_method_class_name: error in JVMTI GetMethodDeclaringClass

[2023-10-26T21:50:51.190Z] serviceability_jvmti_j9_1_FAILED

JDK21 ppc64le_linux

@pshipton
Copy link
Member

Having two failures in a test cycle doesn't seem to meet the definition of highly intermittent. Moving back to the Java 21 milestone while we consider this.

babsingh added a commit to babsingh/openj9 that referenced this issue Nov 8, 2023
@babsingh
Re-fetch objects after VM access is released and reacquired
cb4f50b 
Stale objects can manifest many issues. In this case, stale objects
cause complex and intermittent synchronization issues.

Related: eclipse-openj9#17865
Related: eclipse-openj9#17869
Related: eclipse-openj9#18370

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
This was referenced Nov 8, 2023
Merged
Merged
babsingh added a commit to babsingh/openj9 that referenced this issue Nov 9, 2023
@babsingh
Re-fetch objects after VM access is released and reacquired
a1cdacf 
Stale objects can manifest many issues. In this case, stale objects
cause complex and intermittent synchronization issues.

Related: eclipse-openj9#17865
Related: eclipse-openj9#17869
Related: eclipse-openj9#18370

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
@babsingh babsingh mentioned this issue Nov 10, 2023
Merged
@JasonFengJ9
Copy link
Member Author

JDK21 aarch64_linux(ubu20-aarch64-2)

[2023-11-11T17:24:55.887Z] TEST: serviceability/jvmti/stress/StackTrace/Suspended/GetStackTraceSuspendedStressTest.java

[2023-11-11T17:24:55.889Z] STDERR:
[2023-11-11T17:24:55.889Z] 
[2023-11-11T17:24:55.889Z] Fatal error: Error in ResumeThread
[2023-11-11T17:24:55.889Z] 
[2023-11-11T17:24:55.889Z] Fatal error: jvmti_common get_carrier_thread: Error in JVMTI extension GetCarrierThread

@babsingh babsingh mentioned this issue Nov 13, 2023
Closed
babsingh added a commit to babsingh/openj9 that referenced this issue Nov 13, 2023
@babsingh
Spin during VirtualThread MountBegin and UnmountBegin
feac356 
Currently, the halt flag is set in VirtualThread MountEnd if a virtual
thread is suspended via JVMTI, and in VirtualThread UnmountEnd if a
carrier thread is suspended via JVMTI.

In the above approach, the halt flag is set too late. As soon as the
continuation swaps the J9VMThread context, the thread begins execution
and is capable of triggering JVMTI events.

To avoid the above issue, the above steps are moved into VirtualThread
MountBegin and UnmountBegin. This prevents the continuation to swap
the J9VMThread context.

Currently, the halt flag is set without invoking
exitVThreadTransitionCritical. This prevents JVMTI to resume the halted
thread and cause a hang. The new approach spins, invokes
exitVThreadTransitionCritical  and releases VM access to allow JVMTI to
resume the suspended thread.

The better approach will be to fail mount if the thread is suspended
and retry later. Currently, his approach cannot be implemented because
VirtualThread.java does not support this approach.

Related: eclipse-openj9#17865
Related: eclipse-openj9#17869
Related: eclipse-openj9#18370

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
babsingh added a commit to babsingh/openj9 that referenced this issue Nov 14, 2023
@babsingh
Update the fully mounted check in JVMTI Suspend/Resume Thread
1cd50dd 
During JVMTI suspend/resume thread operations, the current check to
determine if a thread is fully mounted is flaky. It depends on
J9VMThread->theadObject which might change after getVMThread due to
Java level changes, which are accepted as-is from the RI.

The check to determine if a thread is fully mounted is moved into
getVMThread. The check is updated to use the continuation state, which
is more stable since the Continuation implementation is maintained
within OpenJ9.

Related: eclipse-openj9#17865
Related: eclipse-openj9#17869
Related: eclipse-openj9#18370

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
@babsingh babsingh mentioned this issue Nov 15, 2023
Merged
babsingh added a commit to babsingh/openj9 that referenced this issue Nov 16, 2023
@babsingh @fengxue-IS
Add JVMTI synchronization in JVM_VirtualThreadHideFrames
8a2b61e 
J9VMThread->threadObject can be modified between
JVM_VirtualThreadHideFrames's first invocation with hide=true and
second invocation with hide=false. Synchronization to prevent JVMTI
operations is acquired between these two invocations. This prevents
JVMTI functions to see an unstable J9VMThread->threadObject.

Related: eclipse-openj9#17865
Related: eclipse-openj9#17869
Related: eclipse-openj9#18370

Co-authored-by: Jack Lu <Jack.S.Lu@ibm.com>
Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
@babsingh
Copy link
Contributor

Fixed via

babsingh added a commit to babsingh/openj9 that referenced this issue Nov 20, 2023
@babsingh @fengxue-IS
Add JVMTI synchronization in JVM_VirtualThreadHideFrames
38907fd 
J9VMThread->threadObject can be modified between
JVM_VirtualThreadHideFrames's first invocation with hide=true and
second invocation with hide=false. Synchronization to prevent JVMTI
operations is acquired between these two invocations. This prevents
JVMTI functions to see an unstable J9VMThread->threadObject.

Related: eclipse-openj9#17865
Related: eclipse-openj9#17869
Related: eclipse-openj9#18370

Co-authored-by: Jack Lu <Jack.S.Lu@ibm.com>
Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
babsingh added a commit to babsingh/openj9 that referenced this issue Nov 20, 2023
@babsingh @fengxue-IS
Add JVMTI synchronization in JVM_VirtualThreadHideFrames
ecdbc7b 
J9VMThread->threadObject can be modified between
JVM_VirtualThreadHideFrames's first invocation with hide=true and
second invocation with hide=false. Synchronization to prevent JVMTI
operations is acquired between these two invocations. This prevents
JVMTI functions to see an unstable J9VMThread->threadObject.

Related: eclipse-openj9#17865
Related: eclipse-openj9#17869
Related: eclipse-openj9#18370

Co-authored-by: Jack Lu <Jack.S.Lu@ibm.com>
Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@babsingh babsingh

Labels
comp:jvmti comp:vm jdk21 project:loom Used to track Project Loom related work test failure
Projects
None yet
Milestone
Java 21 (0.42)
Development

No branches or pull requests
4 participants
@babsingh @fengxue-IS @JasonFengJ9 @pshipton