[GR-40463] Add initial support for JMX to Native Image #4786

roberttoyonaga · 2022-08-03T13:29:04Z

Related issue: #4732

Currently, a native image JMX server can be started and a java client can connect. Methods can be invoked on managed beans remotely. The server will start automatically if JMX is specified at executable runtime (ie. -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=9999). JMX should work with password authentication and SSL for both client and server.

To build a native image as a JMX server, use the option --enable-monitoring=jmxserver (or --enable-monitoring=jmxserver,jvmstat to also export JMX connector address to instrumentation buffer so that it can be discovered and read by other JVMs on the same system).
To build as a JMX client, use --enable-monitoring=jmxclient

...vm.core/src/com/oracle/svm/core/jdk/management/Target_jdk_internal_agent_FileSystemImpl.java

...ratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/jdk/JMXInitializationFeature.java

christianhaeubl

Thanks for contributing. I added a few comments.

...tratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/jdk/management/JmxServerFeature.java

...ratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/jdk/management/ManagementSupport.java

christianhaeubl · 2022-09-01T11:49:11Z

...tratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/jdk/management/JmxServerFeature.java

+ * This feature provides the start up hook required to initialize the JMX server.
+ * This feature is required by com.oracle.svm.hosted.jdk.JmxServerFeature.
+ */
+public class JmxServerFeature implements Feature{


The goal is that this works in the same way as the existing management features, such as JFR or jvmstat. For that reason, please annotate all your features with @AutomaticFeature and implement isInConfiguration(...).

The existing management features can be enabled selectively via the API option --enable-monitoring (see VMInspectionOptions). For your new functionality, you can add new values to that option. There were changes in that area recently, so please look at the latest version that is available in master.

Ok both the jmx client and server features are now @AutomaticFeature and can be included using --enable-monitoring=jmxclient,jmxserver

christianhaeubl · 2022-09-01T11:50:27Z

...re/src/com/oracle/svm/core/jdk/management/Target_java_lang_management_ManagementFactory.java

-
-    @Substitute
-    private static <T extends PlatformManagedObject> T getPlatformMXBean(MBeanServerConnection connection, Class<T> mxbeanInterface) throws java.io.IOException {
-        return null;


I think that those substitutions are still needed if the JMX support is not included into Native Image.

I added them back and also created a new BooleanSupplier: JmxClientNotIncluded. If the JmxClientFeature is not included, then those methods will remain substituted. Is this a good approach? I also did a similar thing for JMXConnectorFactory.connect method.

christianhaeubl · 2022-09-01T12:42:40Z

.../com.oracle.svm.core/src/com/oracle/svm/core/jdk/management/Target_sun_rmi_transport_GC.java

+@TargetClass(className = "sun.rmi.transport.GC")
+final class Target_sun_rmi_transport_GC {
+    // Work around that essentially prevents supplemental GCs on RMI client.
+    // Required because JVM_MaxObjectInspectionAge is not implemented in SVM. Maybe there is a better way?


JVM_MaxObjectInspectionAge should be easy to implement for the serial GC:

add a field that holds the timestamp of the last full garbage collection to GCImpl.

set this field when a full collection finishes (see GCImpl.doCollectOnce)

add a new method to the classes Heap and HeapImpl so that it is possible to query the new GCImpl field.

add a substitution for JVM_MaxObjectInspectionAge that calls the new Heap method.

I will take care of the G1-specific implementation once this PR is ready for integration.

I implemented your suggestion. However, I am not sure how to substitute JVM_MaxObjectInspectionAge because it is a C function that JNI native method Java_sun_rmi_transport_GC_maxObjectInspectionAge calls. So I have just changed the existing sun.rmi.transport.GC.maxObjectInspectionAge java method substitution. Is this what you meant?

christianhaeubl · 2022-09-01T12:56:49Z

substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/jdk/JmxServerFeature.java

+
+    private static void configureSerialization(BeforeAnalysisAccess access) {
+        String[] classes = new String[] {
+                "[B", "com.oracle.svm.core.jdk.UnsupportedFeatureError",


I suppose this configuration is JDK-specific?

@vjovanov, @christianwimmer : is there any better way to do that? There is quite a lot of configuration in this PR.

Yes it is JDK specific

If a feature requires that much configuration, it is better to use config-*.json files. Combined with https://www.graalvm.org/22.0/reference-manual/native-image/Reflection/#conditional-configuration it should be possible to make sure that the config data is only applied if JMX support is requested.

If a feature requires that much configuration, it is better to use config-*.json files.

I just realized that unfortunately that does not work in this case because this is not an external Feature thus the META-INF/native-image based approach cannot be used. Sorry for the noise!

At the moment I don't see a better way to do this. For the build-time init future programming model changes will help in the future. For reachability metadata, we would need both programming model changes and the JDK modifications.

substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/jdk/JmxServerFeature.java

substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/jdk/JmxClientFeature.java

fniephaus · 2022-11-14T10:50:16Z

...oracle.svm.core/src/com/oracle/svm/core/jdk/management/Target_jdk_internal_vm_VMSupport.java

+
+    @Substitute
+    private static Properties initAgentProperties(Properties properties) {
+        JavaMainSupport support = ImageSingletons.lookup(JavaMainSupport.class);


It seems this is causing a build to fail: https://github.com/oracle/graal/actions/runs/3441384844/jobs/5776652403#step:13:1356

Maybe this should use ManagementFactory.getRuntimeMXBean().getInputArguments()?

Oh I see! Thanks! I think SubstrateRuntimeMXBean.getInputArguments also uses com.oracle.svm.core.JavaMainWrapper$JavaMainSupport, but it does a check first to see if it's actually available.

Right, and you get exactly a SubstrateRuntimeMXBean when you use ManagementFactory.getRuntimeMXBean(). 😉

fniephaus · 2022-11-14T15:53:52Z

Running the internal gates against e00d0ec now. Would be great to add some tests for this in a follow up PR.

fniephaus · 2022-11-14T21:06:46Z

When building the native-image driver or the native-image-configure tool, I see:

The following classes have unspecified initialization policy:
com.sun.jmx.remote.util.EnvHelp
com.sun.jmx.mbeanserver.Introspector
java.beans.Introspector
com.sun.jmx.mbeanserver.JavaBeansAccessor
com.sun.jmx.mbeanserver.MXBeanLookup
Error: To fix the error either specify the initialization policy for given classes or set -H:-AssertInitializationSpecifiedForAllClasses
Error: Use -H:+ReportExceptionStackTraces to print stacktrace of underlying exception

If you could work on a fix for that, that'd be great. I'm going to look into implementing getMaxObjectInspectionAge() for the G1 GC.

roberttoyonaga · 2022-11-14T21:12:56Z

com.sun.jmx.mbeanserver.Introspector
Ok no problem! These errors are with jdk 17 right?

fniephaus · 2022-11-14T21:15:32Z

These errors are with jdk 17 right?

Yes, that's correct.

roberttoyonaga · 2022-11-14T21:32:37Z

When building the native-image driver or the native-image-configure tool:
I'm pretty sure I know what the problem is, but I can't seem to reproduce those errors.

fniephaus · 2022-11-14T21:38:33Z

Try building with:

$ cd substratevm
$ mx --dy /vm --native-images=lib:native-image-agent --extra-image-builder-argument=-H:+AssertInitializationSpecifiedForAllClasses build

This gives me a few more classes:

The following classes have unspecified initialization policy:
com.sun.jmx.mbeanserver.MBeanInstantiator
com.sun.jmx.remote.security.FileLoginModule
com.sun.jmx.remote.internal.ArrayNotificationBuffer
java.beans.Introspector
com.sun.jmx.mbeanserver.Introspector
com.sun.jmx.remote.security.HashedPasswordManager
com.sun.jmx.remote.internal.ServerNotifForwarder
com.sun.jmx.remote.util.EnvHelp
com.sun.jmx.mbeanserver.MXBeanLookup
com.sun.jmx.defaults.JmxProperties
com.sun.jmx.remote.security.JMXPluggableAuthenticator
com.sun.jmx.remote.security.JMXSubjectDomainCombiner
com.sun.jmx.remote.internal.ServerCommunicatorAdmin
com.sun.jmx.mbeanserver.JavaBeansAccessor

roberttoyonaga · 2022-11-14T22:20:14Z

mx --dy /vm --native-images=lib:native-image-agent --extra-image-builder-argument=-H:+AssertInitializationSpecifiedForAllClasses build

Strangely, building with this and jdk 17, I was still unable to reproduce the error complaining about the unspecified init policies. But I have specified policies for them anyway and pushed the changes.

fniephaus · 2022-11-16T10:26:04Z

When I build GraalPy, I see the following problem popping up:

Error: Detected a string in the image heap that contains a user directory. This means that file system information from the native image build is persisted and available at image runtime, which is most likely an error.
String that is problematic: sdk/mxbuild/linux-amd64/GRAALVM_7CA0CDF709_JAVA17_STAGE1/graalvm-7ca0cdf709-java17-23.0.0-dev/conf/management/jmxremote.password
Disallowed substring with user directory: sdk/mxbuild/linux-amd64/GRAALVM_7CA0CDF709_JAVA17_STAGE1/graalvm-7ca0cdf709-java17-23.0.0-dev
This check can be disabled using the option -H:-DetectUserDirectoriesInImageHeap
Detailed message:
Trace: Object was reached by
  trying to constant fold static field com.sun.jmx.remote.security.FileLoginModule.DEFAULT_PASSWORD_FILE_NAME
    at com.sun.jmx.remote.security.FileLoginModule.initialize(FileLoginModule.java:197)
  parsing method com.sun.jmx.remote.security.FileLoginModule.initialize(FileLoginModule.java:174) reachable via the parsing context
    at javax.security.auth.login.LoginContext.invoke(LoginContext.java:745)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:679)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:677)
    at com.oracle.svm.core.jdk.Target_java_security_AccessController.executePrivileged(SecuritySubstitutions.java:147)
    at java.security.AccessController.doPrivileged(AccessController.java:569)
    at com.oracle.svm.core.containers.CgroupUtil.readAllLinesPrivileged(CgroupUtil.java:72)
    at com.oracle.svm.core.containers.CgroupSubsystemFactory.determineType(CgroupSubsystemFactory.java:121)
    at com.oracle.svm.core.containers.CgroupSubsystemFactory.create(CgroupSubsystemFactory.java:50)
    at com.oracle.svm.core.containers.CgroupMetrics.getInstance(CgroupMetrics.java:164)

#5400 may actually fix this already but I still have to verify that.

Nonetheless, I also found more classes with an unspecified initialization policy:

The following classes have unspecified initialization policy:
com.sun.jmx.remote.security.JMXPluggableAuthenticator$FileLoginConfig

com.sun.jmx.mbeanserver.Introspector
com.sun.jmx.mbeanserver.JavaBeansAccessor
com.sun.jmx.mbeanserver.MXBeanLookup
com.sun.jmx.remote.util.EnvHelp
java.beans.Introspector
java.rmi.dgc.VMID
java.rmi.MarshalledObject$MarshalledObjectInputStream
java.rmi.server.LogStream
java.rmi.server.ObjID
java.rmi.server.RemoteServer
java.rmi.server.RMIClassLoader
java.rmi.server.UID
jdk.management.jfr.MBeanUtils
jdk.management.jfr.SettingDescriptorInfo
sun.rmi.registry.RegistryImpl
sun.rmi.runtime.Log
sun.rmi.runtime.Log$LoggerLog
sun.rmi.runtime.Log$LogStreamLog
sun.rmi.runtime.NewThreadAction
sun.rmi.runtime.RuntimeUtil
sun.rmi.server.LoaderHandler
sun.rmi.server.MarshalInputStream
sun.rmi.server.UnicastRef
sun.rmi.server.UnicastRef2
sun.rmi.server.UnicastServerRef
sun.rmi.server.Util
sun.rmi.transport.ConnectionInputStream
sun.rmi.transport.DGCAckHandler
sun.rmi.transport.DGCClient
sun.rmi.transport.DGCImpl
sun.rmi.transport.GC
sun.rmi.transport.ObjectTable
sun.rmi.transport.tcp.TCPChannel
sun.rmi.transport.tcp.TCPEndpoint
sun.rmi.transport.tcp.TCPTransport
sun.rmi.transport.Transport

@roberttoyonaga could you please take care of the missing initialization policies while I try and fix the GraalPy and a couple of other issues?

substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/jdk/JmxClientFeature.java

roberttoyonaga · 2022-11-16T17:00:21Z

When I build GraalPy, I see the following problem popping up:

@fniephaus It seems like the problem is that, in the JDK code, the pathFileLoginModule.DEFAULT_PASSWORD_FILE_NAME is computed and stored as a static final member. I think this would cause FileLoginModule.DEFAULT_PASSWORD_FILE_NAME to get initialized at build time (due to me specifying build time init for class FileLoginModule), which ends up storing this path computed in hosted mode in the image heap. I changed the initialization policy to runtime, which I think should fix the problem.

Nonetheless, I also found more classes with an unspecified initialization policy:

All of the classes listed there should have already had their initialization policies specified in JmxServerFeature and JmxClientFeature (for jdk 17+). Is it possible any of the tests are using jdk version <17?

I've now updated things such that those classes have their policies specified regardless of jdk version used to build with.

roberttoyonaga · 2022-11-16T17:15:35Z

Running the internal gates against e00d0ec now. Would be great to add some tests for this in a follow up PR.

I added some tests for JMX locally. Should I include them as part of the PR or is it better to still separate them into a following pull request?

fniephaus · 2022-11-16T21:36:52Z

Is it possible any of the tests are using jdk version <17?

Yes, we still run some gates against JDK11, although we are winding them down as we speak.

I added some tests for JMX locally. Should I include them as part of the PR or is it better to still separate them into a following pull request?

Tests are always welcome! 🙂

roberttoyonaga · 2022-11-17T14:10:36Z

Yes, we still run some gates against JDK11, although we are winding them down as we speak.

Ok the initialization policies should now be specified for jdk 11 too.

Tests are always welcome! slightly_smiling_face

Ok I included some tests! :)

fniephaus · 2022-11-18T12:02:31Z

I changed the initialization policy to runtime, which I think should fix the problem.

Thanks! Now, I only see this again:

The following classes have unspecified initialization policy:
com.sun.jmx.remote.security.JMXPluggableAuthenticator$FileLoginConfig

I'm going to fix this. Our internal CE gate seem to pass, I'm looking into fixes for our EE tests.

...c/com.oracle.svm.core/src/com/oracle/svm/core/jdk/management/ManagementAgentStartupHook.java

...ratevm/src/com.oracle.svm.core.genscavenge/src/com/oracle/svm/core/genscavenge/HeapImpl.java

Co-authored-by: Fabio Niephaus <fabio.niephaus@oracle.com>

fniephaus · 2023-03-03T07:12:19Z

This was merged via #5596. 🎉

oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Aug 3, 2022

roberttoyonaga marked this pull request as draft August 3, 2022 13:29

roberttoyonaga mentioned this pull request Aug 3, 2022

[GR-40463] Add experimental support for remote management over JMX to Native Image #4732

Closed

roberttoyonaga commented Aug 3, 2022

View reviewed changes

...vm.core/src/com/oracle/svm/core/jdk/management/Target_jdk_internal_agent_FileSystemImpl.java Outdated Show resolved Hide resolved

roberttoyonaga commented Aug 3, 2022

View reviewed changes

...ratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/jdk/JMXInitializationFeature.java Outdated Show resolved Hide resolved

fniephaus linked an issue Aug 3, 2022 that may be closed by this pull request

[GR-40463] Add experimental support for remote management over JMX to Native Image #4732

Closed

roberttoyonaga marked this pull request as ready for review August 9, 2022 19:51

fniephaus requested a review from christianhaeubl August 17, 2022 07:19

fniephaus changed the title ~~Add initial support for JMX in native images~~ [GR-40463] Add initial support for JMX to Native Image Aug 17, 2022

fniephaus self-assigned this Aug 17, 2022

christianhaeubl reviewed Sep 1, 2022

View reviewed changes

christianhaeubl requested a review from christianwimmer September 4, 2022 09:18

roberttoyonaga requested review from christianhaeubl and removed request for christianwimmer September 6, 2022 20:26

fniephaus added this to the Future milestone Sep 9, 2022

fniephaus assigned roberttoyonaga and unassigned fniephaus Sep 9, 2022

fniephaus reviewed Nov 14, 2022

View reviewed changes

fniephaus self-assigned this Nov 14, 2022

fniephaus modified the milestones: Future, 23.0.0 Release (January 24, 2023) Nov 14, 2022

fniephaus reviewed Nov 16, 2022

View reviewed changes

substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/jdk/JmxClientFeature.java Outdated Show resolved Hide resolved

fniephaus reviewed Dec 7, 2022

View reviewed changes

...c/com.oracle.svm.core/src/com/oracle/svm/core/jdk/management/ManagementAgentStartupHook.java Outdated Show resolved Hide resolved

fniephaus reviewed Dec 7, 2022

View reviewed changes

...ratevm/src/com.oracle.svm.core.genscavenge/src/com/oracle/svm/core/genscavenge/HeapImpl.java Outdated Show resolved Hide resolved

graalvmbot mentioned this pull request Dec 12, 2022

[GR-40463] Add experimental support for JMX to Native Image. #5596

Merged

fniephaus removed a link to an issue Dec 12, 2022

[GR-40463] Add experimental support for remote management over JMX to Native Image #4732

Closed

fniephaus linked an issue Dec 12, 2022 that may be closed by this pull request

[GR-40463] Add experimental support for remote management over JMX to Native Image #4732

Closed

roberttoyonaga and others added 2 commits December 12, 2022 17:00

Add initial support for JMX to Native Image.

3b1ef93

Co-authored-by: Fabio Niephaus <fabio.niephaus@oracle.com>

Set monitoring features explicitly for launchers.

23684af

oracle-contributor-agreement bot added OCA Required At least one contributor does not have an approved Oracle Contributor Agreement. and removed OCA Verified All contributors have signed the Oracle Contributor Agreement. labels Dec 12, 2022

roberttoyonaga force-pushed the rjmx branch from e01c60c to ba12ae1 Compare December 12, 2022 18:49

oracle-contributor-agreement bot added OCA Verified All contributors have signed the Oracle Contributor Agreement. and removed OCA Required At least one contributor does not have an approved Oracle Contributor Agreement. labels Dec 12, 2022

roberttoyonaga added 2 commits December 12, 2022 15:22

factor out common code into third feature

04281d8

add initialization policies for FlightRecorder if JFR is enabled.

6da807a

roberttoyonaga force-pushed the rjmx branch from 289e85a to 6da807a Compare December 12, 2022 22:01

roberttoyonaga added 2 commits December 12, 2022 20:18

add an additional initialization policy. Comment for clarity.

a03b957

fix typo

ddb6238

fniephaus closed this Mar 3, 2023

[GR-40463] Add initial support for JMX to Native Image #4786

[GR-40463] Add initial support for JMX to Native Image #4786

Uh oh!

Conversation

roberttoyonaga commented Aug 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

christianhaeubl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

roberttoyonaga Sep 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

olpaw Nov 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fniephaus commented Nov 14, 2022

Uh oh!

fniephaus commented Nov 14, 2022

Uh oh!

roberttoyonaga commented Nov 14, 2022

Uh oh!

fniephaus commented Nov 14, 2022

Uh oh!

roberttoyonaga commented Nov 14, 2022

Uh oh!

fniephaus commented Nov 14, 2022

Uh oh!

roberttoyonaga commented Nov 14, 2022

Uh oh!

fniephaus commented Nov 16, 2022

Uh oh!

Uh oh!

roberttoyonaga commented Nov 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

roberttoyonaga commented Nov 16, 2022

Uh oh!

fniephaus commented Nov 16, 2022

Uh oh!

roberttoyonaga commented Nov 17, 2022

Uh oh!

fniephaus commented Nov 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

roberttoyonaga commented Aug 3, 2022 •

edited

Loading

roberttoyonaga Sep 6, 2022 •

edited

Loading

olpaw Nov 14, 2022 •

edited

Loading

roberttoyonaga commented Nov 16, 2022 •

edited

Loading

fniephaus commented Nov 18, 2022 •

edited

Loading