Provide official API for commonly used svm.jar implementation details

[kristofdho]

Copied from #4783 (comment):

We have some usages that I think are not yet covered by #4783 (workarounds are very welcome):

1. com.oracle.svm.core.SubstrateOptions
   • Used to get the value of SubstrateOptions.Class, as we have multiple entrypoints within a module, and need to switch config based on which one is used.
2. com.oracle.svm.core.jdk.NativeLibrarySupport, com.oracle.svm.hosted.c.NativeLibraries and com.oracle.svm.core.jdk.JNIRegistrationUtil
   • Used to write library support in a similar way to JNIRegistrationJava.
   • The JNIRegistrationUtil superclass can be substituted by our own implementation, but without the other classes we have no way to do the static linking ourselves.
3. com.oracle.svm.core.jdk.Resources#registerResource(java.lang.String, java.lang.String, java.io.InputStream)
   • Used in our custom support for the fontmanager library, to add the fontconfig.bfc file to the resources, to be used from the substitution.
   • We can possibly work around this by loading the resource in a static byte array instead in our substitutions.
4. com.oracle.svm.hosted.FeatureImpl.DuringSetupAccessImpl#registerClassReachabilityListener
   • Used to add reachability handlers based on conditions not currently available. For us specifically a listener that fires for all classes with a specific annotation.
5. com.oracle.svm.hosted.classinitialization.ConfigurableClassInitialization
   • Hack to conditionally add resource bundles during analysis, used in the same way as here.
   • Any workaround suggestions to add simple resource-bundles on the fly would be much appreciated.

/cc @olpaw

[olpaw]

In addition to the above here is a list of -H: options used by Quarkus: quarkusio/quarkus#27784
kindly provided by @zakkak

[olpaw]

@kristofdho we discussed your suggestions:

• com.oracle.svm.core.SubstrateOptions

  • Used to get the value of SubstrateOptions.Class, as we have multiple entrypoints within a module, and need to switch config based on which one is used.

As mentioned before use e.g. -Dmy.main.class=<MainClass> to get entrypoint information into the Feature

• com.oracle.svm.core.jdk.NativeLibrarySupport, com.oracle.svm.hosted.c.NativeLibraries and com.oracle.svm.core.jdk.JNIRegistrationUtil

  • Used to write library support in a similar way to JNIRegistrationJava.
  • The JNIRegistrationUtil superclass can be substituted by our own implementation, but without the other classes we have no way to do the static linking ourselves.

Please copy whatever you need from there into your source base. We have no intention making theses helper methods public API.

• com.oracle.svm.core.jdk.Resources#registerResource(java.lang.String, java.lang.String, java.io.InputStream)

  • Used in our custom support for the fontmanager library, to add the fontconfig.bfc file to the resources, to be used from the substitution.
  • We can possibly work around this by loading the resource in a static byte array instead in our substitutions.

This is an interesting usecase.

• We need to add a new variant to the API (GR-40906):

org.graalvm.nativeimage.hosted.RuntimeResourceAccess#addResource(Module module, String resourcePath, byte[] resource)

that will allow you to do that. Thanks for bringing this up.

• com.oracle.svm.hosted.FeatureImpl.DuringSetupAccessImpl#registerClassReachabilityListener

  • Used to add reachability handlers based on conditions not currently available. For us specifically a listener that fires for all classes with a specific annotation.

There are

org.graalvm.nativeimage.hosted.Feature.BeforeAnalysisAccess#registerClassInitializerReachabilityHandler

and

org.graalvm.nativeimage.hosted.Feature.BeforeAnalysisAccess#registerReachabilityHandler
org.graalvm.nativeimage.hosted.Feature.BeforeAnalysisAccess#registerMethodOverrideReachabilityHandler
org.graalvm.nativeimage.hosted.Feature.BeforeAnalysisAccess#registerSubtypeReachabilityHandler

in the API. The only thing left is to add API that allows you to query annotations without triggering JDK internal annotation handling. We have that internally since com.oracle.svm.util.GuardedAnnotationAccess was added in June.

• Make GuardedAnnotationAccess accessible as public API (GR-40905)

• com.oracle.svm.hosted.classinitialization.ConfigurableClassInitialization

  • Hack to conditionally add resource bundles during analysis, used in the same way as here.
  • Any workaround suggestions to add simple resource-bundles on the fly would be much appreciated.

This needs to be fixed as well:

• Ensure org.graalvm.nativeimage.hosted.RuntimeResourceAccess#addResourceBundle API methods are usable during analysis (GR-40907)

[olpaw]

@zakkak from the reported -H: options we have identified the following to be added as proper @APIOptions:

• -H:IncludeLocales
• -H:BuildOutputJSONFile
• -H:+AddAllCharsets
• -H:NativeLinkerOption

They will be added soon (GR-40904).

For the other option-use you mentioned there are the following remarks:

• -H:-ParseOnce
  We recommend not using this option anymore.
• -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime
  What is the reasoning behind that? Did you measure that this is actually beneficial?
• -H:+AllowFoldMethods
  This should not be necessary. Please try to remove the option and report back if it causes problems.
• -H:FallbackThreshold=5
  This is equivalent to API option --auto-fallback
• -H:FallbackThreshold=0
  This is equivalent to API option --no-fallback
• -H:EnableURLProtocols
  This is equivalent to API option --enable-url-protocols
• -H:-InlineBeforeAnalysis
  This option should not be used anymore.
• -H:-SpawnIsolates
  Except for native-debugging on Windows this option should not be used anymore.
• -H:-StackTrace
  What is the reasoning behind that? Did you measure that this is actually beneficial?

[zakkak]

They will be added soon (GR-40904).

Thank you @olpaw

For the other option-use you mentioned there are the following remarks:

* `-H:-ParseOnce`
  We recommend not using this option anymore.

This is only used with GraalVM < 22.2 so there is no issue moving forward.

* `-H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime`
  What is the reasoning behind that? Did you measure that this is actually beneficial?

It seems like this was based on some past evaluation of the then options and needs to be re-evaluated. There are people working on it.

* `-H:+AllowFoldMethods`
  This should not be necessary. Please try to remove the option and report back if it causes problems.

@galderz it looks like this was introduced by you in quarkusio/quarkus#13376. Can you please have a look?

* `-H:FallbackThreshold=5`
  This is equivalent to API option `--auto-fallback`

* `-H:FallbackThreshold=0`
  This is equivalent to API option `--no-fallback`

* `-H:EnableURLProtocols`
  This is equivalent to API option `--enable-url-protocols`

The above have been replaced in quarkusio/quarkus#27783

* `-H:-InlineBeforeAnalysis`
  This option should not be used anymore.

The option adding this flag has been deprecated in quarkusio/quarkus#27783

* `-H:-SpawnIsolates`
  Except for native-debugging on Windows this option should not be used anymore.

This was introduced back in Dec 2018, when isolates where spawned by default(?). We should mark this as deprecated and eventually remove it. Users (and testers) wil still disable full stack traces using quarkus.native.additional-build-args.

* `-H:-StackTrace`
  What is the reasoning behind that? Did you measure that this is actually beneficial?

@Sanne this was introduced as an option in quarkusio/quarkus@8d16a42 do you remember why it was needed for?
In any case I believe we should deprecate the corresponding option and eventually delete it. Users will still be able to disable full stack traces using quarkus.native.additional-build-args.

[Sanne]

[Regarding -H:-StackTrace I've commented on https://github.com/quarkusio/quarkus/issues/27784#issuecomment-1241806173 ]

Unrelated to the -H options, I'm more concerned about the move of the annotations.

I'm understanding that many annotations are being moved from the svm artifact to the new API artifact - this is nice.

However it would seem that some of them have a different package; as we're currently building Quarkus with the old annotations (we have to until GraalVM releases this), this is compiling fine (no errors, no warnings), but then the new GraalVM version isn't going to apply these in practice. Examples: @NeverInline, @AlwaysInline, @Uninterruptible, @UnknownObjectField, ...

This concerns me as we can only observe the problem via obscure side-effects; could we maintain the existing annotations in the current package in addition to the new one, deprecate the old ones, and have the compiler take into account both flavours?

Alternatively, may I suggest to only keep them in the original package, mark them as deprecated (so to express they are still not meant as stable API) and address this problem in the future. Essentially postpone the move to a new package and treat that as a separate task to be addressed in a different version, that would keep the current migration simpler and give some more time to consider these implications.

[kristofdho]

@olpaw

• com.oracle.svm.core.jdk.NativeLibrarySupport, com.oracle.svm.hosted.c.NativeLibraries and com.oracle.svm.core.jdk.JNIRegistrationUtil
• • Used to write library support in a similar way to JNIRegistrationJava.
• • The JNIRegistrationUtil superclass can be substituted by our own implementation, but without the other classes we have no way to do the static linking ourselves.

Please copy whatever you need from there into your source base. We have no intention making theses helper methods public API.

I agree that we should just copy whatever util function that we use, but that only covers the usages of JNIRegistrationUtil.

We're still doing calls like:

PlatformNativeLibrarySupport.singleton().addBuiltinPkgNativePrefix("some_prefix");
NativeLibrarySupport.singleton().preregisterUninitializedBuiltinLibrary("library");
NativeLibraries nativeLibraries = ((FeatureImpl.BeforeAnalysisAccessImpl) a).getNativeLibraries();
nativeLibraries.addStaticJniLibrary("library");
nativeLibraries.addStaticNonJniLibrary("otherLibrary");
nativeLibraries.addDynamicNonJniLibrary("anotherLibrary");

---

• com.oracle.svm.hosted.FeatureImpl.DuringSetupAccessImpl#registerClassReachabilityListener

There are

org.graalvm.nativeimage.hosted.Feature.BeforeAnalysisAccess#registerClassInitializerReachabilityHandler

and

org.graalvm.nativeimage.hosted.Feature.BeforeAnalysisAccess#registerReachabilityHandler
org.graalvm.nativeimage.hosted.Feature.BeforeAnalysisAccess#registerMethodOverrideReachabilityHandler
org.graalvm.nativeimage.hosted.Feature.BeforeAnalysisAccess#registerSubtypeReachabilityHandler

I'm probably missing something here, but are you suggesting the functionality I need is already available here?
These methods always have some sort of matching criteria, what I'm requesting is a method that returns classes that have a certain annotation on them, eg:

void registerAnnotatedClassReachabilityHandler(BiConsumer<DuringAnalysisAccess, Class<?>> callback,
                                               Class<? extends Annotation> annotation);

So it can be used like:

registerAnnotatedClassReachabilityHandler(MyFeature::annotatedClassReachable, javax.inject.Singleton.class);

Possibly extend it to allow searching for any annotated element:

void registerAnnotatedElementReachabilityHandler(BiConsumer<DuringAnalysisAccess, java.lang.reflect.AnnotatedElement> callback,
                                                 Class<? extends Annotation> annotation,
                                                 java.lang.annotation.ElementType elementType); // elementType could be optional

I don't follow exactly how GuardedAnnotationAccess fits into that. Just to return the instance of the annotation, instead of calling Class#getAnnotation? I guess that could still be kept internal, if the callback includes the annotation instance.

[andrejusc]

Side question - could we get/generate concise API doc/html with versions and since/deprecated notion in them assuming that list of those -H options continues to grow/mutate?

[olpaw]

@kristofdho regarding:

Ensure org.graalvm.nativeimage.hosted.RuntimeResourceAccess#addResourceBundle API methods are usable during analysis (GR-40907)

Are you sure this does not work with the new RuntimeResourceAccess#addResourceBundle API method?

I tested the following:

Implemented DemoFeature that provides org.graalvm.nativeimage.hosted.Feature#duringAnalysis that uses

if (addBundle) {
   Module module = ModuleLayer.boot().findModule("jdk.compiler").orElseThrow();
   String bundleName = "com.sun.tools.javac.resources.javac";
   System.out.printf("Adding ResourceBundle %s:%s%n", module.getName(), bundleName);
   RuntimeResourceAccess.addResourceBundle(module, bundleName);
   System.out.printf("ResourceBundle added%n");
   addBundle = false;
}

Then in the Demo.main method I use:

Module compilerModule = ModuleLayer.boot().findModule("jdk.compiler").orElseThrow();
String bundleName = "com.sun.tools.javac.resources.javac";
ResourceBundle bundle = ResourceBundle.getBundle(bundleName, compilerModule);
System.out.println(bundle);
bundle.getKeys().asIterator().forEachRemaining(System.out::println);

This works as expected and gives me

com.sun.tools.javac.resources.javac@2d2802e4
javac.opt.Xlint.desc.synchronization
javac.opt.arg.key.equals.value
javac.opt.Xdoclint.subopts
javac.opt.g
javac.opt.Xlint.desc.unchecked
javac.msg.usage.header
javac.opt.d
javac.opt.X
javac.opt.sourcepath
....
javac.opt.Xlint.all
javac.opt.Xlint.desc.module
javac.opt.release
javac.opt.modulepath

at image runtime.