JEP 389: Foreign Linker API (Incubator) #11195

tajila · 2020-11-16T15:31:36Z

https://openjdk.java.net/jeps/389

Issues
Add NativeMethodHandle stubs - #11255
Add LibraryLookup native - #11260

tajila · 2020-11-16T15:32:01Z

Will most likely be targeted for JDK16 as an incubator release

pshipton · 2020-11-20T15:16:12Z

It's proposed to jdk16 now.

JasonFengJ9 · 2020-12-04T18:52:14Z

As per #11027 (comment), following are JEP389 tests:

java/foreign/stackwalk/TestStackWalk.java 
java/foreign/valist/VaListTest.java 
java/foreign/StdLibTest.java 
java/foreign/TestDowncall.java 
java/foreign/TestIllegalLink.java 
java/foreign/TestIntrinsics.java 
java/foreign/TestUpcall.java 
java/foreign/TestUpcallHighArity.java 
java/foreign/TestUpcallStubs.java 
java/foreign/TestVarArgs.java

JasonFengJ9 · 2020-12-04T18:59:12Z

From an internal build Test_openjdknext_j9_sanity.openjdk_x86-64_linux_Nightly/363:

22:36:37  --------------------------------------------------
22:36:37  TEST: java/foreign/stackwalk/TestStackWalk.java

22:36:37  java.lang.UnsatisfiedLinkError: sun/hotspot/WhiteBox.registerNatives()V
22:36:37  	at sun.hotspot.WhiteBox.<clinit>(WhiteBox.java:66)
22:36:37  	at TestStackWalk.<clinit>(TestStackWalk.java:66)
22:36:37  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
22:36:37  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78)
22:36:37  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
22:36:37  	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
22:36:37  	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
22:36:37  	at java.base/java.lang.Thread.run(Thread.java:853)
22:36:37  STATUS:Failed.`main' threw exception: java.lang.UnsatisfiedLinkError: sun/hotspot/WhiteBox.registerNatives()V

22:36:37  --------------------------------------------------
22:36:40  TEST: java/foreign/valist/VaListTest.java

22:36:40  Caused by: java.lang.ClassCastException: java.lang.invoke.DirectHandle incompatible with java.lang.invoke.BoundMethodHandle
22:36:40  	at java.base/java.lang.invoke.VarHandles.noCheckedExceptions(VarHandles.java:636)
22:36:40  	at java.base/java.lang.invoke.VarHandles.filterCoordinates(VarHandles.java:476)
22:36:40  	at java.base/java.lang.invoke.MethodHandleImpl$1.filterCoordinates(MethodHandleImpl.java:1790)
22:36:40  	at jdk.incubator.foreign/jdk.incubator.foreign.MemoryHandles.filterCoordinates(MemoryHandles.java:374)
22:36:40  	at jdk.incubator.foreign/jdk.internal.foreign.Utils.fixUpVarHandle(Utils.java:92)
22:36:40  	at jdk.incubator.foreign/jdk.internal.foreign.LayoutPath.dereferenceHandle(LayoutPath.java:159)
22:36:40  	at jdk.incubator.foreign/jdk.incubator.foreign.MemoryLayout.lambda$varHandle$3(MemoryLayout.java:411)
22:36:40  	at jdk.incubator.foreign/jdk.incubator.foreign.MemoryLayout$$Lambda$55/0x0000000000000000.apply(Unknown Source)
22:36:40  	at jdk.incubator.foreign/jdk.incubator.foreign.MemoryLayout.computePathOp(MemoryLayout.java:457)
22:36:40  	at jdk.incubator.foreign/jdk.incubator.foreign.MemoryLayout.varHandle(MemoryLayout.java:411)
22:36:40  	at jdk.incubator.foreign/jdk.internal.foreign.abi.x64.sysv.SysVVaList.<clinit>(SysVVaList.java:108)
22:36:40  	... 33 more
22:36:40  

22:36:40  TEST RESULT: Failed. Execution failed: `main' threw exception: org.testng.TestNGException: An error occurred while instantiating class VaListTest: null

22:36:40  --------------------------------------------------
22:36:40  TEST: java/foreign/StdLibTest.java

Similar exception as above

22:36:40  TEST RESULT: Failed. Execution failed: `main' threw exception: org.testng.TestNGException: Cannot instantiate class StdLibTest

Similar exception as above
22:37:02  ===============================================
22:37:02  java/foreign/TestDowncall.java
22:37:02  Total tests run: 12097, Failures: 12097, Skips: 0
22:37:02  ===============================================

Similar exception as above
22:37:06  ===============================================
22:37:06  java/foreign/TestIllegalLink.java
22:37:06  Total tests run: 9, Failures: 9, Skips: 0
22:37:06  ===============================================

22:37:06  --------------------------------------------------
22:37:06  TEST: java/foreign/TestIntrinsics.java

Similar exception as above
22:37:06  TEST RESULT: Failed. Execution failed: `main' threw exception: java.lang.Exception: failures: 1

Similar exception as above
22:37:26  ===============================================
22:37:26  java/foreign/TestUpcall.java
22:37:26  Total tests run: 12097, Failures: 0, Skips: 12097
22:37:26  Configuration Failures: 1, Skips: 1
22:37:26  ===============================================

22:37:26  --------------------------------------------------
22:37:26  TEST: java/foreign/TestUpcallHighArity.java

Similar exception as above

22:37:26  TEST RESULT: Failed. Execution failed: `main' threw exception: org.testng.TestNGException: An error occurred while instantiating class TestUpcallHighArity: null

Similar exception as above
22:37:26  ===============================================
22:37:26  java/foreign/TestUpcallStubs.java
22:37:26  Total tests run: 3, Failures: 3, Skips: 0
22:37:26  ===============================================

22:37:26  --------------------------------------------------
22:37:26  TEST: java/foreign/TestVarArgs.java

Similar exception as above

22:37:26  TEST RESULT: Failed. Execution failed: `main' threw exception: org.testng.TestNGException: An error occurred while instantiating class TestVarArgs: null

These tests have been excluded via adoptium/aqa-tests#2088

babsingh · 2021-01-21T17:20:05Z

Below is the updated list of JEP389 test failures after ibmruntimes/openj9-openjdk-jdk16#9 is merged for the 0.25 release:

java/foreign/TestNative.java
java/foreign/stackwalk/TestStackWalk.java
java/foreign/valist/VaListTest.java
java/foreign/StdLibTest.java
java/foreign/TestDowncall.java
java/foreign/TestIntrinsics.java
java/foreign/TestUpcall.java
java/foreign/TestUpcallHighArity.java
java/foreign/TestUpcallStubs.java
java/foreign/TestVarArgs.java

ChengJin01 · 2021-03-19T20:17:13Z

Hi @dnakamura,

I notice these UnsatisfiedLinkError related failures above were caused by the missing libraries in the test suites or in the build which are never and should be generated during the compilation via CMake & UMA.

https://github.com/ibmruntimes/openj9-openjdk-jdk16/tree/openj9/test/jdk/java/foreign
https://github.com/ibmruntimes/openj9-openjdk-jdk16/blob/openj9/test/jdk/java/foreign
stackwalk/libStackWalk.c
valist/libVaList.c
libIntrinsics.c
libLookupTest.c
libNativeAccess.c
libTestDowncall.c
libTestUpcall.c
libTestUpcallHighArity.c
libVarArgs.c

I am wondering what scripts/setting we should add/update on OpenJDK/OpenJ9 side to generate these libraries (at least via CMake) to get these test suites working as expected.

pshipton · 2021-03-19T20:25:48Z

We put OpenJDK test libraries into the test-images artifact. Did you make these libraries available when running tests?
i.e. the OpenJ9 nightly build last night https://ci.eclipse.org/openj9/job/Build_JDK16_x86-64_linux_Nightly/44/, see the test-images.tar.gz artifact. https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Build_JDK16_x86-64_linux_Nightly/44/test-images.tar.gz

pshipton · 2021-03-19T20:29:11Z

If you are doing your own build, look in the build//images/test directory. If it's not there, try running make test.

pshipton · 2021-03-19T20:31:46Z

I see OpenJDK nightly testing using an argument like -nativepath:"/home/jenkins/workspace/Test_openjdk16_j9_sanity.openjdk_ppc64_aix_Nightly/openjdkbinary/openjdk-test-image/jdk/jtreg/native"

ChengJin01 · 2021-03-19T20:42:42Z

I see OpenJDK nightly testing using an argument like -nativepath:"/home/jenkins/workspace/Test_openjdk16_j9_sanity.openjdk_ppc64_aix_Nightly/openjdkbinary/openjdk-test-image/jdk/jtreg/native"

I tried to export the option to OpenJ9 but ended up with failure due to the unrecognised option when compiling one of these tests.

export OPENJ9_JAVA_OPTIONS="-nativepath:\"/test-images/test-images/jdk/jtreg/native\""
JVMJ9VM007E Command-line option unrecognised: -nativepath:/test-images/test-images/jdk/jtreg/native

pshipton · 2021-03-19T20:52:36Z

It's not a VM option, it's a test option. Here is the entire command line used by the nightly build.

23:53:45  "/home/jenkins/workspace/Test_openjdk16_j9_sanity.openjdk_ppc64_aix_Nightly/openjdkbinary/j2sdk-image/bin/java" -Xmx512m -jar "/home/jenkins/workspace/Test_openjdk16_j9_sanity.openjdk_ppc64_aix_Nightly/openjdk-tests/TKG/../../jvmtest/openjdk/jtreg/lib/jtreg.jar" \
23:53:45  -agentvm -a -ea -esa -v:fail,error,time,nopass -retain:fail,error,*.dmp,javacore.*,heapdump.*,*.trc -ignore:quiet -timeoutFactor:8 -xml:verify -concurrency:1 -nativepath:"/home/jenkins/workspace/Test_openjdk16_j9_sanity.openjdk_ppc64_aix_Nightly/openjdkbinary/openjdk-test-image/jdk/jtreg/native" -vmoptions:"-Xmx512m  -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -XX:+UseCompressedOops " \
23:53:45  -timeoutHandler:jtreg.openj9.CoreDumpTimeoutHandler -timeoutHandlerDir:"/home/jenkins/workspace/Test_openjdk16_j9_sanity.openjdk_ppc64_aix_Nightly/openjdk-tests/TKG/../TKG/lib/openj9jtregtimeouthandler.jar" \
23:53:45  -w ""/home/jenkins/workspace/Test_openjdk16_j9_sanity.openjdk_ppc64_aix_Nightly/openjdk-tests/TKG/../TKG/output_16161210798203/jdk_lang_native_0"/work" \
23:53:45  -r "/home/jenkins/workspace/Test_openjdk16_j9_sanity.openjdk_ppc64_aix_Nightly/openjdk-tests/TKG/../../jvmtest/openjdk/report" \
23:53:45  -jdk:"/home/jenkins/workspace/Test_openjdk16_j9_sanity.openjdk_ppc64_aix_Nightly/openjdkbinary/j2sdk-image" \
23:53:45  "/home/jenkins/workspace/Test_openjdk16_j9_sanity.openjdk_ppc64_aix_Nightly/openjdk-tests/TKG/../openjdk/openjdk-jdk/test/jdk/jdk/internal/loader/NativeLibraries/Main.java"; \

pshipton · 2021-03-19T20:58:16Z

I assume you are running via TKG. There is probably some doc somewhere about where to put the test image, and TKG will just find and use it.

Try extracting the tar file so it's at the same level as the JVM, and rename test-images to openjdk-test-image

<some dir>
    <jvm dir>
    openjdk-test-image

pshipton · 2021-03-19T21:00:27Z

Or try running the test via a grinder, does it have the same problem? If not, then look at what the grinder did.

ChengJin01 · 2021-03-19T21:24:23Z

There are two issues with the missing libraries when manually running the jtreg tests via TKG with openjdk-test-image, one of which is UnsatisfiedLinkError that come from the java test case invoking the native code e.g.

/openj9-openjdk-jdk16/test/jdk/java/foreign/TestNative.java

    static {
        System.loadLibrary("NativeAccess"); <------ 
    }
    
Caused by: java.lang.UnsatisfiedLinkError: Can't load NativeAccess
	at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:1816)
	at java.base/java.lang.System.loadLibrary(System.java:602)
	at TestNative.<clinit>(TestNative.java:217)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:78)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
	at org.testng.internal.ObjectFactoryImpl.newInstance(ObjectFactoryImpl.java:29)
	at org.testng.internal.ClassHelper.createInstance1(ClassHelper.java:382)
	... 23 more

The other one is failure to locate the library in test code. e.g.

/openj9-openjdk-jdk16/test/jdk/java/foreign/TestDowncall.java
public class TestDowncall extends CallGeneratorHelper {

    static LibraryLookup lib = LibraryLookup.ofLibrary("TestDowncall"); <----- 
    static CLinker abi = CLinker.getInstance();

Caused by: java.lang.IllegalArgumentException: Library not found: TestDowncall
	at jdk.incubator.foreign/jdk.internal.foreign.LibrariesHelper.lookup(LibrariesHelper.java:94)
	at jdk.incubator.foreign/jdk.internal.foreign.LibrariesHelper.loadLibrary(LibrariesHelper.java:60)
	at jdk.incubator.foreign/jdk.incubator.foreign.LibraryLookup.ofLibrary(LibraryLookup.java:150)
	at TestDowncall.<clinit>(TestDowncall.java:69)
	... 30 more

against the Spec at https://download.java.net/java/early_access/jdk16/docs/api/jdk.incubator.foreign/jdk/incubator/foreign/LibraryLookup.html#ofLibrary(java.lang.String)

 the resulting name is then looked up in the standard native library path 
(which can be overriden, by setting the java.library.path property).

Probably need to add settings to ensure they are reachable in the test code (e.g. setting up java.library.path for LibraryLookup.ofLibrary).

pshipton · 2021-03-19T21:30:57Z

Did you try the grinder? If the grinder doesn't work then perhaps the foreign tests aren't setup correctly to use the native test libraries. Both those libraries are in the test image.

pshipton · 2021-03-19T21:34:17Z

I do see $(JDK_NATIVE_OPTIONS) referenced in the jdk_foreign_native playlist. Also I expect these tests run and pass on Hotspot.
https://github.com/AdoptOpenJDK/openjdk-tests/blob/master/openjdk/playlist.xml#L951

Likely you aren't setting up the environment correctly.

pshipton · 2021-03-19T21:40:29Z

Look at your TKG output for a -nativepath option, and confirm the directory being used by TKG is where you put the test images.

pshipton · 2021-03-19T22:17:27Z

Figured it out, you need to export TESTIMAGE_PATH to the test image directory, and then -nativepath is used when running the tests.

babsingh · 2021-04-05T18:52:12Z

java/foreign/TestNative.java NoSuchMethodError

Initial investigation: VarHandle.get is a VarHandle MH/VH polymorphic method. The below stack suggests that invokedynamic is invoked on VarHandle.get, which should never happen. For MH/VH polymorphic methods, the correct bytecode is invokevirtual which gets rewritten to invokehandle. Here, a Lambda is also involved. For Lambdas, OJDK code dynamically generates the bytecodes. Below, invokedynamic on VarHandle.get is called from Lambda generated bytecodes. We will need to investigate why Lambda generated bytecodes are calling invokedynamic on VarHandle.get.

Lambda bytecode generation: https://github.com/ibmruntimes/openj9-openjdk-jdk16/blob/openj9/src/java.base/share/classes/java/lang/invoke/InnerClassLambdaMetafactory.java

Java property flag to dump Lambda bytecodes: -Djdk.internal.lambda.dumpProxyClasses=<DIR>.

test TestNative.testNativeAccess(TestNative$$Lambda$108/0x00000000e1e70ee0@9414ac56, TestNative$$Lambda$101/0x00000000e1e70460@15d3e380, [100:b8]): failure
java.lang.NoSuchMethodError: java/lang/invoke/VarHandle.get(Ljdk/incubator/foreign/MemorySegment;Ljava/lang/Long;)Ljava/lang/Object; (loaded from jrt:/java.base by <Bootstrap Loader>) called from class TestNative (loaded from file:/.../openjdk-tests/TKG/output_16171331559140/jdk_custom_0/work/classes/0/java/foreign/TestNative.d/ by jdk.internal.loader.ClassLoaders$AppClassLoader@3144ebb).
	at java.base/java.lang.invoke.MethodHandleResolver.getCPMethodHandleAt(Native Method)
	at java.base/java.lang.invoke.MethodHandleResolver.getAdditionalBsmArg(MethodHandleResolver.java:382)
	at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:181)
	at TestNative.lambda$nativeAccessOps$17(TestNative.java:243)
	at TestNative$$Lambda$108/0x00000000e1e70ee0.accept(Unknown Source)
	at TestNative.testNativeAccess(TestNative.java:159)

ChengJin01 · 2021-04-05T18:52:22Z

The foundational reason to exclude TestIllegalLink.java is that it expects MemoryAddress.NULL rather than LibraryLookup.Symbol as Adddressable parameter in downcallHandle() which is incorrect according to the Spec. In addition, we have similar test cases to cover all these illegal situations in our own test suites.

If we choose to include TestIllegalLink.java, we can just simply change the error messages set up in the test to OpenJ9-specific messages implemented in our prototype.

babsingh · 2021-04-05T19:11:11Z

java/foreign/stackwalk/TestStackWalk.java (should be excluded)

https://github.com/ibmruntimes/openj9-openjdk-jdk16/blob/501ea92242cffe194f9aa41d2ffb8f7be0951b0c/test/jdk/java/foreign/stackwalk/TestStackWalk.java#L108

I think sun.hotspot.WhiteBox.verifyFrames(boolean log) is identical to swalk.c::walkStackFrames. verifyFrames is only used in the second phase of the test when armed is true. The first phase of the test is not dependent on verifyFrames. We can potentially update the test to work with OpenJ9.

TestStackWalk.java also depends on CLinker.upcallStub, which has not been implemented yet.

java.lang.UnsatisfiedLinkError: sun/hotspot/WhiteBox.registerNatives()V
	at sun.hotspot.WhiteBox.<clinit>(WhiteBox.java:66)
	at TestStackWalk.<clinit>(TestStackWalk.java:66)

ChengJin01 · 2021-04-05T19:55:43Z

TestStackWalk.java is literally specific to the the implementation of upcallStub() as it simply prepares the downcall handle for the upcall. We will see whether we need to walk the stackframes in implementing the prototype; otherwise, it should be excluded or something similar can be added to our own cases to verify our solution instead of replying on this jtreg test.

babsingh · 2021-04-05T20:02:53Z

For CLinker.downcallHandle, Addressable parameter must be LibraryLookup.Symbol rather than MemoryAddress according to Spec

@ChengJin01 I was going through your spec references, and I didn't see an explicit note that LibraryLookup.Symbol is expected for CLinker.downcallHandle's Addressable parameter. If that was the case, then CLinker.downcallHandle should explicitly expect LibraryLookup.Symbol as the first parameter not Addressable. After looking at the Javadoc and RI, I feel that CLinker.downcallHandle should work with any implementation of Addressable:

interface Symbol extends Addressable
interface MemoryAddress extends Addressable

https://download.java.net/java/early_access/jdk16/docs/api/jdk.incubator.foreign/jdk/incubator/foreign/CLinker.html#downcallHandle(jdk.incubator.foreign.Addressable,java.lang.invoke.MethodType,jdk.incubator.foreign.FunctionDescriptor)

https://download.java.net/java/early_access/jdk16/docs/api/jdk.incubator.foreign/jdk/incubator/foreign/LibraryLookup.html#ofLibrary(java.lang.String)

ChengJin01 · 2021-04-05T20:15:17Z

@babsingh, the links above are outdated.

Looking at the Spec at
http://cr.openjdk.java.net/~mcimadamore/8254231_v3/javadoc/jdk/incubator/foreign/CLinker.html#downcallHandle(jdk.incubator.foreign.Addressable,java.lang.invoke.MethodType,jdk.incubator.foreign.FunctionDescriptor)

MethodHandle downcallHandle(Addressable symbol,
 MethodType type,
 FunctionDescriptor function)
Obtain a foreign method handle, with given type, which can be used to call a target foreign function at a given address and featuring a given function descriptor.
Parameters:
symbol - downcall symbol. <---------
type - the method type.
function - the function descriptor.

If you look at all jtreg tests specific to Clinker.downcallHandle(), none of them accepts MemoryAddress but LibraryLookup.Symbol.
e.g.

test/jdk/java/foreign/TestDowncall.java
   public void testDowncall(String fName, Ret ret, List<ParamType> paramTypes, List<StructFieldType> fields) throws Throwable {
        List<Consumer<Object>> checks = new ArrayList<>();
        List<MemorySegment> segments = new ArrayList<>();
        LibraryLookup.Symbol addr = lib.lookup(fName).get(); <-----------
        MethodHandle mh = abi.downcallHandle(addr, <----------- addr is the LibraryLookup.Symbol from lib.lookup(fName).get()
methodType(ret, paramTypes, fields), function(ret, paramTypes, fields));

which is explained at http://cr.openjdk.java.net/~mcimadamore/8254231_v3/javadoc/jdk/incubator/foreign/LibraryLookup.html

Package jdk.incubator.foreign
Interface LibraryLookup
public interface LibraryLookup

A native library lookup. Exposes a lookup operation for searching symbols, 
see lookup(String). A given native library remains loaded as long as there 
is at least one live library lookup instance referring to it. 

All symbol instances (see LibraryLookup.Symbol) generated by a given library 
lookup object contain a strong reference to said lookup object, 
therefore preventing library unloading; in turn method handle instances obtained 
from CLinker.downcallHandle(Addressable, MethodType, FunctionDescriptor)) 
also maintain a strong reference to the addressable parameter used for their construction. 

This means that there is always a strong reachability chain from a native method handle 
to a lookup object (the one that was used to lookup the native library symbol 
the method handle refers to); this is useful to prevent situations where a native 
library is unloaded in the middle of a native call.

The symbol is the key to CLinker.downcallHandle() to keep the strong reference to the native library to
prevent it from being unloaded during the downcall while a MemoryAddress (simply wrapping up a memory address) can't achieve this.

babsingh · 2021-04-05T20:59:06Z

The symbol is the key to CLinker.downcallHandle() to keep the strong reference to the native library to
prevent it from being unloaded during the downcall while a MemoryAddress (simply wrap up a memory address) can't achieve this.

The old and new links have identical content. I still feel the same. @tajila Can you provide a second opinion?

I don't think having a strong reference is a requirement for CLinker.downcallHandle. The doc says that you will get strong reachability and protection against unloading if LibraryLookup.Symbol is used as the Addressable parameter. But, it does not say that CLinker.downcallHandle won't accept a MemoryAddress as the Addressable parameter. Otherwise, CLinker.downcallHandle should explicitly ask for a LibraryLookup.Symbol as the first parameter instead of an Addressable.

If MemoryAddress is used as the Addressable parameter, then as a user, I would still expect CLinker.downcallHandle to return a NativeMethodHandle, which will point to the address (NativeEntryPoint). If the address is invalid, reachability is compromised or the library is unloaded after the NativeMethodHandle is created, then an error should be thrown while invoking the NativeMethodHandle.

As per http://cr.openjdk.java.net/~mcimadamore/8254231_v3/javadoc/jdk/incubator/foreign/LibraryLookup.html, you can also have a standalone MemoryAddress (pointer to a MemorySegment) with strong reachability out of a lookup symbol.

In cases where a client wants to create a memory segment out of a lookup symbol, the client might want to attach the lookup symbol to the newly created segment, so that the symbol will be kept reachable as long as the memory segment is reachable; this can be achieved by creating the segment using the MemoryAddress.asSegmentRestricted(long, Runnable, Object) restricted segment factory

ChengJin01 · 2021-04-05T21:06:21Z

As per http://cr.openjdk.java.net/~mcimadamore/8254231_v3/javadoc/jdk/incubator/foreign/LibraryLookup.html, you can also have a standalone MemoryAddress (pointer to a MemorySegment) with strong reachability out of a lookup symbol.

This doesn't work (already tried before) especially on Windows and randomly crashed on Linux (at very low frequency) due to the invalid address (the library was already unloaded and there is no way to validate the address until the native function is called and no way stop them during the call).

...this can be achieved by creating the segment using the MemoryAddress.asSegmentRestricted(long, Runnable, Object) restricted segment factory

The prerequisite for this is that you have to know the Object (which is the Symbol) for asSegmentRestricted(), which is up to user to do that rather than in our code as there is no way to know the Symobl of the passed-in MemoryAddress in CLinker.downcallHandle().

ChengJin01 · 2021-04-05T21:13:44Z

I don't think having a strong reference is a requirement for CLinker.downcallHandle. The doc says that you will get strong reachability and protection against unloading if LibraryLookup.Symbol is used as the Addressable parameter. But, it does not say that CLinker.downcallHandle won't accept a MemoryAddress as the Addressable parameter. Otherwise, CLinker.downcallHandle should explicitly ask for a LibraryLookup.Symbol as the first parameter instead of an Addressable.

If MemoryAddress is used as the Addressable parameter, then as a user, I would still expect CLinker.downcallHandle to return a NativeMethodHandle, which will point to the address (NativeEntryPoint). If the address is invalid, reachability is compromised or the library is unloaded after the NativeMethodHandle is created, then an error should be thrown while invoking the NativeMethodHandle.

As I mentioned above, simply passing a memory address of the native function over to CLinker.downcallHandle() will end up with unexpected behaviour because there is no guarantee whether the address itself is valid or not and it is useless to hold strong reference to the MemoryAddress parameter as the corresponding library is highly likely unloaded prior to the call.

ChengJin01 · 2021-04-05T23:57:47Z

I just checked a simple test on both Windows & Linux with my latest updated code (simply keeps holding the passed-in addressable parameter in addition to getting the raw long value of the corresponding function address and nothing else changed in there)

public class xxxInvoker {
	private final Addressable functionAddr;     <--------
		xxxInvoker(Addressable addr, ...) {
		functionAddr = addr;  <---- hold the passed-in Addressable parameter in code during the downcall

and it is quit interesting that the problem with MemoryAddress disappeared without any error (tried many times but nothing wrong happened) in which case I assume it has been resolved in this way.
e.g.
a symbol is passed to the downcallHandle():

		Symbol functionSymbol = LibraryLookup.ofLibrary("clinkerffitests").lookup("foo").get();
		MethodHandle mh =  CLinker.getInstance().downcallHandle()(functionSymbol, mt, fd); <-----
               mh.invokeExact(...)

or the memory address of the symbol is passed to the downcallHandle():

		Symbol functionSymbol = LibraryLookup.ofLibrary("clinkerffitests").lookup("foo").get();
		MemoryAddress addr = functionSymbol.address();  <-------
		MethodHandle mh = clinker.downcallHandle(addr, mt, fd); <-----
                mh.invokeExact(...);

Both of tests above work well whether the passed-in addressable parameter is a memory address or not. Will add tests with MemoryAddress as parameters to our own test suites and keep an eye on this issue in case it reoccurred in the future.

pshipton · 2022-02-10T19:37:35Z

Closing as it's for jdk16.

tajila added project:panama Used to track Project Panama related work Epic labels Nov 16, 2020

pshipton added the JEP label Nov 16, 2020

pshipton added the jdk16 label Nov 24, 2020

pshipton added this to the Release 0.25 (Java 16) milestone Nov 24, 2020

pshipton changed the title ~~JEP 389: Foreign Linker API~~ JEP 389: Foreign Linker API (Incubator) Nov 24, 2020

babsingh mentioned this issue Jan 21, 2021

openjdk jdk16 java/foreign/TestNative.java ClassCastException: java.lang.invoke.DirectHandle incompatible with java.lang.invoke.BoundMethodHandle #11724

Closed

pshipton modified the milestones: Release 0.25 (Java 16), Release 0.26 (Java 8, 11, 16) Apr refresh Jan 29, 2021

tajila modified the milestones: Release 0.26 (Java 8, 11, 16) Apr refresh, Release 0.27 (Java 8, 11, 16) July refresh Mar 4, 2021

This was referenced Apr 12, 2021

Implement the functionalities of Foreign Linker API #12412

Closed

JEP389 Foreign Linker API: DownCall (Phase 1 / Primitive support) #12413

Merged

tajila added comp:vm comp:jit labels Jun 15, 2021

tajila modified the milestones: Release 0.27 (Java 8, 11, 16) July refresh, Release 0.28 (Java 17) Jun 15, 2021

tajila modified the milestones: Release 0.28 (Java 17), Release 0.29 (Java 8, 11, 17) Oct refresh Aug 6, 2021

tajila modified the milestones: Release 0.29 (Java 8, 11, 17) Oct refresh, Release 0.30 (Java 8, 11, 17) Jan refresh Sep 21, 2021

tajila modified the milestones: Release 0.30 (Java 8, 11, 17) Jan refresh, Release 0.31 (Java 18) Nov 23, 2021

tajila modified the milestones: Release 0.31 (Java 18), Release 0.32 (Java 8, 11, 17, 18) Apr refresh Jan 18, 2022

pshipton modified the milestones: Release 0.32 (Java 8, 11, 17, 18) Apr refresh, Tracking Feb 10, 2022

pshipton closed this as completed Feb 10, 2022

pshipton removed this from the Backlog milestone Feb 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JEP 389: Foreign Linker API (Incubator) #11195

JEP 389: Foreign Linker API (Incubator) #11195

tajila commented Nov 16, 2020 •

edited

Loading

tajila commented Nov 16, 2020

pshipton commented Nov 20, 2020

JasonFengJ9 commented Dec 4, 2020

JasonFengJ9 commented Dec 4, 2020

babsingh commented Jan 21, 2021

ChengJin01 commented Mar 19, 2021 •

edited

Loading

pshipton commented Mar 19, 2021

pshipton commented Mar 19, 2021

pshipton commented Mar 19, 2021

ChengJin01 commented Mar 19, 2021 •

edited

Loading

pshipton commented Mar 19, 2021

pshipton commented Mar 19, 2021 •

edited

Loading

pshipton commented Mar 19, 2021

ChengJin01 commented Mar 19, 2021 •

edited

Loading

pshipton commented Mar 19, 2021 •

edited

Loading

pshipton commented Mar 19, 2021 •

edited

Loading

pshipton commented Mar 19, 2021

pshipton commented Mar 19, 2021

babsingh commented Apr 5, 2021

ChengJin01 commented Apr 5, 2021 •

edited

Loading

babsingh commented Apr 5, 2021

ChengJin01 commented Apr 5, 2021 •

edited

Loading

babsingh commented Apr 5, 2021 •

edited

Loading

ChengJin01 commented Apr 5, 2021 •

edited

Loading

babsingh commented Apr 5, 2021 •

edited

Loading

ChengJin01 commented Apr 5, 2021 •

edited

Loading

ChengJin01 commented Apr 5, 2021 •

edited

Loading

ChengJin01 commented Apr 5, 2021 •

edited

Loading

pshipton commented Feb 10, 2022

JEP 389: Foreign Linker API (Incubator) #11195

JEP 389: Foreign Linker API (Incubator) #11195

Comments

tajila commented Nov 16, 2020 • edited Loading

tajila commented Nov 16, 2020

pshipton commented Nov 20, 2020

JasonFengJ9 commented Dec 4, 2020

JasonFengJ9 commented Dec 4, 2020

babsingh commented Jan 21, 2021

ChengJin01 commented Mar 19, 2021 • edited Loading

pshipton commented Mar 19, 2021

pshipton commented Mar 19, 2021

pshipton commented Mar 19, 2021

ChengJin01 commented Mar 19, 2021 • edited Loading

pshipton commented Mar 19, 2021

pshipton commented Mar 19, 2021 • edited Loading

pshipton commented Mar 19, 2021

ChengJin01 commented Mar 19, 2021 • edited Loading

pshipton commented Mar 19, 2021 • edited Loading

pshipton commented Mar 19, 2021 • edited Loading

pshipton commented Mar 19, 2021

pshipton commented Mar 19, 2021

babsingh commented Apr 5, 2021

ChengJin01 commented Apr 5, 2021 • edited Loading

babsingh commented Apr 5, 2021

ChengJin01 commented Apr 5, 2021 • edited Loading

babsingh commented Apr 5, 2021 • edited Loading

ChengJin01 commented Apr 5, 2021 • edited Loading

babsingh commented Apr 5, 2021 • edited Loading

ChengJin01 commented Apr 5, 2021 • edited Loading

ChengJin01 commented Apr 5, 2021 • edited Loading

ChengJin01 commented Apr 5, 2021 • edited Loading

pshipton commented Feb 10, 2022

tajila commented Nov 16, 2020 •

edited

Loading

ChengJin01 commented Mar 19, 2021 •

edited

Loading

ChengJin01 commented Mar 19, 2021 •

edited

Loading

pshipton commented Mar 19, 2021 •

edited

Loading

ChengJin01 commented Mar 19, 2021 •

edited

Loading

pshipton commented Mar 19, 2021 •

edited

Loading

pshipton commented Mar 19, 2021 •

edited

Loading

ChengJin01 commented Apr 5, 2021 •

edited

Loading

ChengJin01 commented Apr 5, 2021 •

edited

Loading

babsingh commented Apr 5, 2021 •

edited

Loading

ChengJin01 commented Apr 5, 2021 •

edited

Loading

babsingh commented Apr 5, 2021 •

edited

Loading

ChengJin01 commented Apr 5, 2021 •

edited

Loading

ChengJin01 commented Apr 5, 2021 •

edited

Loading

ChengJin01 commented Apr 5, 2021 •

edited

Loading