[API Proposal]: Provide an API to get the native module handle of the native process entry-point module

[jkoritzinsky]

Background and motivation

As mentioned in #7267, there is a strong desire for enabling using the "__Internal" name in a DllImport to point to methods in the hosting executable module of the process in both CoreCLR and Mono. Mono has had this support for many years, but CoreCLR doesn't have built-in support, and we don't have a desire to provide built-in support for this "hidden" name.

However, we understand the need for a way to resolve this name to the correct handle. We propose adding an API that gets the native module handle of the entry-point module.

API Proposal

namespace System.Runtime.InteropServices
{
     public static class NativeLibrary
     {
          public static IntPtr GetMainProgramHandle();
     }
}

Alternatively, we considered allowing null as a parameter to NativeLibrary.Load(string) to match how the implementation on non-Windows is dlopen(NULL). The interop team decided that design was an abstraction leak and undesirable since the Windows implementation doesn't follow the same model. Additionally, the other NativeLibrary.Load overloads have additional behaviors and aren't thin wrappers, which makes the null option less desirable.

API Usage

NativeLibrary.SetDllImportResolver(assembly, (name, _, paths) =>
{
     if (name == "__Internal")
     {
          return NativeLibrary.GetMainProgramHandle();
     }
     return IntPtr.Zero;
});

Risks

None

[jkoritzinsky]

cc: @AaronRobinsonMSFT @elinor-fung

[AaronRobinsonMSFT]

/cc @marek-safar @lambdageek

[GrabYourPitchforks]

Implementation question: how does this method affect the handle's reference count?

[jkoritzinsky]

This method will cache the handle after it's been called once, so it will only increase the refcount of the handle by 1. On Windows, we'll use GetModuleHandle which doesn't increase the refcount at all.

Since the module is the process's entry-point module, we don't need to worry about releasing the handle since it is valid until process tear-down (which will release the handle anyway).

[am11]

Alternatively, we considered allowing null as a parameter to NativeLibrary.Load(string) to match how the implementation on non-Windows is dlopen(NULL).

Another approach is to return (IntPtr)-1 which works on Windows and Unix alike.

runtime/src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/Marshal.CoreCLR.cs

Line 250 in 57bfe47

return (IntPtr)(-1);

[lambdageek]

Alternatively, we considered allowing null as a parameter to NativeLibrary.Load(string) to match how the implementation on non-Windows is dlopen(NULL).

nit: this doesn't work on every unix. In particular on Android dlopen (NULL) doesn't work, but dlsym (RTLD_DEFAULT) does.

[bartonjs]

Looks good as proposed.

namespace System.Runtime.InteropServices
{
     public static class NativeLibrary
     {
          public static IntPtr GetEntryPointModuleHandle();
     }
}