Mono does not implement coreclr_create_delegate hosting API or ComponentActivator

[jkoritzinsky]

Update the hosting API has been added to Mono. This issue is mainly open now to track the remaining work needed in order to run the DllImportGenerator tests on mobile platforms (or indeed other apps that may want the desktop hosting API on mobile). See #59815 (comment) for one path forward.

---

Mono does not implement the coreclr_create_delegate hosting API method nor the ComponentActivator class used by the nethost library. This is blocking us from running the DllImportGenerator.Tests suite, since we use DNNE to enable easier managed->native testing in the libraries suite by implementing the "native" side primarily in managed code.

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[lambdageek]

@jkoritzinsky This CompnentActivator ? What's DNNE - I'm not familiar with the acronym.

[jkoritzinsky]

Yes, that is the ComponentActivator I'm referring to. DNNE is DotNetNativeExports, an project that @AaronRobinsonMSFT started that makes exposing managed code as native exports much simpler (and has significantly streamlined our ability to test the DllImportGenerator source generator).

[lambdageek]

@jkoritzinsky What's a good way to get an overview of how the DllImportGenerator testing is designed? You're telling me some pieces and I'm having a hard time putting the whole thing together in my head.

@steveisok I think we need to have a discussion about how this testing will look on mobile.

[AaronRobinsonMSFT]

What's a good way to get an overview of how the DllImportGenerator testing is designed?

@lambdageek The design is pretty simple. The tests are XUnit based and have a series of P/Invoke signatures. The DNNE package generates native platform library that represents specific exports from the managed assembly. When the test P/Invokes into the generated native library it does two things: (1) verifies the runtime is loaded and (2) finds the associated managed function that was indicated to be exported. Loading the runtime is straight forward enough and mono supports that without issue. However, discovery of the managed function by returning a function pointer to that function isn't. That is typically done via coreclr_create_delegate. The basic scenario is similar to what is documented at https://docs.microsoft.com/dotnet/core/tutorials/netcore-hosting.

DNNE's implementation can be found at https://github.com/AaronRobinsonMSFT/DNNE/blob/master/src/platform/platform.c.

[lambdageek]

The DNNE package generates native platform library that represents specific exports from the managed assembly.

ok, I think I understand so far.

In C# you write a P/Invoke declaration. It's body is some boilerplate that ultimately calls back to C#. DNNE generates that boilerplate. Then from a C# xunit test you invoke the P/Invoke, that calls into the generated native library which invokes a managed method and the managed method records the test success in some way?

When the test P/Invokes into the generated native library it does two things: (1) verifies the runtime is loaded and (2) finds the associated managed function that was indicated to be exported.

I'm unclear on what (1) means. Isn't this axiomatically true: the runtime is loaded because the test is running.
Maybe I'm misunderstanding what DNNE platform.c is doing - is it really initializing a new runtime? (in a separate process?) or it just needs to get some handle that it can then use to get the function pointer to the managed method?

I'm slightly unclear on what (2) means: essentially you want to get back to managed land from native. One way to have done that was to use something like GetFunctionPointerForDelegate which would require passing the function pointer from managed to native. But an alternate approach is to call coreclr_create_delegate directory and get the pointer from the runtime. Is that what's going on?

---

The bigger issue is that Mono does not have corehost (and hence the corefxr entrypoints) on mobile. But it doesn't seem like that's an essential part of this scheme - you could use the coreclr_initialize, coreclr_create_delegate functions directly, right?

[AaronRobinsonMSFT]

DNNE itself is typically used for providing native exports of managed assemblies. The generated native library can be consumed naturally from a native application and the fact that the underlying implementation is managed is an implementation detail that is hidden.

The usage of DNNE in this case is to avoid having any explicit native assets to test the DllImport source generator. Testing the DllImport source generator then becomes entirely managed with the the thin native veneer useful to properly consume the test target via P/Invoke.

For (1), yes in this scenario the runtime will already be initialized and that is handled opaquely by hostfxr so DNNE and the test harness don't care. For (2) that is part of the veneer and an implementation detail of DNNE that the test relies upon. We could have written this all in native code but then we would have to introduce CMake and all that nonsense so we avoid that by using DNNE.

But an alternate approach is to call coreclr_create_delegate directory and get the pointer from the runtime. Is that what's going on?

That isn't the alternative approach that is how DNNE works - imagine a world where a managed assembly wants to provide the implementation to native and only project a native API. The way to do that is via DNNE and a small bit of C code that hides the runtime activation and managed to function pointer conversion away.

[lambdageek]

Thanks for the clarifications @jkoritzinsky @AaronRobinsonMSFT.

For (1), yes in this scenario the runtime will already be initialized and that is handled opaquely by hostfxr so DNNE and the test harness don't care. For (2) that is part of the veneer and an implementation detail of DNNE that the test relies upon. We could have written this all in native code but then we would have to introduce CMake and all that nonsense so we avoid that by using DNNE.

Ok, great. So I understood what's going on ;-)

But an alternate approach is to call coreclr_create_delegate directory and get the pointer from the runtime. Is that what's going on?

That isn't the alternative approach that is how DNNE works - imagine a world where a managed assembly wants to provide the implementation to native and only project a native API. The way to do that is via DNNE and a small bit of C code that hides the runtime activation and managed to function pointer conversion away.

I didn't mean that it's an alternative for DNNE. Just trying to make sure I understand what the tests are meant to do.

---

So I think in summary to run on mobile where dynamic linking is allowed we would need to:

1. Run dnne-gen
2. Compile the resulting code with the compiler toolchain for the target plaform (Android NDK, or the Xcode SDK for ios)
3. Either use the real hostfxr and a mono-mobile-specific hostpolicy or a stub standalone hostfxr that cuts through all the indirection and just calls the mono coreclr_XXX functions directly.
4. Finally implement coreclr_create_delegate (this is yet another variation on mono_marshal_get_managed_wrapper+mono_ldftn; or more ambitiously it's a call to the appropriate function in ComponentActivator in mono SPC))

For static linking (ie ios device builds; and webassembly) we'd need to do... something else - probably a different version of DNNE's platform.c where we can statically link the runtime plus DNNE-created libraries plus the DNNE platform code into a single executable.

[AaronRobinsonMSFT]

@lambdageek I don't fully understand why all these steps are needed in this case. Is it because of mono's hosting semantics? I was assuming we could treat DNNE generated native libraries as if they were any other native binary – P/Invoke into them and be none the wiser?

Step (4) is all that I was expecting as actual work.

[lambdageek]

@lambdageek I don't fully understand why all these steps are needed in this case.

@AaronRobinsonMSFT My assumption is: we want to run the dll import generator tests on Android,ios and wasm, not just on Linux & mac desktops with Mono instead of CoreCLR as the underlying runtime.

If we only want to run on desktops then (4) is sufficient.

Is it because of mono's hosting semantics?

I'd say it's the hosting model, rather than something specific to mono. There's no nethost, hostfxr or hostpolicy on mobile and any code that depends on them will not work.

I was assuming we could treat DNNE generated native libraries as if they were any other native binary – P/Invoke into them and be none the wiser?

The DNNE generated native libraries are probably ok (assuming we compile them with the right toolchain). But they depend on nethost, hostfxr and hostpolicy, right? So the functions that platform.c depends on get_hostfxr_path, hostfxr_get_runtime_delegate etc are not present on mobile.

We haven't tried the desktop versions of hostfxr and hostpolicy on mobile - I have no idea how much work is required to get them working. (If I had to guess... Android and iOS Simulator are probably not too much work - they may even just work if we start building them and bundling them into the tests; iOS device and WebAssembly probably require moderate to significant changes since they're static linking platforms).

My guess is that the most straightforward way would be to build a mobile hostfxr library that exports the symbols that DNNE needs but implements them by directly calling the coreclr_ functions in the current process (ie there's no hostpolicy layer, and the implementation assumes the runtime is already loaded in the current process). Then the only other change needed is platform.c would need to skip get_hostfxr_path and get the symbols from RTLD_DEFAULT.