DllNotFoundException thrown ONLY on mac catalyst release mode on arm64

[borrrden]

Apple platform

Mac Catalyst

Framework version

net8.0-*

Affected platform version

.NET 8.0.403

Description

I develop a nuget package that has a native framework inside of it. I've been developing a testing application for various platforms (including mac catalyst) that makes use of the package. I don't have an Apple Silicon mac, so I've been developing it on my Intel mac where everything works just as expected. However, as soon as I bring it over to Apple Silicon the same exact program crashes with DllNotFoundException UNLESS I run it through rosetta to use the x86_64 variant, or run a debug build.

I have verified the following:

• Both the main binary and the framework are inside of the app bundle, and contain slices for both x86_64 and arm64
• The RPATH of the main binary is set to @executable_path/../Frameworks
• otool -L shows that the main binary is linked with the framework

I suppose this means it has something to do with AOT compilation? Here are some of the references to what I am doing. I use the nuget package called Couchbase.Lite.Enterprise where I make use of 100+ DllImport calls such as this one:

https://github.com/couchbase/couchbase-lite-net/blob/5302dba59ae4bacef39f2a33d173d1edc8909616/src/LiteCore/src/LiteCore.Shared/Interop/C4Log_native.cs#L38

You can see that the name of the library is defined as "LiteCore" https://github.com/couchbase/couchbase-lite-net/blob/5302dba59ae4bacef39f2a33d173d1edc8909616/src/LiteCore/src/LiteCore.Shared/Constants.cs#L24

Furthermore, for the benefit of iOS / mac catalyst (not sure actually if this is needed or not anymore) I set up a resolver function to remap this to @rpath/LiteCore.framework/LiteCore:

https://github.com/couchbase/couchbase-lite-net/blob/5302dba59ae4bacef39f2a33d173d1edc8909616/src/LiteCore/src/LiteCore.Shared/Interop/NativeActivate.cs#L39-L41

This works just fine for iOS simulator, ios device, mac catalyst x64 in any config, and mac catalyst arm64 in debug. However running a release build on arm64 crashes immediately and I see in the output:

*** Terminating app due to uncaught exception 'System.TypeInitializationException', reason: 'TypeInitialization_Type, Couchbase.Lite.Database (System.TypeInitializationException)
   at TestServer.MauiProgram.CreateMauiApp()
   at TestServer.AppDelegate.CreateMauiApp()
   at Microsoft.Maui.MauiUIApplicationDelegate.WillFinishLaunching(UIApplication application, NSDictionary launchOptions)
LiteCore (System.DllNotFoundException)
   at Couchbase.Lite.Logging.FileLogger.SetupDomainObjects()
   at Couchbase.Lite.Logging.FileLogger..ctor()
   at Couchbase.Lite.Logging.Log..ctor()
   at Couchbase.Lite.Database..cctor()

Which lines up to the first time a DllImport function is called here https://github.com/couchbase/couchbase-lite-net/blob/5302dba59ae4bacef39f2a33d173d1edc8909616/src/Couchbase.Lite.Shared/API/Database/Database.cs#L255

I'm not even sure what I can do to debug this, let alone get any further information but it would be nice if it ran without crashing.

Steps to Reproduce

1. Create a new mac catalyst project on Apple Silicon
2. Add the Couchbase.Lite.Enterprise nuget package
3. Add something like var db = new Database("test") inside the app delegate finished launching function
4. Make a release build and try to run the main binary directly (i.e. app.app/Contents/MacOS/app)

Did you find any workaround?

Don't compile for arm64 (leave it out of the runtime identifiers)

Relevant logs

No response

[rolfbjarne]

Furthermore, for the benefit of iOS / mac catalyst (not sure actually if this is needed or not anymore) I set up a resolver function to remap this to @rpath/LiteCore.framework/LiteCore:

couchbase/couchbase-lite-net@5302dba/src/LiteCore/src/LiteCore.Shared/Interop/NativeActivate.cs#L39-L41

Is the resolve function executed at all on Mac Catalyst/x64?

That said, I think the first question would be: is the LiteCore library where it's supposed to be?

You can figure that out by looking at the files in the app bundle (might have to fix the path slightly depending on your build settings):

$ find ./bin/Release/*.app/

and there should be one file among the result like this: app.app/Contents/Frameworks/LiteCore.framework/LiteCore.

If that's not the case, then please provide an MSBuild binlog, because it sounds like a build issue.

If the library is there, then it might be a problem where the OS refuses to load it.

One way to figure this out is to open the library manually, and checking the error afterwards:

using ObjCRuntime;

static void CheckLibrary ()
{
	var rv = Dlfcn.dlopen ("@rpath/LiteCore.framework/LiteCore", Dlfcn.Mode.Now);
    if (rv == IntPtr.Zero) {
		var error = Dlfcn.dlerror ();
		Console.WriteLine ("Failed to load library: {error}");
	} else {
		Console.WriteLine ("Loaded library successfully");
	}
}

and then where to go from here depends on the actual error.

[borrrden]

Ok here is what I have found from looking into what you said. The library is for sure in the location:

I assume that the resolve function is being used, because using just "LiteCore" as a DllImport name on mac catalyst throws a DllNotFoundException on x64. It HAS to be the rpath variant.

ls -l tmp.app/Contents/Frameworks/LiteCore.framework
total 0
lrwxr-xr-x@ 1 mobile  staff   24 Nov  8 13:57 Headers -> Versions/Current/Headers
lrwxr-xr-x@ 1 mobile  staff   25 Nov  8 13:57 LiteCore -> Versions/Current/LiteCore
lrwxr-xr-x@ 1 mobile  staff   24 Nov  8 13:57 Modules -> Versions/Current/Modules
lrwxr-xr-x@ 1 mobile  staff   26 Nov  8 13:57 Resources -> Versions/Current/Resources
drwxr-xr-x@ 4 mobile  staff  128 Oct 29 14:37 Versions

Interestingly, running your code for dlopen I got random errors that were just garbage, but a library that refused to load. Then I tried using the absolute path and I got "Library loaded successfully". I took this back to my x64 machine and the rpath variant worked fine without converting it to an absolute path. This may be where the fault lies.

[borrrden]

One more interesting piece of info...the following works on arm64 as well:

@executable_path/../Frameworks/LiteCore.framework/LiteCore

and also just to double check, I built the app as x64 on the arm64 machine and tried the rpath variant again and dlopen succeeded.

EDIT I deleted my next comment about insta-crashing because it's a different problem. I have to often delete the bin and obj folders when building again in release mode because the AOT information gets out of whack or something (I always forget about that).