Remove always searching executable directory for native libraries in single-file applications

[elinor-fung]

Currently, in single-file .NET applications, the directory of the single-file executable is added to the NATIVE_DLL_SEARCH_DIRECTORIES property during startup. This means that the application directory is always probed when loading unmanaged libraries. This includes for p/invoke library load in the runtime itself, which the developer cannot control via the resolution extension points (DllImportResolver, AssemblyLoadContext.LoadUnmanagedDll).

This means that the following p/invoke will actually look in the application directory first and load lib from there if it exists:

[DllImport("lib")
[DefaultDllImportSearchPaths(DllImportSearchPath.System32)]
static extern void Method()

In NativeAOT on non-Windows, we set the rpath to the application directory, such that the OS search will always look in the application directory. As a result, the above p/invoke on non-Windows will still look in the application directory.

At the time this was done, self-contained, single-file publish on Windows still had separate native runtime libraries next to the app (such that it would always have the executable location in the search directories anyways). This is no longer the case and the behaviour has caused confusion for developers. NativeAOT behaviour was made to match the regular single-file behaviour.

We should make single-file stop always adding the executable directory and properly handle bundled assembly p/invoke resolution in the runtime to have assembly directory mean next to the bundled app - the option not taken in #42772. NativeAOT should also stop setting the rpath.

This should result in:

// Look in System32 on Windows, OS search on non-Windows
[DllImport("lib")
[DefaultDllImportSearchPaths(DllImportSearchPath.System32)]
static extern void Method()

// Look next to the single-file app because assembly directory means application directory for single-file
[DllImport("lib")
[DefaultDllImportSearchPaths(DllImportSearchPath.AssemblyDirectory)]
static extern void Method()

// Look next to the single-file app (because we search assembly directory by default), then default OS search
[DllImport("lib")
static extern void Method()

Related:

• Dll Search Path Behavior for Windows System Dlls for certain dotnet apps (WPF?) #101857
• [Single-File] DllImport fails to find native library #42772

cc @dotnet/appmodel @dotnet/interop-contrib @GrabYourPitchforks

[dotnet-policy-service]

Added needs-breaking-change-doc-created label because this issue has the breaking-change label.

1. Create and link to this issue a matching issue in the dotnet/docs repo using the breaking change documentation template, then remove this needs-breaking-change-doc-created label.

Tagging @dotnet/compat for awareness of the breaking change.

[dotnet-policy-service]

Tagging subscribers to this area: @agocke, @vitek-karas, @VSadov
See info in area-owners.md if you want to be subscribed.

[jkotas]

Do you plan to change the default for nativeaot too?

[elinor-fung]

No - if I understand correctly, that is based on OS behaviour. I'm not suggesting we try to change that portion for single-file either, just stop explicitly adding it to our own default search (that is, it will be searched for p/invokes regardless of what 'DllImportSearchPath` is specified).

NativeAOT does already consider DllImportSearchPath.AssemblyDirectory as the app directory, so I think updating single-file to do the same would actually help with consistency here.

[jkotas]

The OS behavior is configurable. We tried to align the defaults in native AOT with the regular single-file defaults:

On Unix, we set custom RPath in NativeAOT to make the native libraries next to app binary just work:

runtime/src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.Unix.targets

Lines 67 to 68 in 40f209c

	<IlcRPath Condition="'$(IlcRPath)' == '' and '$(_IsApplePlatform)' != 'true'">$ORIGIN</IlcRPath>
	<IlcRPath Condition="'$(IlcRPath)' == '' and '$(_IsApplePlatform)' == 'true'">@executable_path</IlcRPath>

On Windows, we opted to not set DEPENDENTLOADFLAG to prevent loading native libraries next to the app binary to match regular single-file behavior.

[jkotas]

cc @MichalStrehovsky for NativeAOT

[elinor-fung]

That still means we respect any specified DllImportSearchPath specified (for Windows) though. So if I say DllImportSearchPath.System32 for a p/invoke in nativeaot, it won't find the library next to the app - unlike in single-file today.

[jkotas]

It is going to look for it next to the app if the import is opted into DirectPInvoke. We have warning about DirectPInvoke at https://learn.microsoft.com/en-us/dotnet/core/deploying/native-aot/interop#direct-pinvoke-calls to wash our hands. I think our story is going to have cracks if we make regular single file stricter by default but keep Native AOT defaults as is.

[elinor-fung]

For user-defined p/invokes that is an opt-in though - if they don't want that behaviour, they still have the option of a regular p/invoke. Regular single-file today has no option to respect only the specified DllImportSearchPath.

I think it would actually make the two match better for regular single-file to:

• remove the executable directory from NATIVE_DLL_SEARCH_DIRECTORIES
• add rpaths and don't set DEPENDENTLOADFLAG (like in NativeAOT)

Then, regular p/invokes would respect DllImportSearchPath in both single-file and NativeAOT. Default would be able to find libraries next to the application in both. And regular single-file just doesn't have a DirectPInvoke equivalent.

[jkotas]

I may be misunderstanding the underlying motivation for this change. What's a simple user facing description of what we want to achieve by this change? I thought we want to be able to say something like "Single file apps won't try to load any native libraries (.dlls/.so/.dylib) from the application directory by default.".