Exception when loading native dlls from the application path when using AssemblyDirectory and System32 as the DllImportSearchPath

[ThomasGoulet73]

Description

There seems to be an issue when loading a native dll in the application path that depends on a native dll also in the application path when using DllImportSearchPath.AssemblyDirectory | DllImportSearchPath.System32 as the search path. When both of those scenario occurs, it'll throw an exception that the dll could not be loaded. This issue is not reproducible when using only DllImportSearchPath.AssemblyDirectory or if a native dll does not have a dependency on another native dll in the application path. This also doesn't seem to affect native dlls shipped in a runtime pack because the .Net host seems to include a custom search path for the runtime packs.

This is a bug I found while investigating a problem when running WPF unit tests, see dotnet/wpf#10304 (comment) for more info. I'm not sure if it's the expected behavior, a bug or if I simply misunderstood the bug in WPF.

/cc @dipeshmsft

Reproduction Steps

1. Clone https://github.com/ThomasGoulet73/DllSearchPathRepro
2. Build TestLibrary1.
3. Build TestLibrary2.
4. Run ConsoleApp. It should exit without issue.
5. Open Program.cs in ConsoleApp.
6. Comment the line in Main with Test1WithoutSystem32
7. Uncomment the line in Main with Test1WithSystem32
8. Run ConsoleApp. It should exit with an exception that TestLibrary1.dll could not be loaded.

Expected behavior

Methods Test1WithoutSystem32 and Test1WithSystem32 should both succeed.

Actual behavior

Methods Test1WithoutSystem32 succeeds and Test1WithSystem32 fails.

Regression?

I tested using .Net 6 through .Net 9 and they all have the same behavior.

Known Workarounds

Change the DllImportSearchPath.

Configuration

.Net 9.0
Windows 11 x64
Don't know if it is specific to my configuration.

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/interop-contrib
See info in area-owners.md if you want to be subscribed.

[AaronRobinsonMSFT]

@elinor-fung can you take a look?

[elinor-fung]

We end up loading with LOAD_WITH_ALTERED_SEARCH_PATH | LOAD_LIBRARY_SEARCH_SYSTEM32. The LoadLibrary documentation indicates that LOAD_WITH_ALTERED_SEARCH_PATH cannot be combined with any LOAD_LIBRARY_SEARCH flag.

runtime/src/coreclr/vm/nativelibrary.cpp

Line 708 in 1da2ea4

hmod = LoadFromPInvokeAssemblyDirectory(callingAssembly, currLibNameVariation, loadWithAlteredPathFlags | dllImportSearchPathFlags, pErrorTracker);

I'm not sure why it only fails in the case where it needs to load a dependency of the library, but we should probably remove any LOAD_LIBRARY_SEARCH flags when explicitly passing LOAD_WITH_ALTERED_SEARCH_PATH (like we do for the assembly directory case above).

I think that is also true for the case where the DllImport specifies an absolute path:

runtime/src/coreclr/vm/nativelibrary.cpp

Lines 690 to 698 in 1da2ea4

	if (!libNameIsRelativePath)
	{
	DWORD flags = loadWithAlteredPathFlags;
	if ((dllImportSearchPathFlags & LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR) != 0)
	{
	// LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR is the only flag affecting absolute path. Don't OR the flags
	// unconditionally as all absolute path P/Invokes could then lose LOAD_WITH_ALTERED_SEARCH_PATH.
	flags |= dllImportSearchPathFlags;
	}

[ericstj]

I happened across this same behavior on .NETFramework when helping someone there with a near identical problem. It looked to me like the runtime was trying to resolve the file itself and then tell load-library to load that specific file path, but then it disabled the flag that would allow loadlibrary to look next to that library for it's native dependencies (so it only probed System32/SysWOW64).

I bet this is busted for WPF unit tests and not for the product due to missing entries in the deps.json for the unit tests. For the product the framework deps file lists all these native dependencies and the runtime must do something different with those paths to ensure LoadLibrary is able to locate them.