Symbolic link change in .NET 9 breaks previously functioning apps

[jhudsoncedaron]

Description

Ref: https://learn.microsoft.com/en-us/dotnet/core/compatibility/deployment/9.0/assembly-load-directory

This broke us pretty badly. We have two hundred fifty copies of the application on one server; so the entire application except for the config files is symbolic-linked in from a pile install. Due to differing version dependencies; it's not possible to run from the pile install at all; the symbolic tree is necessary to make any runnable directory structure.

"If your application relied on the previous behavior, ensure that all relevant binaries are located in the directory behind the symbolic link, rather than next to it."

This recommendation is eight years too late.

We had actually discovered the behavior was different on Linux and used a LD_PRELOAD to prevent the loader from noticing it was a symbolic link.

Reproduction Steps

There's no meaningful minimal reproduction for this one.

Expected behavior

Revert to .NET 8 behavior.

Actual behavior

Application fails to load altogether

A fatal error was encountered. The library 'hostpolicy.dll' required to execute the application was not found in 'C:\Program Files\dotnet'.
Failed to run as a self-contained app.

• The application was run as a self-contained app because 'C:\Program Files\Cedaron Form Builder\Shared\trees\Form Builder\10.0.3035.902\Cedaron.FormBuilder.WebAPI\sqlexec.runtimeconfig.json' did not specify a framework.
• If this should be a framework-dependent app, specify the appropriate framework in 'C:\Program Files\Cedaron Form Builder\Shared\trees\Form Builder\10.0.3035.902\Cedaron.FormBuilder.WebAPI\sqlexec.runtimeconfig.json'.

Fixing hostpolicy is possible but other things will go wrong. Everything depends on Path.GetDirectory(typeof(Program).GetAssemblyFileName) resolving the config files.

Regression?

Yes, .NET 8

Known Workarounds

We don't have one yet. Hard links ought to work but don't due to the inability to replace a file with multiple hard links atomically.

Configuration

.NET 9.0.4

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @vitek-karas, @agocke, @VSadov
See info in area-owners.md if you want to be subscribed.

[agocke]

Thanks for the feedback. We'll take a close look at the feedback here.

We don't have one yet. Hard links ought to work but don't due to the inability to replace a file with multiple hard links atomically.

Can you place the apphost (blah.exe) in the target directory, and symlink everything else?

[jhudsoncedaron]

Can you place the apphost (blah.exe) in the target directory, and symlink everything else?

Well that's what I get for working late; the apphost is 147 KB not 7MB. It will take a day to find out if it works at all but it's not a prohibitive copy.

Oh yes; this only works well if the apphost can tolerate the root dll being swapped out behind its back (which I haven't tried since .net 1.1). Emergency patches are applied to the shared copy and all instances restarted. This is very fast; but depends on the symbolic link tree working as intended.

[agocke]

The change only really affects the load resolution -- so if the app is restarted and it goes through resolution, it will likely find everything it needs next to the app, at which point it will load as it always has. If this worked before (we didn't know people were actually using this in prod!) then I think it should work again.

That said, we would definitely appreciate an update on whether that works for you!

[jhudsoncedaron]

So it passed a preliminary test. I'm surprised this works at all because of the nice line:

                var myjson = typeof(Program).Assembly.AssemblyDirectory() + "appsettings.json";

only ten lines into main. (AssemblyDirectory() is a custom extension method; it's just Path.GetDirectoryName(this.Location) on .NET Runtime targets; on .NET Framework it contains a bunch of junk because Assembly.Location is broken in .NET Framework)

This means the only symbolic link that's being followed is the symbolic link to the executable itself.

So now the instance installer needs to deal with a whole host of issues that used to be impossible that all stem from the same root cause: CopyFile is not and cannot be atomic so it has to handle truncated executables. And I need to ship a list of affected filenames (matching *.exe isn't actually right...) In all ways, very ugly.

And I thought my LD_PRELOAD to fix it on Linux was ugly; this is uglier.

Hilariously if there were a backout switch in *.launchsettings.json I would be able to set it. The launchsettings.json found by following symbolic link out of the exe is the real one. It's the framework dlls that aren't there, as that's a package dependency and resolved as such by the symtree builder.

[agocke]

FYI, based on feedback from users who were depending on the previous symlink behavior, we're considering reverting the change in .NET 9 and returning to .NET 8 behavior in perpetuity.