Description
Hi there!
I'm not sure if this is really considered a bug since it's probably quite uncommon but I've noticed a difference between running tests with the nunit3-console and dotnet test, the test runner in VS 2022 and the ReSharper test runner. All yield the same result except for the nunit3-console where some test suddenly fail.
Context
For performance reasons I've started moving the test execution from dotnet test to the nunit3-console but I quickly ran into some testcases that fail with the nunit-console but pass with all other runners I've tested so far. The failing testcases were all related to PowerShell in some way or another (most of the projects-under-test contain either PowerShell cmdlets or PowerShell hosts). To cover some of the PowerShell specific functionalities we have setup some test fixtures to open a PowerShell runspace and run PowerShell code in the unit test.
After quite a lot of debugging I've noticed a difference in where some of the assemblies are loaded. When I use any of the before mentioned test runners, everything is loaded on the DefaultAssemblyLoadContext but when I run the test with the nunit-console the assembly-under-test is loaded on a TestAssemblyLoadContext. This initially seemed fine but there is a known issue / missing feature in PowerShell (PowerShell/PowerShell#17492) (issue is marked as closed, but AFAIK has not been fixed as of writing this) that prevents a PowerShell from being hosted in an AssemblyLoadContext other than the DefaultAssemblyLoadContext. I assume that this is the problem I am facing but I didn't invest the time to check how exactly the PowerShell implementation loads its dependencies. I also think that this problem is not exclusive to PowerShell but every assembly that loads dependencies via Assembly.LoadFrom() as it will end up on the DefaultAssemblyLoadContext. (Behavior documented here (MS Doc))
Minimal Working Example
I've tried to create a minimal working example to demonstrate the issue, I've also managed to reproduce it without using/hosting PowerShell, by using Assembly.LoadFrom() to load a the "dependency" assembly. I've created a repo with a Reproduce.ps1 script that should show the dotnet test result and the one from the nunit3 console runner (only tested with windows powershell). viktorgobbi/NUnitConsoleALCIssue
I'm not too familiar with the NUnit engine / console code base but my primary question is: Is there a way to get the agent to load everything on the DefaultAssemblyLoadContext? Since I cannot really change the PowerShell behavior I see that as the only good way to resolve this. I think the critical part causing this behavior is in the NUnitNetCore31Driver.
Versions used
- All projects target NET8.0
- VisualStudio 2022 Version 17.9.5
- JetBrains ReSharper 2025.1.2
- MSBuild 17.9.6+a4ecab324
- NUnit framework: NUnit3.14 (I think I've tested upgrading to 4 but with the same result)
- NUnit.ConsoleRunner versions tested 3.20.1, 3.20, 3.18, and 3.17 (all reproduce the issue)
- OS Version: Microsoft Windows NT 6.2.9200.0
- Runtime: .NET Framework CLR v4.0.30319.42000
In case any other details are required, just let me know.
Thanks!