[mono?] Cannot run host tests due to xunit version mismatch

[uweigand]

Description

Attempting to run the host tests on a Mono-based linux-s390x runtime fails with:

[ERROR] FATAL UNHANDLED EXCEPTION: System.TypeLoadException: Could not load type of field 'Xunit.ConsoleClient.ConsoleRunner:completionMessages' (3) due to: Could not load file or assembly 'xunit.runner.utility.netcoreapp10, Version=2.4.2.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c' or one of its dependencies.

Reproduction Steps

Run ./build host.tests --test on the current release/6.0 branch on linux-s390x.

Expected behavior

Test execution starts.

Actual behavior

Test fails to start with the error message shown above.

Regression?

Yes. This worked before 491ed9a .

Known Workarounds

Revert XUnitVersion back to 2.4.1. (This also requires moving to back-level versions of MicrosoftDotNetXUnitExtensionsVersion and MicrosoftDotNetXUnitConsoleRunnerVersion.)

Configuration

Runtime from current .NET 6 branch built on linux-s390x.

Other information

The root cause of the issue appears to be that the host tests (e.g. Microsoft.NET.HostModel.AppHost.Tests) depend on both

          "xunit.runner.console": "2.4.2-pre.9",

and

          "xunit.runner.visualstudio": "2.4.2"

Both of these packages bring their own copy of xunit.runner.utility.netcoreapp10.dll, but in two different versions. xunit.runner.console version 2.4.2-pre.9 brings version 2.4.2 of that DLL, while xunit.runner.visualstudio version 2.4.2 brings version 2.4.1 of that DLL.

During restore, it looks like one of those copies "wins", and this happens to be the older one (2.4.1). This causes the Mono loader error while loading dependencies of xunit.runner.console, which expects the newer version (2.4.2).

I tried to re-create the issue on Intel, but there I'm not seeing the error, even though the overall situation looks identical. So maybe part of the root cause may also involve the Mono loader - maybe this is more strict in checking version matches than the CoreCLR loader for some reason?

CC @steveisok @directhex @maryamariyan

[steveisok]

/cc @akoeplinger

[akoeplinger]

Can you try if you see the issue on main too or just release/6.0?

[uweigand]

Can you try if you see the issue on main too or just release/6.0?

On main, I don't get that far: here the problem is that it tries to build the tests as net7.0 code, but using the host SDK, which doesn't even support net7.0 yet ... Not sure what's going on there.

[lambdageek]

@uweigand can you run with MONO_LOG_LEVEL=debug MONO_LOG_MASK=asm on s390x - the loader should indicate if it found the assembly but rejected it for some reason

[lambdageek]

To the best of our ability we've tried to match mono's loader to the the assembly binder behavior in CoreCLR with respect to versions

• mono

  runtime/src/mono/mono/metadata/assembly.c

  Line 3019 in 862a90f

  if (result && assembly_names_compare_versions (wanted_name, candidate_name, -1) > 0)

• coreclr

  runtime/src/coreclr/binder/assemblybindercommon.cpp

  Lines 52 to 112 in 862a90f

  	bool IsCompatibleAssemblyVersion(/* in */ AssemblyName *pRequestedName,
  	/* in */ AssemblyName *pFoundName)
  	{
  	AssemblyVersion *pRequestedVersion = pRequestedName->GetVersion();
  	AssemblyVersion *pFoundVersion = pFoundName->GetVersion();
  	if (!pRequestedVersion->HasMajor())
  	{
  	// An unspecified requested version component matches any value for the same component in the found version,
  	// regardless of lesser-order version components
  	return true;
  	}
  	if (!pFoundVersion->HasMajor() || pRequestedVersion->GetMajor() > pFoundVersion->GetMajor())
  	{
  	// - A specific requested version component does not match an unspecified value for the same component in
  	// the found version, regardless of lesser-order version components
  	// - Or, the requested version is greater than the found version
  	return false;
  	}
  	if (pRequestedVersion->GetMajor() < pFoundVersion->GetMajor())
  	{
  	// The requested version is less than the found version
  	return true;
  	}
  	if (!pRequestedVersion->HasMinor())
  	{
  	return true;
  	}
  	if (!pFoundVersion->HasMinor() || pRequestedVersion->GetMinor() > pFoundVersion->GetMinor())
  	{
  	return false;
  	}
  	if (pRequestedVersion->GetMinor() < pFoundVersion->GetMinor())
  	{
  	return true;
  	}
  	if (!pRequestedVersion->HasBuild())
  	{
  	return true;
  	}
  	if (!pFoundVersion->HasBuild() || pRequestedVersion->GetBuild() > pFoundVersion->GetBuild())
  	{
  	return false;
  	}
  	if (pRequestedVersion->GetBuild() < pFoundVersion->GetBuild())
  	{
  	return true;
  	}
  	if (!pRequestedVersion->HasRevision())
  	{
  	return true;
  	}
  	if (!pFoundVersion->HasRevision() || pRequestedVersion->GetRevision() > pFoundVersion->GetRevision())
  	{
  	return false;
  	}
  	return true;
  	}

It's possible there's some non-determinism that causes Mono to load the older xunit assembly and CoreCLR to load the newer one.

I'm not sure whether xunit is using ALCs here - it's also possible we're either looking in the wrong ALC or doing some unintended sharing.

[uweigand]

@uweigand can you run with MONO_LOG_LEVEL=debug MONO_LOG_MASK=asm on s390x - the loader should indicate if it found the assembly but rejected it for some reason

Well, it rejects it because of the version:

Mono: Request to load xunit.runner.utility.netcoreapp10 in alc 0x2aa31299db0
Mono: netcore preload hook: did not find 'xunit.runner.utility.netcoreapp10'.
Mono: ApplicationBase is /home/uweigand/runtime-net6/artifacts/bin/Microsoft.NET.HostModel.AppHost.Tests/Debug/net6.0/
Mono: Assembly Loader probing location: '/home/uweigand/runtime-net6/artifacts/bin/Microsoft.NET.HostModel.AppHost.Tests/Debug/net6.0/xunit.runner.utility.netcoreapp10.dll'.
Mono: Image addref xunit.runner.utility.netcoreapp10[0x2aa313fb790] (asmctx DEFAULT) -> /home/uweigand/runtime-net6/artifacts/bin/Microsoft.NET.HostModel.AppHost.Tests/Debug/net6.0/xunit.runner.utility.netcoreapp10.dll[0x2aa31464e10]: 2
Mono: Predicate: wanted = xunit.runner.utility.netcoreapp10, Version=2.4.2.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c
Mono: Predicate: candidate = xunit.runner.utility.netcoreapp10, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c
Mono: Predicate: candidate and wanted names don't match, returning FALSE
Mono: Predicate returned FALSE, skipping 'xunit.runner.utility.netcoreapp10' (/home/uweigand/runtime-net6/artifacts/bin/Microsoft.NET.HostModel.AppHost.Tests/Debug/net6.0/xunit.runner.utility.netcoreapp10.dll)

Mono: Unloading image /home/uweigand/runtime-net6/artifacts/bin/Microsoft.NET.HostModel.AppHost.Tests/Debug/net6.0/xunit.runner.utility.netcoreapp10.dll [0x2aa31464e10].
Mono: The following assembly referenced from /home/uweigand/.nuget/packages/xunit.runner.console/2.4.2-pre.9/tools/netcoreapp2.0/xunit.console.dll could not be loaded:
     Assembly:   xunit.runner.utility.netcoreapp10    (assemblyref_index=10)
     Version:    2.4.2.0
     Public Key: 8d05b1bb7a6fdb6c

Mono: Failed to load assembly xunit.console[0x2aa3138c150].

It also seems to be the default ALC according to those logs.

As to:

It's possible there's some non-determinism that causes Mono to load the older xunit assembly and CoreCLR to load the newer one.

As far as I can see, at execution time there is just one assembly present. During restore, files from the two dependent packages are copied over, and given that each of those two packages contains a copy of that assembly (just at different versions), at the end of the restore process, one copy actually made it into the application directory. And that is the one the Mono loader finds (I don't think it even could find any other version at this point), but it is has the wrong version.

[lambdageek]

As far as I can see, at execution time there is just one assembly present. During restore, files from the two dependent packages are copied over, and given that each of those two packages contains a copy of that assembly (just at different versions), at the end of the restore process, one copy actually made it into the application directory. And that is the one the Mono loader finds (I don't think it even could find any other version at this point), but it is has the wrong version.

yea doesn't seem like the loader has a lot of options here. It would be nice to confirm that the coreclr loader isn't being less strict in this case for some reason, but it seems like an issue with the build.

[uweigand]

As far as I can see, at execution time there is just one assembly present. During restore, files from the two dependent packages are copied over, and given that each of those two packages contains a copy of that assembly (just at different versions), at the end of the restore process, one copy actually made it into the application directory. And that is the one the Mono loader finds (I don't think it even could find any other version at this point), but it is has the wrong version.

yea doesn't seem like the loader has a lot of options here. It would be nice to confirm that the coreclr loader isn't being less strict in this case for some reason, but it seems like an issue with the build.

I did confirm that in my Intel build (using CoreCLR), where the tests succeed, we also have the older version of xunit.runner.utility.netcoreapp10.dll in the application directory, so there doesn't appear to be any difference during restore. It's just that tests executes successfully anyway ...