The nunit engine does not use the extension with the latest version if multiple versions are available

[m4ggo]

If an extension exists in multiple versions within the packages folder the nunit engine does not select the highest version but always the first it finds - which is in many cases the lowest version.

Precondition:
One extension is installed in multiple versions in the packages directory, e.g.
\packages\nunit.extension.sample\1.0.0
\packages\nunit.extension.sample\2.0.0

Repro steps:

• Navigate into the folder for the nunit3-console.exe, e.g. packages\nunit.consolerunner\3.20.0\tools
• List the extensions for the runner via: nunit3-console.exe --list-extensions

Result:

• The extension with the lower version (1.0.0) is listed

Extension Point: /NUnit/Engine/TypeExtensions/IDriverFactory
Extension: SampleExtension.SampleExtension(.NET 4.0)
Version: 1.0.0.0
Path: C:.nuget\packages\nunit.extension.sample\1.0.0\net48\tools\SampleExtension.dll

Expected Result:
Extension Point: /NUnit/Engine/TypeExtensions/IDriverFactory
Extension: SampleExtension.SampleExtension(.NET 4.0)
Version: 2.0.0.0
Path: C:.nuget\packages\nunit.extension.sample\2.0.0\net48\tools\SampleExtension.dll

Affected Versions:
nunit.consolerunner.3.19.x
nunit.consolerunner.3.20.0

Workaround:
Switch to nunit.consolerunner.3.18.3 version

Root cause:
Project: nunit.engine.core
Class: Services\ExtensionManager.cs

In method ProcessCandidateAssembly the extensions are compared via IsBetterVersionOf but the new assembly is only added ByName and not updated in the _assemblies list.

line 437:

 if (candidateAssembly.IsBetterVersionOf(existing))
     _assemblies.ByName[assemblyName] = candidateAssembly;

line 444 - executed for the first found extension:
_assemblies.Add(candidateAssembly);

In method EnsureExtensionsAreLoaded the _assemblies is iterated but this list only contains the first extension, the latest extension is only in the 'ByName' Dictionary
line 323:

	foreach (var candidate in _assemblies)
            FindExtensionsInAssembly(candidate);

As local hotfix I iterated the ByName values via _assemblies.ByName.Values and after that it selected the latest version again, but I don't know if this is a valid .

	foreach (var candidate in _assemblies.ByName.Values)
            FindExtensionsInAssembly(candidate);

I added the two sample package folders which only contain an extension dll in different major versions which can be used to reproduce the behaviour of the --list-extensions command.
nunit.extension.sample.zip

[CharliePoole]

@m4ggo

Is this change the only one you made?

	foreach (var candidate in _assemblies.ByName.Values)
            FindExtensionsInAssembly(candidate);

If so, then there is some issue with the ExtensionAssemblyTracker class. It represents a list of assemblies and ByName and ByPath are there only to provide lookup convenience and _assemblies.ByName.Values should give the same list as _assemblies. I'll look into this.

[CharliePoole]

This actually highlights a build problem present in both the version 3 and version 4 builds.

We carefully control extension versions in our package tests (system level) so that only those we want to load are present. In our unit tests, we use a single FakeExtensions assembly, so everything in it is the same version. I'll have to look carefully at how we are proving that we find the best extension when multiple versions are present. Probably we aren't.

@m4ggo That said, could you explain why you have multiple versions of your extension present when you are running tests? I can't imagine you want both to be looked at for the same run.

[CharliePoole]

I've managed to incorporate this into my package tests and I have verified that the first version of an extension assembly encountered is always used and never replaced by a better version.

[m4ggo]

Hi Charlie,

the main reason for this is, that the extensions as well as the actual nunit3-console are located in the 'packages' directory and also executed from that location and not copied into the output directory of the tests. Due to the size of the project we are currently not allowed to copy anything other than the test dlls into the 'test output' directory.
The test itself shall only use one extension - the latest one. On the development machines this is not much of a problem, I can control what is in the package directory and do a cleanup if necessary. But on the test agents I don't have any access and the packages directory is not cleared before every execution. So it happens that multiple versions of the extensions are still available in the directory for some time.

Regarding the code change

	foreach (var candidate in _assemblies.ByName.Values)
            FindExtensionsInAssembly(candidate);

Yes, this was the only change I made but I realized that this is probably just a quick and dirty hotfix for the problem on this selected location and a proper fix would required some adaptions to the ExtensionAssemblyTracker. If you need assistance there, I can also take a look into that class and provide a merge request which a rework.

[CharliePoole]

@m4ggo Thanks, If you want to update your PR with the further fix, I'll make use of it.

For now, I'm working on the test side because it's important that some of our tests actually work with extensions created in more than one version. I plan to fix this and release 3.20.1 with the fix.

[m4ggo]

Hi Charlie,

I added a fix proposal here:

m4ggo@b6b2a1f

I also added some tests for different assembly versions.