Add support for running xUnit tests using .NET Core

[jcansdale]

[jcansdale]

[jcansdale]

TestDriven.NET-4.0.3258_Extraterrestrial_Alpha (1).zip

I haven't updated the release notes yet, but this versions supports running xUnit, NUnit and "Ad hoc" tests using .NET core.

The .NET Core test runner requires that unit tests be inside a Console Application project, but any method/property that doesn't have a test attribute can be executed as an "Ad hoc" test (even if it's inside a Class Library project). Anything returned from the method/property will be expanded on the output window.

Please let me know how you get on and report any issues here. 😃

[jeremydmiller]

@jcansdale,

I'm trying it out on a small solution, but it isn't able to resolve the xunit assembly. Should we be copying the xUnit assemblies to wherever TestDriven.Net is testing it, or something about how files are being copied?

Test 'M:Baseline.Testing.LightweightCacheTests.can_remove' failed: Could not load file or assembly 'xunit.core, Version=2.1.0.3179, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c' or one of its dependencies. The system cannot find the file specified.
    System.IO.FileNotFoundException: Could not load file or assembly 'xunit.core, Version=2.1.0.3179, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c' or one of its dependencies. The system cannot find the file specified.
    File name: 'xunit.core, Version=2.1.0.3179, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c'
    at System.ModuleHandle.ResolveType(RuntimeModule module, Int32 typeToken, IntPtr* typeInstArgs, Int32 typeInstCount, IntPtr* methodInstArgs, Int32 methodInstCount, ObjectHandleOnStack type)
    at System.ModuleHandle.ResolveTypeHandleInternal(RuntimeModule module, Int32 typeToken, RuntimeTypeHandle[] typeInstantiationContext, RuntimeTypeHandle[] methodInstantiationContext)
    at System.Reflection.RuntimeModule.ResolveType(Int32 metadataToken, Type[] genericTypeArguments, Type[] genericMethodArguments)
    at System.Reflection.CustomAttribute.FilterCustomAttributeRecord(CustomAttributeRecord caRecord, MetadataImport scope, Assembly& lastAptcaOkAssembly, RuntimeModule decoratedModule, MetadataToken decoratedToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, Object[] attributes, IList derivedAttributes, RuntimeType& attributeType, IRuntimeMethodInfo& ctor, Boolean& ctorHasParameters, Boolean& isVarArg)
    at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeModule decoratedModule, Int32 decoratedMetadataToken, Int32 pcaCount, RuntimeType attributeFilterType, Boolean mustBeInheritable, IList derivedAttributes, Boolean isDecoratedTargetSecurityTransparent)
    at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeAssembly assembly, RuntimeType caType)
    at System.Reflection.RuntimeAssembly.GetCustomAttributes(Type attributeType, Boolean inherit)
    at System.Attribute.GetCustomAttributes(Assembly element, Type attributeType, Boolean inherit)
    at System.Attribute.GetCustomAttribute(Assembly element, Type attributeType, Boolean inherit)
    at System.Reflection.CustomAttributeExtensions.GetCustomAttribute[T](Assembly element)
    at TestDriven.TestRunner.TargetFrameworkUtilities.TargetFrameworkAgent.GetTargetFrameworkName(String assemblyPath)
    at TestDriven.TestRunner.TargetFrameworkUtilities.TargetFrameworkAgent.GetTargetFrameworkName(String assemblyPath)
    at TestDriven.TestRunner.TargetFrameworkUtilities.FindTargetFrameworkName(String assemblyPath)
    at TestDriven.NetCore.TestRunner.NetCoreTestRunner.isNetCoreApp(String assemblyFile)
    at TestDriven.NetCore.TestRunner.NetCoreTestRunner.Run(ITestListener testListener, String assemblyFile, String cref)
    at TestDriven.TestRunner.AdaptorTestRunner.Run(ITestListener testListener, ITraceListener traceListener, String assemblyPath, String testPath)
    at TestDriven.TestRunner.ThreadTestRunner.Runner.Run()

    Assembly manager loaded from:  C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
    Running under executable  C:\Program Files (x86)\TestDriven.NET 4\ProcessInvocation86.exe
    --- A detailed error log follows.

    === Pre-bind state information ===
    LOG: DisplayName = xunit.core, Version=2.1.0.3179, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c
     (Fully-specified)
    LOG: Appbase = file:///C:/Program Files (x86)/TestDriven.NET 4/NetCore
    LOG: Initial PrivatePath = NULL
    Calling assembly : Baseline.Testing, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.
    ===
    LOG: This bind starts in LoadFrom load context.
    WRN: Native image will not be probed in LoadFrom context. Native image will only be probed in default load context, like with Assembly.Load().
    LOG: Using application configuration file: C:\Program Files (x86)\TestDriven.NET 4\ProcessInvocation86.exe.Config
    LOG: Using host configuration file:
    LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
    LOG: Post-policy reference: xunit.core, Version=2.1.0.3179, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c
    LOG: Attempting download of new URL file:///C:/Program Files (x86)/TestDriven.NET 4/NetCore/xunit.core.DLL.
    LOG: Attempting download of new URL file:///C:/Program Files (x86)/TestDriven.NET 4/NetCore/xunit.core/xunit.core.DLL.
    LOG: Attempting download of new URL file:///C:/Program Files (x86)/TestDriven.NET 4/NetCore/xunit.core.EXE.
    LOG: Attempting download of new URL file:///C:/Program Files (x86)/TestDriven.NET 4/NetCore/xunit.core/xunit.core.EXE.
    LOG: Attempting download of new URL file:///C:/code/baseline/src/Baseline.Testing/bin/Debug/netcoreapp1.0/xunit.core.DLL.
    LOG: Attempting download of new URL file:///C:/code/baseline/src/Baseline.Testing/bin/Debug/netcoreapp1.0/xunit.core/xunit.core.DLL.
    LOG: Attempting download of new URL file:///C:/code/baseline/src/Baseline.Testing/bin/Debug/netcoreapp1.0/xunit.core.EXE.
    LOG: Attempting download of new URL file:///C:/code/baseline/src/Baseline.Testing/bin/Debug/netcoreapp1.0/xunit.core/xunit.core.EXE.

[jeremydmiller]

@jcansdale It does work when I put the xunit assemblies under the /NetCore directory, so there's that.

In my project configuration, I did notice that the xunit assemblies are not being copied into bin/Debug/netcoreapp1.0. They do show up in bin\Debug\net451\win7-x64 if that helps. Is there another setting in the project.json to force it to copy the dependency assemblies there?

[jcansdale]

@jeremydmiller It should just work, you shouldn't have to copy anything! I'll investigate and get back to you. It works on my machine. ;)

[jeremydmiller]

I blame VS/dotnet anyway;)

[jcansdale]

Are you using any assembly level custom attributes? I.e. [assembly: FooBar(....)]

[jeremydmiller]

Other than the normal versioning stuff, just this:

[assembly: CollectionBehavior(DisableTestParallelization = true)]

Which I think we could get away w/ dropping on this project.

[jcansdale]

Ahhhh, that makes sense. Didn't expect that though. I expect it would have worked for most people. Thanks for the bug report!

[jeremydmiller]

A little more feedback, I can trigger tests via the actual test, test classes, and namespaces. It's throwing this against the Assembly or by the option to run all tests in the solution:

Error: 
Error: Unhandled Exception: System.IndexOutOfRangeException: Index was outside the bounds of the array.
Error:    at TestDriven.NetCore.AdHoc.XUnitNetCoreTestRunner.toArgs(ITestListener testListener, String assemblyFile, String cref)
Error:    at TestDriven.NetCore.AdHoc.XUnitNetCoreTestRunner.Run(ITestListener testListener, String assemblyFile, String cref)
Error:    at TestDriven.NetCore.AdHoc.Program.run(String assemblyFile, String cref)
Error:    at TestDriven.NetCore.AdHoc.Program.Main(String[] args)

[jcansdale]

@jeremydmiller Here's a new build with those issues resolved.

TestDriven.NET-4.0.3264_Extraterrestrial_Alpha.zip

The first issue was to do with how it detects a .NET Core assembly from .NET code. It needed to read the [assembly: TargetFramework(...)] attribute an look for ".NETCoreApp" or ".NETStandard". The issue was it also found your xUnit assembly level attribute, but couldn't resolve the assembly it was defined in. I'm now detecting .NET Core targets in a different way.

The other issue was an oversight. I'd already fixed it in the code that supports NUnit. When you execute all tests in a solution, there will still be some output for non-test projects (I'll fix this later).

Please keep the bug reports coming!