Description
openedon Jul 12, 2024
Can I do this today?
<Project Sdk="Microsoft.NET.Test.Sdk">
<PropertyGroup>
<TargetFrameworks>net8;net8-windows</TargetFrameworks>
</PropertyGroup>
<PropertyGroup>
<!-- MSTest, VSTest, NUnit, xUnit, ... -->
<UnitTestFramework>xUnit</UnitTestFramework>
<!-- MSCodeCoverage, Coverlet, ... -->
<CoverageTestFramework>Coverlet</CoverageTestFramework>
<!-- Benchmark.NET, ... -->
<PerformanceTestFramework>Benchmark.NET</PerformanceTestFramework>
</PropertyGroup>
<PropertyGroup Condition="'$(TargetFramework)' == 'net8-windows'">
<UseWinUI>true</UseWinUI>
<TestRunner>VSTest</TestRunner> <!-- Default: new MSTest Runner -->
<OutputType>AppContainerExe</OutputType>
</PropertyGroup>
<ItemGroup Condition="'$(TargetFramework)' == 'net8-windows'">
<PackageReference Include="Microsoft.WindowsAppSDK" />
</ItemGroup>
</Project>Assuming Default Includes in the Test SDK
<Project>
<!-- Implicitly Included in the Test SDK -->
<ItemGroup>
<_ImplicitFrameworkReference Include="Microsoft.NET.Testing.App" />
</ItemGroup>
<!-- Implicit References from the above Shared Framework -->
<!-- MSTest Includes -->
<ItemGroup Condition="'$(UnitTestFramework)' == 'MSTest'">
<_ImplicitReference Include="System.Testing" />
<_ImplicitUsing Include="System.Testing" />
</ItemGroup>
<!-- VSTest (Legacy MSTest) Includes -->
<ItemGroup Condition="'$(UnitTestFramework)' == 'VSTest'">
<_ImplicitReference Include="Microsoft.VisualStudio.TestTools.UnitTesting" />
<_ImplicitUsing Include="Microsoft.VisualStudio.TestTools.UnitTesting" />
</ItemGroup>
<!-- NUnit Includes -->
<ItemGroup Condition="'$(UnitTestFramework)' == 'NUnit'">
<_ImplicitReference Include="NUnit" />
<_ImplicitUsing Include="NUnit.Framework" />
</ItemGroup>
<!-- xUnit Includes -->
<ItemGroup Condition="'$(UnitTestFramework)' == 'xUnit'">
<_ImplicitReference Include="xUnit" />
<_ImplicitUsing Include="Xunit" />
</ItemGroup>
<!-- Similar Includes for Coverage and Performance frameworks -->
</Project>Here's a crazy thought
The .NET currently doesn't have a default testing framework in BCL, so why not donate testfx to dotnet and make it into a base framework, say, System.Testing (with customizable AppHost, Async Testing, proper API and runtime hooks for benchmarks, coverage, reporting, tracing, exception handling, etc...).
The basic testing infrastructure, say, System.Testing (not just unit tests but also code coverage, performance (#2340), functional and end to end testing) so that other testing frameworks can extend it instead of competing with it. The basic premise is that the feature set of v1 of this should enable the testing of dotnet/runtime and siblings without extending too much.
I'm thinking along the lines of Microsoft.NET.Test.App or Microsoft.NET.Testing.App framework packages. It'll come with all the benefits of framework packages. Then an SDK package (Microsoft.NET.Test.Sdk) that comes inbox in .NET SDK. The main reason why I want this instead of bunch of library references is simplicity and maintainability when doing tests separately from build in isolated environment (no networking).
Some background on Testing Framework package requirement
We have certain (legacy) projects where tests are kept separately from the source. Only unit tests covering the source are included within the source repo. Every other test case is added on user report, vulnerability reports, etc... So, you can safely say that there are millions of test cases added over the years. We don't ever remove them even if they become redundant. Thus, building those tests and running them along with the source takes hours. That adds up with every commit. So, we test them asynchronously and separately from the main build only on hot path code change, before deployment, on private and public test environments, and before production deployment. This is what we call a build-less deployment testing. It has its ups and downs but in our case it works.
Here comes the problem: Since we are building the tests separately, we need to deploy both binaries, test data to several pre-configured environments which puts pressure on our internal network. So, in order to avoid network congestion, first we had a solution that restored the assemblies from deps.json but that was not feasible in the long run for several reasons. So, now, we build empty projects (another repo) with copy-local dependencies converted to Shared Framework package manually and pre-deployed in those environments once for every version update. This sort of mimics the built-in framework model and also enables us to do auditing. Recently we moved towards MSTest Runner as the default, which already removed a bunch of dependencies, custom code and simplified our environments.
Problem solved right? Nope not even close, as this also comes with a cost of maintaining and synchronising across all the repos. The actual binaries built from source are around 10-15. The rest of them are static versioned binaries that just need to be there in the environment. We tried using sharedfx Sdk that dotnet/runtime uses but it was too complicated for our use case. I just wanted a simple shared framework for our dependencies. So, As I said before, we package them manually.
The shared framework for our tests contains our own test abstractions and dependencies like MSTest, xUnit, Coverlet and Benchmark.NET, etc.... It'll be better if an all-encompassing Testing framework was included in NET SDK. That way we don't have to redistribute our base testing libraries every single time. If anyone says NuGet, yes, we do use a private feed but only during development and Source Build CI.
Note
I'm aware of some aspects of what is available today and what works now. So, this is more of a high-level direction than product feedback.