Description
Summary
Introduce a new dotnet test experience that is focused toward the new testing platform Microsoft.Testing.Platform (aka MTP).
This ticket will act as the hub of all implementations related to the new dotnet test experience.
Background and Motivation
We have introduced a new testing platform as an alternative to VSTest, this new testing platform, named Microsoft.Testing.Platform changes fundamentally the way the test applications are structured.
VSTest:
Microsoft.Testing.Platform:
Currently, Microsoft.Testing.Platform allows tests to be run through dotnet test using some workaround by injecting a named target that will be loaded by dotnet test code in place of the one from VSTest. This behavior is fine when doing basic dotnet test call but is showing problem when we want to use more advanced features as the options are statically declared and tuned for VSTest. For example, we need to do dotnet test -- --minimum-expected-tests 10 or dotnet test -p:TestingPlatformCommandLineArguments="--minimum-expected-tests 10". This is also causing confusions to users for some arguments, for example, doing dotnet test --filter A will not forward the filter down to the new platform but will eat the argument for VSTest (which is not plugged in that context) and users should instead do dotnet test -- --filter A or dotnet test -p:TestingPlatformCommandLineArguments="--filter A".
One downside to the new testing platform is that each test application is working in "isolation" to other test applications while some features are expected to be provided as cross-test apps (e.g. users would expect at least one merge trx report and not only 1 trx per test application).
Proposed Feature
Instead of shipping an extra client executable, we will use dotnet process to act as a client and offer pre and post run capabilities.
The new testing platform offers the possibility to extend the supported command line arguments so we no longer can simply rely on a static way but instead either ask test applications for their supported options (e.g. when user will do dotnet test --help) or forward the extra parameters down to the test applications (e.g. dotnet test --coverage, the --coverage needs to be forward to each test application).
We should allow users to select the dotnet test mode (see #45471) to be either the new experience or the old one.
Under discussions
- We are currently discussing what to do for the case of "mixed mode", at least one project using VSTest and one project using MTP.
Limitations
Other
Note that I will edit this main design as we progress through the design and implementation and incorporate user's comments.
Pinging main test framework authors:
- MSTest: @Evangelink @Youssef1313
- NUnit: @OsirisTerje @rprouse
- TUnit: @thomhurst
- xUnit: @bradwilson