Provide a way for test frameworks to know when you're running in interactive mode (Test Explorer)

[bradwilson]

Summary

As a test framework, xUnit.net wants to do some things differently based on whether you're running in an interactive mode or not (meaning, running inside of Test Explorer or some other environment that expects to enumerate tests, present a UI to the user, and then run specific tests).

Background and Motivation

Today, there are two conditions where this information is valuable.

1. When running interactively, we want to pre-enumerate all test rows, so the user can run individual test rows. When running non-interactively, this is a waste of time and resources and causes the test discovery to be slower.

2. MTP does not offer a node state for "tests which were not run" (in our case, this generally means a test was marked as Explicit). When running interactively, we must try to "guess" the user's intent when running a subset of tests whether they're asking to run just explicit tests or not, since there is no gesture in the UI regarding explicit tests. When running non-interactively, we can rely on the user to pass our --explicit switch to indicate whether they want to run explicit tests or not. This reason this is important is that, from a usability perspective, we don't want to report any test result for a test that wasn't run, so that it'll stay in the "not run" state in the Test Explorer UX, whereas we want to report the test as skipped to dotnet test to remind the user that the test was not run (since there's no MTP test node state for "not run").

Prior to .NET SDK 10, the MTP team had told me to detect the --server switch to indicate that we were running interactively. Now with .NET SDK 10, both dotnet test and Test Explorer are using --server, and the new recommendation is to parse the argument to --server to determine this.

There should be a simple and unchanging way to determine this situation.

Proposed Feature

Add an API to query the information we need to know.

The simplest version would be just to query for an "interactive" flag: Test Explorer would set this, and dotnet test would not.

I don't think a more complex "server capabilities" style API would be appropriate here, because both situations seem relatively specific to xUnit.net (especially the explicit test support).

[Youssef1313]

As a test framework, xUnit.net wants to do some things differently based on whether you're running in an interactive mode or not (meaning, running inside of Test Explorer or some other environment that expects to enumerate tests, present a UI to the user, and then run specific tests).

Taking the case of explicit tests as example, I generally tend to think it's a better idea to flip this around. Instead of xunit determining where it is running and taking decisions on how the data is presented, IMO it's a better idea for xunit to have better ways to "express" the data it has (e.g, explicit tests), and once you have a way to express your data more precisely, it's then up to each client (IDE, dotnet test, etc) to present that data in the most suitable way. So the responsibility of how the data is represented is client responsibility. I'll need to also discuss with the rest of the team though, but does this approach sound reasonable to your scenarios?

When running interactively, we want to pre-enumerate all test rows, so the user can run individual test rows. When running non-interactively, this is a waste of time and resources and causes the test discovery to be slower.

This can impact how the data is represented in TRX and consequently in AzDO if PublishTestResults task is used, or even any custom third-party extension implementing an IDataConsumer that does some actions on test node update messages. I'm not sure it's a good idea to change whether or not theories are pre-enumerated when run in IDE vs command-line.

[bradwilson]

IMO it's a better idea for xunit to have better ways to "express" the data it has

MTP has no test node state for reporting this (only "passed", "failed", and "skipped"). That's the point of #2538.

You can ask me to invent my own state for this, though as long as 2538 is open I think it makes sense to wait. It's been open for 18 months and there's been no move to either implement or decline the issue, so I have to assume it's still under consideration (especially since there is at least one other test framework that has this exact same issue: NUnit).

But let's assume that you decline 2538 and I create a new test node state to report this. Do you believe that I would be able to get Test Explorer to implement a one-off, custom test node state? Because at the end of the day, what my users care about is how this all works in the face of Test Explorer. They don't care about MTP protocols and custom types. They just want a test that's not run to stay in the not run state in Test Explorer. Similarly, how do I get dotnet test to report this test as anything other than "still running"? We call the test "skipped" in dotnet test today because that's the only reasonable thing to do. How does dotnet test handle test node updates into states it doesn't understand? I'm guessing it silently ignores them.

This can impact how the data is represented in TRX and consequently in AzDO if PublishTestResults task is used, or even any custom third-party extension implementing an IDataConsumer that does some actions on test node update messages. I'm not sure it's a good idea to change whether or not theories are pre-enumerated when run in IDE vs command-line.

I'll give this some thought. I hadn't considered the consistency-of-TRX via IDE vs. command line as something that was important.

[bradwilson]

It's also worth pointing out that, due to the differentiation between Test Explorer (a not run test stays not run) vs. dotnet test (a not run test is marked as "skipped" so it's not considered hung) means that it's impossible for perfect TRX consistency between the two environments.

[Youssef1313]

MTP has no test node state for reporting this (only "passed", "failed", and "skipped").

Yes, what I meant is that "we" provide a new state that you can use in xunit. The new state has to be especially recognized by Test Explorer as well.

[OsirisTerje]

We (NUnit) use the DesignMode setting to detect if we are called from an Ide or not, and that info comes from the testrunner and appears in the runsettings received from the runner. It is true for IDE and not for others, and it works in MTP.

Regarding the test node state, as @bradwilson also talks about above

(especially since there is at least one other test framework that has this exact same issue: NUnit)

I can confirm we have the same issue, so "what @bradwilson said".

Regarding "explicit" tests, we are using a lot of magic to try to figure out if the call is explicit or not. We only allow strict explicit calls to run explicit tests.

In a long forgotten version of VS, the adapter was called differently when Run All was used, compared to run some selection. Knowing HOW the Test Explorer selected the tests would help here. If All tests are started, it is not likely to be an explicit call.

Knowing the intention of the user, what they try to - would be very helpful.

[bradwilson]

We (NUnit) use the DesignMode setting to detect if we are called from an Ide or not, and that info comes from the testrunner and appears in the runsettings received from the runner. It is true for IDE and not for others, and it works in MTP.

RunSettings is not supported by MTP; it's a VSTest feature. I assume you're saying "RunSettings" because you're still using your VSTest adapter with their VSTest-to-MTP bridge. We (xUnit.net) support MTP natively.

Our server mode detection mechanism, based on feedback from the TestFx team, is this:

https://github.com/xunit/xunit/blob/f73db5fbfeb1ba804af03def266b1d0486205409/src/xunit.v3.runner.inproc.console/TestingPlatform/TestPlatformTestFramework.cs#L269-L270

I assume the VSTest bridge is materializing this into RunSettings for you using similar code.

Knowing the intention of the user, what they try to - would be very helpful.

Agreed. The best we can do is say "if everything we were asked to run is explicit, then we'll run them, even without an explicit flag", but only do that when the call is from Test Explorer (in server mode, per the above code).

Despite the fact that MSTest is the sole holdout missing this feature, I expect Test Explorer will not implement support for it until and unless MSTest does.

[bradwilson]

I also feel like MTP v2 would've been a good time to add this feature, and it wasn't added, so I assume we're stuck command line parsing for the foreseeable future.

[Youssef1313]

We (NUnit) use the DesignMode setting to detect if we are called from an Ide or not, and that info comes from the testrunner and appears in the runsettings received from the runner. It is true for IDE and not for others, and it works in MTP.

If you rely on this for MTP, it's an implementation detail of Test Explorer that sets an environment variable for runsettings which sets DesignMode. While the env variable is available even for MTP (and even when not using the VSTestBridge), it's an implementation detail as DesignMode is VSTest-specific.

---

I also feel like MTP v2 would've been a good time to add this feature, and it wasn't added, so I assume we're stuck command line parsing for the foreseeable future.

I do agree. Unfortunately we didn't yet get to it.