In-assembly parallel execution

[pvlakshm]

Description

Enable tests runs to complete faster by allowing tests within a single assembly to execute in parallel.

Please see RFC here: 004-In-Assembly-Parallel-Execution

[AbhitejJohn]

Here is the RFC for this. Feedback is welcome.

[andersforsgren]

Should parallel test execution not (at least optionally) be possible for non parallel-ready code? A container with 1000 tests that can't run in parallel can be divided in chunks and run sequentially in N appdomains/processes.

[rajbos]

We have been using ‪ https://github.com/sscobici/ParallelTestRunner for this for a while now. Can split on Class level, Method level and group tests via annotations. Would be very nice to have this build in, so there is no longer a need to run an executable to split the tests, run them and then summarize the testresults.

[AdamDotNet]

I'm all for letting tests run in parallel, especially since it doesn't seem to work right now in MSTestV2, at least in Visual Studio 2015. The RFC linked mentions VSTest is capable of allowing all test providers to do test parallelization by spawning multiple processes.

I know this feature is for in assembly parallelization, but for my sake, can someone confirm to me that regular old VSTest multiprocess parallelization isn't meant to work right now in MSTestV2? I would appreciate the clarity. Thank you!

[AbhitejJohn]

@AdamDotNet : The container level parallelization for vstest works irrespective of the Framework(mstestv2/mstest/xunit/nunit). If your solution has multiple test projects, it should spawn a host process for each test container and run them in parallel when enabled just like this blog details. This only works if there is more than one test container. You aren't seeing that happen in VS 2015?

[AdamDotNet]

@AbhitejJohn Thanks for the information. I never realized that multiprocess parallelism was dependent on having multiple projects. I just observed it working in a sample project with two test projects.

Back to this feature of in assembly parallelism, if I'm reading the RFC right, you can mark an assembly with how you want it to handle parallelism in that assembly. I'm imagining in my solution, it would be generally acceptable to mark the assembly as parallel by class by default, but it would be nice if certain classes could run faster by running the test methods in a particular class in parallel.

So, will it be possible to default an assembly to class level parallelization, but you can override that behavior by placing attributes on a specific class to enable method parallelization for that test class? Will this work with only one test project?

To illustrate, I have one test project with the desire for the following behavior:

• Classes that contain tests that are totally in memory using Moq: run as class parallelization because each test takes only a handful of milliseconds to run.
• Classes that contain a number of tests that integrate with other pieces: run as method parallelization because each test takes about 5 seconds to run.

[AbhitejJohn]

@AdamDotNet : That's a great point. This is essentially a mix and match of multiple modes in the same assembly. Our thought process was that integration/End-to-End tests would actually be split into a different assembly from unit tests, whose test mode can then be configured accordingly.

That said, for the scenario described above, as long as the fast running tests within a test class do not interfere with each other, the mode for the assembly can be made method level as a whole. That ensures that all methods irrespective of what class they are under are run in parallel.

However this does look like a valid ask and could surface more when we add in custom modes. The only choice that the RFC provides today is to opt-out of the default mode for an assembly. We hope that these scenarios however are relatively fewer in number and come under a more advanced set of capabilities that we can add in the future. Do let us know otherwise.

[JoshCollinsMSFT]

This is something that is incredibly frustrating to my team. This seems like a regression in behavior from the parallelization control we got from the testsettings file. In fact, my team is still currently are using a testsettings file to get that level of parallelization of our tests because one of our test categories takes around 3 hours to complete without it, and only 20 minutes to complete with it (with a setting of parallelTestCount="18").

Unfortunately we're banging our heads against this problem right now because without using a runsettings file we can't easily use test parameters, and using a runsettings file we can't get parallel execution.

The sooner we can get support for this in MSTest while using a runsettings file the better.

[AbhitejJohn]

Thanks for letting us know @JoshCollinsMSFT . We are in the phase of collecting feedback on the RFC that Pratap posted above. Do let us know if that meets your requirements. We plan to start off on this in the weeks to come.

[JoshCollinsMSFT]

@AbhitejJohn How do we provide feedback to the RFC?