Test Unit Execution Order and Grouping

[lgreenlee]

It would be most excellent if we were able to order the execution of the facts or theories in a file of some kind on Linux. Preferably with some UI items for configuration under VS Code.

Additionally, it would be awesome if these sequences could be grouped together and run individually easily both from the IDE and from the command line.

[bradwilson]

We have no intention of supported test ordering natively. We strongly believe that unit tests have maximum value when run in a random order, so as not to create unintentional dependencies between tests.

[lgreenlee]

Thanks for the quick response!

So, a question then:

I have a process that is composed of say, 5 methods calls. Method one is the download of a file that is time coded and must be downloaded "now". The file name cannot be hard coded since it is different over time. The remaining four calls operate on the downloaded file in some way, but some also move its location. The path to the file has a time component as well that is dependent on the initial call.

The question then is how do you test this without ordering and receive a result that tells you your code works? Also, we want to be efficient here and not write extra tests that aren't really needed.

Code structure aside - from a development efficiency perspective you are basically forcing me to run the entire set of tests as xunit currently works, save one. I actually use the unit tests to make sure that I am not breaking things as I go along. Usually, I only need to run a handful of tests, on a select set of dependent methods in a class. Hence, the need for grouping and ordering.

Thank you!

[Clockwork-Muse]

@lgreenlee - You're writing your tests wrong, then.
For one thing, anything with a time component is problematic, because these are often difficult or impossible to modify without mucking things up.... unless you do something like injecting a clock (eg, NodaTime provides clocks you can inject and modify for testing).

Next, you say your process is composed of five methods... why are you testing those separately for the process? You should be doing two things here:

1. Testing all five methods independently, and not pushing the results of one into another. These tests should be randomly orderable.
2. Testing your process, which is presumably a single method itself, as a single method. This test has no relation to the previous tests, but should obviously produce the results you expect.

If you can't write the tests this way, you probably need to push a dependency into a method (for instance, the time, file name, or File handle). Once you have that, it becomes easier to mock or otherwise provide testing objects.

[ldiary]

We have no intention of supported test ordering natively. We strongly believe that unit tests have maximum value when run in a random order, so as not to create unintentional dependencies between tests.

It's a sad thing for people like me (currently working on UI Automation of Winforms and uses xunit to run tests) who desperately need an orderly execution of test cases. Fortunately, there is still a way to do it. I've tested this one and it works as desired.

[Clockwork-Muse]

@ldiary - I think you may have missed my earlier point. Each individual test should be independent. The fact you require them to be ordered really means you have a "hidden" dependency - in this case, probably something that should be captured as setup for the tests, not as a test itself (I'm assuming that you're doing things this way to do something like test 1 is for the signin page, test 2 is for rejection, test 3 is for signed in behavior, etc). The idea would be to setup a page and capture the results of each individual action/response. For one thing, that would allow 'later' tests to confirm success independent of an 'earlier' tests failure... your later tests do fail if the earlier ones do, right? (sure, it fails stuff because it Asserts the state of the runner, but that's going to result in the wrong messages) The fact that there's only going to be one ordered runner is also likely to cause problems...

I don't know that xunit is really designed for UI automation tests, but I feel more what's required is some sort of path/flow testing framework, able to capture results and split at each branch point or something. Possibly doable with some sort of Before/After attributes or something, with a test that returns something (or captures the state of the method or something, who knows - pull it apart the CodeDom). Might be better as another library though...

[ldiary]

@Clockwork-Muse Yes, you are absolutely right in all angles, except for the fact that my test dependencies are not "hidden". Usually in my (UI Test Automation) case, dependencies are encouraged because we are testing on user's perspective (i.e. blackbox testing, user acceptance testing, etc). As such, if one of our test case fail, continuing execution of the rest of the test cases has no more meaning whatsoever... it's just a waste of our time and resources.

In short, you are right, we are not supposed to be using xunit which is not designed for that. In Python, tools like the RobotFramework best suits our needs... but my new project is UI Automation of a legacy Winform application and I simply want to get started quickly. Few months later I wouldn't be using xunit anymore... I think.

[bradwilson]

Closed for age.