different tests collected between workers in Python 3.5

[havok2063]

I'm trying to run my tests via xdist in python 3.5 and it fails saying different tests were collected between gw0 and gw1. It works fine in python 2.7. This is the same issue as #149 but that issue does not explain the solution.

[flub]

If it's similar to the other issue it's likely you are using parametrized tests but are using something unordered like for the parametrization. Since the different processes must collect the exact same tests you can solve this by using something ordered for the parametrization.

More generally I wonder if it would be possible to make xdist do the comparison in an unordered way instead to get rid of this annoyance? As long as the set of testids is the same between all the processes the collection ordering should not matter I would think?

[havok2063]

Yeah, it sounds like it should be pretty straightforward to have xdist do this. This would be ideal as you save the user to have to worry about this. How does xdist compare the list of testids as the processes proceed through the tests. Do they check with each other which ids have been run and skip them in their own lists? Do the lists get smaller as tests are completed or are you just comparing the full list once in the beginning that they are the same? Using sets makes it pretty easy to compare different lists. You can even do intersections, differences, or check for sub/supersets.

Is there a reason the same list is passed to all processes rather than say, you collect the tests, then divvy them up among the different processors?

[flub]

At a high level it goes somewhat like this: each worker process needs to run the collection themselves and they all report the collected tests to the primary. This is when the primary checks they are all the same (I'm not sure on the implementation details here. Once all workers have collected their tests the primary gives them a block of tests to run, they run the block of tests and report back at which point they get the next block etc until all tests are run.

I haven't looked at the implementation for a while but I suspect there is probably reason that the obvious-to-us set-comparison is not used. But it's probably worth investigating if it could be improved.

[RonnyPfannschmidt]

closing as i indeed did explain the solution and the issue with sets bring unordered vs a sorted list not being unordered