runtests.py looks like it's hanging at first (though it's not)

[gnprice]

Starting our test suite, it looks like this and stays this way for many seconds:

$ time ./runtests.py 
PARALLEL 4
SUMMARY  151 tasks selected
passed 0, failed 0, pending 150; running 1                                      

When @Nurdok saw this on first trying to run tests on his Windows laptop, he figured the tests were hanging -- and when I first looked at it with him, I thought the same thing too.

There are probably two confusing things here:

• The display doesn't change for a long time, with no progress indication.
• The output says "PARALLEL 4", but then there's only 1 task "running", not 4.

Probably the first of these is most important.

The root cause of both issues is that that 1 "task" which is running is actually a py.test process, which is running many test cases and doing its own parallelism among those -- but neither the parallelism nor the ongoing progress of finishing those test cases is being reported. Before #3019 we ran the big py.test process in parallel with other tasks, some of which finished quickly, which avoided both of these.

The real fix for this will be to complete the migration from our custom runtests.py runner to plain py.test. In the meantime, we should do something simple to make this not confusing.

[JukkaL]

A simple fix might be to count the pytest run as 4 tasks (or whatever level of parallelism we are using).

[gnprice]

Yeah -- though that wouldn't help with the "nothing changes for a long time" problem, which I think is probably the bigger one.

A simple fix that would also address that part would be to revert the bits of #3019 that added a sequential parameter and started using that -- so that we run pytest alongside three other tests. Not optimal for performance, but I'm not sure the difference will be a big one. The way I'd start is to try that, time it before and after, and see if it's OK.

[gvanrossum]

I wonder if the README should focus more on running pytest and less on runtests.py. That's the advice I give beginners. pytest is something they already know, and it's easy to select tests and to run pdb on tests (once you have discovered pytest -n0).

[JukkaL]

We should probably also mention pytest -n0 in the docs.

[gnprice]

Agreed on all counts re the README. I'll open an issue or two for those.

[pkch]

I thought @JukkaL didn't want to run pytest concurrently with other tests because of the very large peak memory usage.

Other than that, the performance impact is not necessarily on the total time, but on the responsiveness of the system while the tests are running: with too many CPU-intensive tasks running concurrently, Windows lags on any user operations. I'm guessing Linux handles it better?

Regardless, do we want to pass through pytest progress bar to the console?

[JelleZijlstra]

For what it's worth, I've also noticed Mac OS becoming unresponsive while I run the mypy test suite.

[JukkaL]

@pkch Earlier we had multiple pytest runs running in parallel (each with multiple processes) and that caused Travis build failures due to very high memory use. Just running a single parallel pytest run together with other, single-process tasks might work, though I share the concern that running many processes will likely cause system unresponsiveness.