Description
Feature or enhancement
This proposes adding an option to the regression test runner to run individual tests multiple times in parallel with the goal of uncovering multithreading bugs, especially in the free threading build.
Note that this is different from -j, --multiprocess, which uses multiple processes to run different tests files in parallel. The motivation of -j is to speed up the time it takes to run tests and the use of processes improves isolation. Here the goal is to detect thread-safety bugs (at the cost of extra time).
This is motivated by the experience of using https://github.com/Quansight-Labs/pytest-run-parallel, which is a pytest plugin written by @lysnikolaou and @andfoy. We've used that effectively to find thread-safety bugs in C API extensions while working on free threading compatibility.
The proposed changes are limited to the Python-internal test.regrtest and test.support package and not the public unittest package.
New command line arguments
These are chosen to match https://github.com/Quansight-Labs/pytest-run-parallel. I think using the same names across the Python ecosystem makes things a bit easier to remember if you work on multiple projects.
--parallel-threads=N [default=0], runs each test in N threads--iterations=N [default=1], number of times to run each test
Other support
Some of our test cases are not thread-safe (even with the GIL) because they modify global state or use non thread-safe unsafe modules (like warnings):
@support.thread_unsafe- marks a test case as not safe to run in multiple threads. The test will be run once in the main thread (like normal) even when--parallel-threads=Nis specified.
Similar projects
There are a few existing projects that do essentially this for pytest and unittest, but they're not directly usable for Python's own test suite.
- https://github.com/Quansight-Labs/pytest-run-parallel
- https://github.com/tonybaloney/pytest-freethreaded
- https://github.com/amyreese/unittest-ft