Closed
Description
The "pythoneval" tests are by far the biggest contributor to the time it takes to run our test suite. For each of 171 sample programs, we type-check them and then run them through the Python interpreter.
From discussion in #1668, running the programs is no longer important for our testing. So cut it out. This will mean
- cutting out that step in
testpythoneval.py
, - cutting out the program output (as opposed to the type-checker output) from all the test cases, and
- cutting out entirely any cases that no longer seem to have any meaningful content when they're only for the type-checker.
Some of the test cases may exercise features of typing
in interesting ways -- any of those should be moved to the typing
repo's own test suite if they aren't covered there already.
After this is done, we can probably get a big further speedup by folding the remaining test cases into just more cases for testcheck.py
, so that we're not forking a mypy subprocess for each one. Will make a separate issue for that.