Skip to content

Cut out evaluation in the "pythoneval" tests #1671

Closed
@gnprice

Description

@gnprice

The "pythoneval" tests are by far the biggest contributor to the time it takes to run our test suite. For each of 171 sample programs, we type-check them and then run them through the Python interpreter.

From discussion in #1668, running the programs is no longer important for our testing. So cut it out. This will mean

  • cutting out that step in testpythoneval.py,
  • cutting out the program output (as opposed to the type-checker output) from all the test cases, and
  • cutting out entirely any cases that no longer seem to have any meaningful content when they're only for the type-checker.

Some of the test cases may exercise features of typing in interesting ways -- any of those should be moved to the typing repo's own test suite if they aren't covered there already.

After this is done, we can probably get a big further speedup by folding the remaining test cases into just more cases for testcheck.py, so that we're not forking a mypy subprocess for each one. Will make a separate issue for that.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions