Description
We aren't running the CPython test suite regularly. We've had a handful of issues recently that would have been caught by the test suite, notably #622.
We advertise the test runner (and copy in the top-level runner which was removed in recent CPython versions) in PYTHON.json, and we have a runner for the runner, but we don't actually run it in CI.
The only thing we run is that the PGO profiling portion of the upstream CPython Makefile runs a subset of the upstream test suite, and so we've had to patch a few of the tests out.
Almost certainly we will have some test failures in the remainder of the test suite.
docs/status.rst has some notes on test failures, but it hasn't been updated in several years.
Let's start by updating that, marking known test failures somehow, and putting this in CI in some fashion. I don't know if it should run on every PR and push to HEAD given how much CI we already expend. Maybe nightly + on release?