Make command line tests faster

[JukkaL]

Command line tests in test-data/unit/cmdline.test are very slow since they use full typeshed stubs and run mypy in a subprocess. It looks like many of them would be easy to migrate to be normal type checker tests, or some of then could be even regular Python unit tests. Alternatively, we could perhaps use test stubs with them.

Speeding up report generation tests may also be easy.

[mattany]

Can I take this?

[hauntsaninja]

Yes, just open a PR!

[manmartgarc]

Is this still the case? Runing pytest .\mypy\test\testcmdline.py runs in 51s for me; what would be considered fast? Also please let me know if this issue refers to something else than what runs when pytest .\mypy\test\testcmdline.py.

[hauntsaninja]

Yes, that's right / they're still slow. Due to pytest-xdist, how long they take is a function of number of cores on your machine. In CI, we have 2 cores, so pytest mypy/test/testcmdline.py -n 2 would give you a better sense of how much time these tests contribute when you open a PR

[hauntsaninja]

Oh hmm, actually I think Github Actions updated its runners to have 4 cores. That should be an easy win, #17185

[manmartgarc]

Oh hmm, actually I think Github Actions updated its runners to have 4 cores. That should be an easy win.

Thanks, but what should be an easy win? Are they not being run with four workers on the GitHub instances? It's running with 8 workers on my machine even though it has 16 logical cores; do you mean tweaking that setting?

[JelleZijlstra]

Yes, we currently run with -n 2, so with a maximum concurrency of 2. I think we set it explicitly because the value reported by os.cpu_count() on GH Actions doesn't match the actual number of cores the action has available.

[manmartgarc]

Gotcha. Edit: I see @hauntsaninja submitted a PR to set n 4

But beyond that, how can we get away from running the CLI as a sub process and still test some of the stdout from the CLI? The original comment mentions converting some of the tests into regular check tests or even unit tests. Could somebody point me to examples of both?

[hauntsaninja]

Yeah, Jukka's main idea is to just make tests that don't actually need to be run as a subprocess into other tests. For example, I think a lot of the "config file" tests in cmdline.test could be made into normal tests (e.g. move them into check-flags.test)