Significant speed degradation from 0.931 to 0.941

[brianmedigate]

Bug Report

I upgraded from 0.931 to 0.941, and the time it takes to run a full run without a cache on my codebase increased from less than 2 minutes to almost 12 minutes.

I was surprised not to find any issues about this yet, maybe I'm just a relatively early adopter, or maybe something about my configuration or codebase is idiosyncratic, although nothing obvious that I can see.

To Reproduce

Not sure exactly how to give enough information to reproduce it, but let's just start with asking if you are aware of an expected performance regression in this version, and if so, if there are any known factors that exacerbate it.

Your Environment

Mypy 0.941 (appears to be the same on 0.940)
Python 3.8.10
Ubuntu 20.04.

An extract of the most salient parts of my configuration:

[mypy]
follow_imports = silent
ignore_missing_imports = True
check_untyped_defs = True
disallow_untyped_defs = True
no_implicit_optional = True
strict_equality = True
warn_redundant_casts = True
python_version = 3.8
platform = linux
plugins = pydantic.mypy
namespace_packages = True

My first suspect was the pydantic plugin but I removed it, to no effect.

[KotlinIsland]

Can you verify that you are using a compiled wheel? run mypy -v . and look for:LOG: Compiled: True

[brianmedigate]

yes

LOG:  Cache Dir:              .mypy_cache
LOG:  Compiled:               True

[KotlinIsland]

The only other think I could recommend is to try some of the other 0.940+dev wheels here: https://github.com/mypyc/mypy_mypyc-wheels and see if you have any luck or can track down where about the slowdown started.

[JukkaL]

This is the first report about a performance regression in 0.941 I remember hearing about, so it seems likely that something in your codebase specifically triggers the regression.

@KotlinIsland's suggestion is a good one. You can also run mypy -v to get more verbose logging and see if most of the time is spent processing a specific module, or whether things are slower overall.

[twoertwein]

Pandas did not see a slowdown: Pandas updated from 0.931 to 0.941 (and at the same time also updated pyright). I'm not certain how much variance there is between github action runs but before mypy+pyright took 4m 27s and afterwards 2m 35s.

[brianmedigate]

Looks like it started being slow in this wheel: https://github.com/mypyc/mypy_mypyc-wheels/releases/tag/v0.940%2Bdev.6c1eb5b36b83ecf959a50100d282ca86186f470f.
Here is the relevant commit (to mypyc): 6c1eb5b
And the commit message mentions:

this slows down the richards benchmark by about 15%

Sounds like it might be relevant?
kudos to @JukkaL for the excellent commit message btw!

I ran with --verbose, and there was a visibly slow SSC of size 111 which took a couple minutes to process, and a couple of visibly slow singletons, but nothing obviously non-proportional to their speeds in the faster version.

Any advice on how to proceed?

[JukkaL]

Any advice on how to proceed?

I can see how that PR could affect performance a little, but over 5x performance loss is much worse than I'd expect. Can you double check a few additional wheels before/after the commit to get more confidence that this is the root cause?

Another idea would be to try running mypy on Python 3.10, and maybe on some different hardware, to investigate if the regression is repeatable or specific to certain configuration.

If the slowdown is consistent, I'd suggest profiling the mypy run to see whether there is some particular function causing the regressions, or whether things are slow overall. I can give instructions if you want to try this (it's not difficult).

As mentioned in the commit message, the "always-defined attributes" optimization could help with performance. I have this close to ready for merging. I could focus on getting it finished, and perhaps you could check performance with this optimization. This will probably take at least one or two weeks, however.

[brianmedigate]

I did a sort of binary search, and everything before it was similar to 0.931, and everything after it was similar to 0.940.
I don't have access to much different hardware, but it replicates on a couple of laptops and in our CI.
I just tried it with python 3.10 and it's about the same.

Yeah I'd be happy to try profiling it, and to try it with the optimization. Thanks!

[brianmedigate]

I tried to find a subset of my code which takes a disproportionate amount of time, but unfortunately it is very interconnected, so it's difficult to tease the parts apart. So far I didn't find anything obviously disproportionate.

[KotlinIsland]

Is the code public? I'm pretty comfortable minifying issues and would be happy to lend a hand.

[brianmedigate]

Unfortunately no, sorry