--use-fine-grained-cache for dmypy hangs

[rggjan]

According to the docs (https://mypy.readthedocs.io/en/stable/additional_features.html?highlight=caching#caching-with-mypy-daemon) enabling the fine grained cache should help speed up initial mypy runs. However, this doesn't seem to be working, and dmypy seems to hang when using it:

# Slow initial run (mypy):
❯ rm -r .mypy_cache
❯ time mypy --cache-fine-grained .
mypy --cache-fine-grained .  82.04s user 9.06s system 97% cpu 1:33.69 total

# Faster cached run (mypy)
❯ time mypy --cache-fine-grained .
mypy --cache-fine-grained .  45.68s user 6.77s system 95% cpu 54.732 total

# Slow initial run (daemon)
❯ dmypy stop
❯ time dmypy run .
dmypy run .  0.32s user 0.31s system 0% cpu 1:23.13 total

# Faster cached run (daemon)
❯ time dmypy run .
dmypy run .  0.08s user 0.06s system 12% cpu 1.124 total

# Initial run using mypy cache (daemon)
❯ dmypy stop
❯ time dmypy run . -- --use-fine-grained-cache
# Nothing happening for at least 10 minutes...

[msullivan]

Hm, interesting. Could you post the full command lines you are running?

And could you try running something like dmypy run --log-file debug.log -- --verbose --use-fine-grained-cache <check directory> and seeing what happens?

Also, I'm a little surprised that the cached run is as slow as it is; is that with no changes made since the previous run?

[msullivan]

Oh, argh, something that you need to do in order to properly create a cache that can be used by dmypy, but which we neglected to document, is to pass --local-partial-types when building the cache. Otherwise the daemon will reject the cache, since the daemon needs local-partial-types. (I'm going to go make this implied by --cache-fine-grained.)

[rggjan]

Hm, interesting. Could you post the full command lines you are running?

I did post the full command lines... this was copied directly from my terminal.

Also, I'm a little surprised that the cached run is as slow as it is; is that with no changes made since the previous run?

Yes, no changes between the previous run, I ran it immediately after each other, as shown above...

[rggjan]

I did post the full command lines... this was copied directly from my terminal.

Oh, except I forgot the --use-fine-grained-cache for the hanging dmypy. Fixed that above.

[rggjan]

OK, now I get a different issue (though also my code changed, so maybe not directly related to the added --local-partial-types):

❯ time mypy --cache-fine-grained --local-partial-types .
Success: no issues found in 2905 source files
mypy --cache-fine-grained --local-partial-types .  95.99s user 13.92s system 93% cpu 1:58.18 total

❯ time mypy --cache-fine-grained --local-partial-types .
Success: no issues found in 2905 source files
mypy --cache-fine-grained --local-partial-types .  28.02s user 6.40s system 88% cpu 38.869 total

❯ time dmypy run .
Daemon started
Success: no issues found in 4436 source files
dmypy run .  0.33s user 0.31s system 0% cpu 1:42.23 total

❯ time dmypy run .
Success: no issues found in 2905 source files
dmypy run .  0.09s user 0.06s system 15% cpu 0.918 total

❯ dmypy stop
Daemon stopped

❯ time dmypy run . -- --use-fine-grained-cache
Daemon started
Daemon crashed!
Traceback (most recent call last):
  File "mypy/dmypy_server.py", line 221, in serve
  File "mypy/dmypy_server.py", line 264, in run_command
  File "mypy/dmypy_server.py", line 323, in cmd_run
  File "mypy/dmypy_server.py", line 380, in check
  File "mypy/dmypy_server.py", line 458, in initialize_fine_grained
  File "mypy/dmypy_server.py", line 545, in fine_grained_increment_follow_imports
  File "mypy/dmypy_server.py", line 679, in find_reachable_changed_modules
KeyError: 'foo.bar.baz'

dmypy run . -- --use-fine-grained-cache  0.33s user 0.31s system 4% cpu 15.230 total

[rggjan]

The KeyError seems to go away when doing:

touch foo/bar/baz/__init__.py

so this is probably a different bug, due to having namespace_packages = True?

In any case, now I get:

❯ dmypy stop
Daemon stopped

❯ time dmypy run . -- --use-fine-grained-cache
Daemon started
Success: no issues found in 4305 source files
dmypy run . -- --use-fine-grained-cache  0.34s user 0.34s system 3% cpu 19.051 total

🚀