Difference between running mypy directly vs with pre-commit run -a

[nicoddemus]

Hi,

I'm seeing different output when I execute mypy directly vs running pre-commit run mypy -a. I've tried a few things, including looking at the docs and tweaking the options, but still can't really understand what's happening.

At the root of the repository all code is in Edgar/.

Running mypy:

λ mypy Edgar
... (bunch of mypy errors)
Found 13 errors in 3 files (checked 229 source files)

Running pre-commit run -a mypy:

(.env37) λ pre-commit run -a mypy
mypy.....................................................................Passed

Both commands above were executed from a clean checkout (no modified files).

The docs say -a/--all-files should run the hook against all files in the repository, for that reason I was expecting the same output from both commands.

Here's the mypy configuration in .pre-commit-config.yaml:

-   repo: https://github.com/pre-commit/mirrors-mypy
    rev: v0.782
    hooks:
    -   id: mypy
        files: ^Edgar/
        args: []

I'm using pre-commit 2.7.1.

Any idea of what the problem might be @asottile?

Cheers!

[asottile]

~generally there'll be two differences -- the first is outlined in the last bit of this (mypy in pre-commit is run in isolation so it will not have access to your dependencies unless you add them to additional_dependencies)

the second is that mypy from pre-commit is run with files as positional arguments -- there's some known bugs in mypy where it will hide / break in different ways depending on how arguments are passed to it (unfortunately) -- most of these are due to import cycles and some of the mypy plugins (namedtuples for example are particularly problematic)

so to get exactly what pre-commit runs you could do something like:

# make an isolated environment
virtualenv mypyenv
# mypyenv/bin/pip install ... (whatever is in additional dependencies)
mypyenv/bin/pip install mypy==0.782
mypyenv/bin/mypy $(git ls-files -- Edgar | grep '\.py$')

[nicoddemus]

~generally there'll be two differences -- the first is outlined in the last bit of this (mypy in pre-commit is run in isolation so it will not have access to your dependencies unless you add them to additional_dependencies)

Nailed it! 👍

In my case all the errors were related to mypy not seeing one of my dependencies. Awesome, thanks a lot! I knew I was missing something. 😁

[ramnes]

Just for reference, mirrors-mypy's default arguments can also make the output different. :)

https://github.com/pre-commit/mirrors-mypy/issues/48

[ssbarnea]

Sadly I do discover serious differences in behavior between vanilla mypy and mypy-inside-pre-commit as becoming more wildspread. Weird sometimes even running locally pre-commit and on CI does produce different results and I am not referring to the missing dependencies, that being a non issue.

For example I discovered that regardless the fact that https://github.com/jazzband/pip-tools has a pre-commit.ci enabled on it and a tox -e checkqa that also runs it, it got broken badly, with 60+ errors being reported. When running locally I even got almost instant pass but when I give mypy directories to look into, it gives totally different results.

Based on the fact that git ls-files | grep '\.py$' | mypy finds only one issue which we can ignore but running mypy . returns 62 failures which look genuine I am inclined to believe that passing individual files to mypy is totally unreliable. Instead passing folders, modules or packages seems to produce more reliable results. Thus is why on other projects I ended up using pass_filenames: false.

I wonder if it would be better to make this behavior default for mypy.

[asottile]

@ssbarnea nah you're just wrong / confused -- when you run it against all files in pip-tools mypy ends early on "duplicate module" -- in pre-commit those have been skipped so it shows all the errors

try for example:

$ mypy . --exclude tests/test_data/packages | wc -l
65

[ssbarnea]

Oops, I wanted to remove my previous comment few minutes after writing but github outage prevented me from doing so. Meanwhile I was able to narrow down the issue with pip-tools and it appears to be caused by pip 21.0 which changed lots of typing used by pip-tools. Ceiling pip to <21.0 does avoid errors with mypy, at least until someone can upgrade the code.

As pip was never pinned when running mypy under pre-commit, we now know why it failed only on some environments on not on others.

[jeroenboeye]

Another difference is that pre-commit only runs on staged/committed files. Took me a while to realize I simply had to git add some new files for pre-commit mypy to find them.