Running mypy only on certain files

[brianmaissy]

I have a large codebase and I would like to run mypy only on a subset of it.

I have the globs to check defined in the files setting in my mypy.ini file, but since the pytest-mypy plugin explicitly passes a list of all of the files, that behavior is overridden.

What do you think of adding a command line argument which turns off the behavior of passing the filenames to mypy, or lets me override it, or something to that effect?

[dmtucker]

Thanks for bringing this up!

My first thought is if we took this out:
https://github.com/dbader/pytest-mypy/blob/ab7505e785d6bf44b246501a336bb035ffb9027c/src/pytest_mypy.py#L55
You could achieve this by removing all MypyItems in pytest_collection_modifyitems (in conftest.py).

An option (e.g. --mypy-no-collect) could be appropriate too, though.

One thing I never liked was that mypy_argv isn't actually like argv precisely because it doesn't include the path args... So another option could be to add collected items to mypy_argv on the fly. That would allow hooking in and customizing too.

[brianmaissy]

If we're just trying to create a workaround, something that requires users to implement pytest_collection_modifyitems would work, but I think it would be a lot more usable for the average user if there was a more straightforward way of doing it.

By --mypy-no-collect do you mean that pytest-mypy actually wouldn't create the MypyItems? Or that it would create them but not pass their paths to mypy.api.run()? Both are a bit problematic. The former would require the user to do their own collection, which goes back to my first point. The latter would create items and then have them pass trivially, which creates misleading pytest output.

I'm tempted to create an option which would run mypy without parameters, and then somehow create the test items afterwards, according to mypy's output, but that would cripple the power of integrating with pytest in the first place.

So I think the best solution would be to define a --mypy-files option, whose semantics would be identical to the files setting in my mypy.ini file, and which would modify the behavior of the plugin's collection. Then it would go on to pass the paths as it does now. Does that sounds reasonable?

[dmtucker]

By --mypy-no-collect do you mean that pytest-mypy actually wouldn't create the MypyItems? Or that it would create them but not pass their paths to mypy.api.run()?

I meant the former, but I forgot that the items are needed for, you know, the whole second-half of the plugin to function 😅

Does that sounds reasonable?

I've spent some time with this over the past few days, but I'm still not sure which direction to go with it. It's a pretty wonky thing conceptually. In some sense, the point of this plugin is to let pytest handle collection 🤔

On that note, why do you want pytest-mypy specifically to do this (e.g. instead of just running mypy directly)?
(Honest question... I just want to make sure I'm not missing something e.g. a use case, etc.)

You mentioned a solution where files to check are explicitly provided to pytest, but I'll note that that solution is very similar to just invoking pytest twice, once with pytest-mypy disabled:

pytest
pytest --mypy -m mypy the.stuff that/goes/ in/the/files.setting

It could be argued that it doesn't actually solve the problem (enabling the files setting in mypy.ini), though.
An alternate solution that occurred to me is to just xfail/skip the files that haven't been ported yet:

PATH_WHITELIST = [
    'the.stuff',
    'that/goes/',  # Note that this won't recurse automatically.
    'in/the/files.setting',
]

def pytest_collection_modifyitems(session, config, items):
    plugin = config.pluginmanager.getplugin('mypy')
    for item in items:
        if all([
                isinstance(item, plugin.MypyItem),
                item.name not in PATH_WHITELIST,
        ]):
            item.add_marker(pytest.mark.xfail)

Again, though, this doesn't enable the files setting in mypy.ini...

In exploring this issue, I created what I think is a somewhat reasonable solution that does enable that setting; however, I'm not sure we should merge it. It is available here and works like this:

• Use --mypy-files to prevent the plugin from specifying collected files to mypy. This is required:

  files (comma-separated list of strings)

  A comma-separated list of paths which should be checked by mypy if none are given on the command line...

• When --mypy-files is provided, the plugin can't tell the difference between a file that has no errors and a file that wasn't checked (as you mentioned), so emit a warning for any collected file that is not associated with any mypy errors.

  MypyWarning: No mypy errors were detected in this file, but since --mypy-files does not require mypy to check files collected by pytest, pytest-mypy cannot be sure that it was actually checked.

• When --mypy-files is provided, inject a test of the mypy exit status. This is necessary to make pytest fail if a file that was not collected by pytest is specified in the files setting and has errors:

# mypy.ini

[mypy]
files = src/foo.py

# src/foo.py

x = 5
x = 'a'

$ venv/bin/pytest --mypy-files tests/test_pytest_mypy.py 
=========================================================== test session starts ===========================================================
platform darwin -- Python 3.5.7, pytest-5.3.2, py-1.8.1, pluggy-0.13.1
Using --randomly-seed=1577738929
rootdir: /Users/dmtucker/Projects/pytest-mypy
plugins: cov-2.5.1, randomly-2.1.1, mypy-0.4.2
collected 11 items                                                                                                                        

Running mypy... done with status 1
src/foo.py:2: error: Incompatible types in assignment (expression has type "str", variable has type "int")
Found 1 error in 1 file (checked 1 source file)

. F                                                                                                                                 [  8%]
tests/test_pytest_mypy.py ...........                                                                                               [100%]

================================================================ FAILURES =================================================================
______________________________________________________________ test session _______________________________________________________________
mypy exited with status 1.
============================================================ warnings summary =============================================================
tests/test_pytest_mypy.py
  /Users/dmtucker/Projects/pytest-mypy/src/pytest_mypy.py:160: MypyWarning: No mypy errors were detected in this file, but since --mypy-files does not require mypy to check files collected by pytest, pytest-mypy cannot be sure that it was actually checked.
    'No mypy errors were detected in this file,'

-- Docs: https://docs.pytest.org/en/latest/warnings.html
================================================ 1 failed, 11 passed, 1 warning in 25.78s =================================================
2019-12-30T12:49:15-0800 exit 1

The solution works this way because I agree that creating test items after collection is not viable. I'm reluctant to merge it because it would mean significant code churn and maintenance burden for a use case that I'm not sure is very common. That said, I'm interested in your thoughts!

[brianmaissy]

Excellent analysis, and I appreciate the effort you went to in implementing a potential solution.

To answer your question about my use case, the only real reason I wanted to do this, was that I thought that if I'm already running my tests with pytest, and I want to run mypy as part of my CI, it would be convenient if it got run as a part of the test suite just to simplify things (single entry-point to run all the checks, I get the output of failures as part of the same junit xml, which is then parsed by my CI tool, etc).

But as you correctly noted, I wanted pytest to delegate the collection stage to a separate tool, which I agree is a large enough aberration from the normal workflow to make this use case fall in the "not worth supporting" category.

A decent compromise is to write a single test which just runs mypy and then either passes or fails depending on its return code, and passes through its output so that it appears in the test report (which doesn't really require a plugin). Violations in different files don't appear as separate tests, but that was never really part of the requirement anyway.

And if it was, multiple invocations of pytest would probably be the best way to achieve that, at the expense of duplicating the configuration of which files to run mypy on.

All in all, I agree that this use case is strange enough, and reasonable enough alternatives exist, to call this an interesting exercise and close the issue. Thanks again for your help!