mypy errors do not fail pytest build when using ignore_missing_imports = true

[tkruse]

When mypy fails with an error, pytest --mypy still succeeds, when setup.cfg contains ignore_missing_imports = true. Should fail.

When setup.cfg does not contain ignore_missing_imports = true, process fails for me (as it should).

$ py.test --mypy -m mypy 
=================================================================== test session starts ===================================================================
platform darwin -- Python 3.7.3, pytest-5.3.1, py-1.8.0, pluggy-0.13.1
rootdir: /Users/tkruse/work/myapp, inifile: setup.cfg, testpaths: test
plugins: pylint-0.14.1, mypy-0.4.2, cov-2.8.1
Running mypy on 6 files... done with status 1
...
error: Argument 1... has incompatible type "str"; expected "List[str]"
...
Found 2 errors in 1 file (checked 6 source files)
=========== 6 passed, 20 deselected, 4 warnings in 1.31s

$ echo $?
0

mypy output:

$ mypy myapp --ignore-missing-imports
...
Found 2 errors in 1 file (checked 13 source files)

$ echo $?
1

[tkruse]

Note errors detected by e.g. disallow_untyped_calls still cause the build to fail, even when using ignore_missing_imports

[dmtucker]

I cannot reproduce this 🤔

• What version of mypy are you using?
  (Maybe pip freeze your environment just to cover all the bases?)
• Is the issue resolved with version 0.5.0?
• As much of an SSCCE as you can provide would be much appreciated. 😊

[tkruse]

Please see branch of this project for a small example: https://gitlab.com/tkruse/python_dummy/-/tree/pytest-mypy-bug

I produce buggy behavior like this:

# in virtualenv
$ python --version
Python 3.6.8

$ pip install -r setup-requirements.txt
Collecting pytest-runner
  Using cached pytest_runner-5.2-py2.py3-none-any.whl (6.8 kB)
Collecting pytest-mypy
  Downloading pytest_mypy-0.5.0-py3-none-any.whl (5.7 kB)
Collecting filelock>=3.0
Collecting pytest>=3.5; python_version >= "3.5"
Collecting mypy>=0.500; python_version >= "3.5" and python_version < "3.8"

$ py.test 
=== test session starts 
platform linux -- Python 3.6.8, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
plugins: mypy-0.5.0
...
=== mypy 

dummy/cli/dummy_main.py:11: error: Argument "description" to "ArgumentParser" has incompatible type "Optional[int]"; expected "Optional[str]"
Found 1 error in 1 file (checked 4 source files)
=== 7 passed in 0.40s 

$ echo $?
0


$ mypy dummy
dummy/cli/dummy_main.py:11: error: Argument "description" to "ArgumentParser" has incompatible type "Optional[int]"; expected "Optional[str]"
Found 1 error in 1 file (checked 4 source files)

$ echo $?
1

py.test gets --mypy from setup.cfg.

Note that I reproduced differently this time in a private project, and I did not include ignore_missing_imports because this already shows the problem, I hope.

I got the same with python 3.7. The example does not show the influence of ignore_missing_imports, because in that sample project I could reproduce the problem in a simpler way.

[dmtucker]

Thank you! This is an interesting issue...

This occurs when pytest-mypy checks file X (only) which imports file Y. mypy checks X and Y, and, if there are errors in Y, pytest-mypy has no MypyItem to report the errors with.

diff --git a/tests/test_pytest_mypy.py b/tests/test_pytest_mypy.py
index e77dcad..087aef7 100644
--- a/tests/test_pytest_mypy.py
+++ b/tests/test_pytest_mypy.py
@@ -179,3 +179,16 @@ def test_pytest_collection_modifyitems(testdir, xdist_args):
     result = testdir.runpytest_subprocess('--mypy', *xdist_args)
     result.assert_outcomes(passed=1)
     assert result.ret == 0
+
+
+def test_mypy_indirect(testdir, xdist_args):
+    """Verify that uncollected files checked by mypy cause a failure."""
+    testdir.makepyfile(bad='''
+        def myfunc(x: int) -> str:
+            return x * 2
+    ''')
+    testdir.makepyfile(good='''
+        import bad
+    ''')
+    result = testdir.runpytest_subprocess('--mypy', *xdist_args, 'good.py')
+    assert result.ret != 0

This is somewhat related to #67 in that the problem stems from mypy checking files not collected by pytest; however, I see this as a bug, not an unsupported feature.

The only solution I can see is to make checking the exit status its own test.