0.9.1: pytest is failing

[kloczek]

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

• python3 -sBm build -w --no-isolation
• because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
• install .whl file in </install/prefix>
• run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pytest-mypy-0.9.1-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pytest-mypy-0.9.1-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.14, pytest-7.1.3, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/pytest-mypy-0.9.1, configfile: tox.ini, testpaths: tests
plugins: mypy-0.9.1, forked-1.4.0, xdist-2.5.0
collected 53 items

tests/test_pytest_mypy.py ....................................FFFFFF...........                                                                                      [100%]

================================================================================= FAILURES =================================================================================
____________________________________________________________________ test_mypy_indirect[True-__init__] _____________________________________________________________________

testdir = <Testdir local('/tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect0')>, xdist_args = ['-n', 'auto'], module_name = '__init__'

    @pytest.mark.parametrize(
        "module_name",
        [
            pytest.param(
                "__init__",
                marks=pytest.mark.xfail(
                    (3, 10) <= PYTEST_VERSION < (6, 2),
                    raises=AssertionError,
                    reason="https://github.com/pytest-dev/pytest/issues/8016",
                ),
            ),
            "good",
        ],
    )
    def test_mypy_indirect(testdir, xdist_args, module_name):
        """Verify that uncollected files checked by mypy cause a failure."""
        testdir.makepyfile(
            bad="""
                def pyfunc(x: int) -> str:
                    return x * 2
            """,
        )
        pyfile = testdir.makepyfile(
            **{
                module_name: """
                    import bad
                """,
            },
        )
        result = testdir.runpytest_subprocess("--mypy", *xdist_args, str(pyfile))
        mypy_file_checks = 1
        mypy_status_check = 1
>       result.assert_outcomes(passed=mypy_file_checks, failed=mypy_status_check)
E       AssertionError: assert {'errors': 0,...pped': 0, ...} == {'errors': 0,...pped': 0, ...}
E         Omitting 4 identical items, use -vv to show
E         Differing items:
E         {'passed': 2} != {'passed': 1}
E         {'failed': 0} != {'failed': 1}
E         Use -v to get more diff

/home/tkloczko/rpmbuild/BUILD/pytest-mypy-0.9.1/tests/test_pytest_mypy.py:310: AssertionError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
running: /usr/bin/python3 -mpytest --basetemp=/tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect0/runpytest-0 --mypy -n auto /tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect0/__init__.py
     in: /tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect0
============================= test session starts ==============================
platform linux -- Python 3.8.14, pytest-7.1.3, pluggy-1.0.0
rootdir: /tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect0
plugins: mypy-0.9.1, forked-1.4.0, xdist-2.5.0
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I / gw12 I / gw13 I / gw14 I / gw15 I / gw16 I / gw17 I / gw18 I / gw19 I / gw20 I / gw21 I / gw22 I / gw23 I / gw24 I / gw25 I / gw26 I / gw27 I / gw28 I / gw29 I / gw30 I / gw31 I / gw32 I / gw33 I / gw34 I / gw35 I / gw36 I / gw37 I / gw38 I / gw39 I / gw40 I / gw41 I / gw42 I / gw43 I / gw44 I / gw45 I / gw46 I / gw47 I
gw0 [2] / gw1 [2] / gw2 [2] / gw3 [2] / gw4 [2] / gw5 [2] / gw6 [2] / gw7 [2] / gw8 [2] / gw9 [2] / gw10 [2] / gw11 [2] / gw12 [2] / gw13 [2] / gw14 [2] / gw15 [2] / gw16 [2] / gw17 [2] / gw18 [2] / gw19 [2] / gw20 [2] / gw21 [2] / gw22 [2] / gw23 [2] / gw24 [2] / gw25 [2] / gw26 [2] / gw27 [2] / gw28 [2] / gw29 [2] / gw30 [2] / gw31 [2] / gw32 [2] / gw33 [2] / gw34 [2] / gw35 [2] / gw36 [2] / gw37 [2] / gw38 [2] / gw39 [2] / gw40 [2] / gw41 [2] / gw42 [2] / gw43 [2] / gw44 [2] / gw45 [2] / gw46 [2] / gw47 [2]

..                                                                       [100%]
===================================== mypy =====================================
Success: no issues found in 1 source file
============================== 2 passed in 7.70s ===============================
______________________________________________________________________ test_mypy_indirect[True-good] _______________________________________________________________________

testdir = <Testdir local('/tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect1')>, xdist_args = ['-n', 'auto'], module_name = 'good'

    @pytest.mark.parametrize(
        "module_name",
        [
            pytest.param(
                "__init__",
                marks=pytest.mark.xfail(
                    (3, 10) <= PYTEST_VERSION < (6, 2),
                    raises=AssertionError,
                    reason="https://github.com/pytest-dev/pytest/issues/8016",
                ),
            ),
            "good",
        ],
    )
    def test_mypy_indirect(testdir, xdist_args, module_name):
        """Verify that uncollected files checked by mypy cause a failure."""
        testdir.makepyfile(
            bad="""
                def pyfunc(x: int) -> str:
                    return x * 2
            """,
        )
        pyfile = testdir.makepyfile(
            **{
                module_name: """
                    import bad
                """,
            },
        )
        result = testdir.runpytest_subprocess("--mypy", *xdist_args, str(pyfile))
        mypy_file_checks = 1
        mypy_status_check = 1
>       result.assert_outcomes(passed=mypy_file_checks, failed=mypy_status_check)
E       AssertionError: assert {'errors': 0,...pped': 0, ...} == {'errors': 0,...pped': 0, ...}
E         Omitting 4 identical items, use -vv to show
E         Differing items:
E         {'passed': 2} != {'passed': 1}
E         {'failed': 0} != {'failed': 1}
E         Use -v to get more diff

/home/tkloczko/rpmbuild/BUILD/pytest-mypy-0.9.1/tests/test_pytest_mypy.py:310: AssertionError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
running: /usr/bin/python3 -mpytest --basetemp=/tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect1/runpytest-0 --mypy -n auto /tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect1/good.py
     in: /tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect1
============================= test session starts ==============================
platform linux -- Python 3.8.14, pytest-7.1.3, pluggy-1.0.0
rootdir: /tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect1
plugins: mypy-0.9.1, forked-1.4.0, xdist-2.5.0
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I / gw12 I / gw13 I / gw14 I / gw15 I / gw16 I / gw17 I / gw18 I / gw19 I / gw20 I / gw21 I / gw22 I / gw23 I / gw24 I / gw25 I / gw26 I / gw27 I / gw28 I / gw29 I / gw30 I / gw31 I / gw32 I / gw33 I / gw34 I / gw35 I / gw36 I / gw37 I / gw38 I / gw39 I / gw40 I / gw41 I / gw42 I / gw43 I / gw44 I / gw45 I / gw46 I / gw47 I
gw0 [2] / gw1 [2] / gw2 [2] / gw3 [2] / gw4 [2] / gw5 [2] / gw6 [2] / gw7 [2] / gw8 [2] / gw9 [2] / gw10 [2] / gw11 [2] / gw12 [2] / gw13 [2] / gw14 [2] / gw15 [2] / gw16 [2] / gw17 [2] / gw18 [2] / gw19 [2] / gw20 [2] / gw21 [2] / gw22 [2] / gw23 [2] / gw24 [2] / gw25 [2] / gw26 [2] / gw27 [2] / gw28 [2] / gw29 [2] / gw30 [2] / gw31 [2] / gw32 [2] / gw33 [2] / gw34 [2] / gw35 [2] / gw36 [2] / gw37 [2] / gw38 [2] / gw39 [2] / gw40 [2] / gw41 [2] / gw42 [2] / gw43 [2] / gw44 [2] / gw45 [2] / gw46 [2] / gw47 [2]

..                                                                       [100%]
===================================== mypy =====================================
Success: no issues found in 1 source file
============================== 2 passed in 7.64s ===============================
____________________________________________________________________ test_mypy_indirect[False-__init__] ____________________________________________________________________

testdir = <Testdir local('/tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect2')>, xdist_args = [], module_name = '__init__'

    @pytest.mark.parametrize(
        "module_name",
        [
            pytest.param(
                "__init__",
                marks=pytest.mark.xfail(
                    (3, 10) <= PYTEST_VERSION < (6, 2),
                    raises=AssertionError,
                    reason="https://github.com/pytest-dev/pytest/issues/8016",
                ),
            ),
            "good",
        ],
    )
    def test_mypy_indirect(testdir, xdist_args, module_name):
        """Verify that uncollected files checked by mypy cause a failure."""
        testdir.makepyfile(
            bad="""
                def pyfunc(x: int) -> str:
                    return x * 2
            """,
        )
        pyfile = testdir.makepyfile(
            **{
                module_name: """
                    import bad
                """,
            },
        )
        result = testdir.runpytest_subprocess("--mypy", *xdist_args, str(pyfile))
        mypy_file_checks = 1
        mypy_status_check = 1
>       result.assert_outcomes(passed=mypy_file_checks, failed=mypy_status_check)
E       AssertionError: assert {'errors': 0,...pped': 0, ...} == {'errors': 0,...pped': 0, ...}
E         Omitting 4 identical items, use -vv to show
E         Differing items:
E         {'passed': 2} != {'passed': 1}
E         {'failed': 0} != {'failed': 1}
E         Use -v to get more diff

/home/tkloczko/rpmbuild/BUILD/pytest-mypy-0.9.1/tests/test_pytest_mypy.py:310: AssertionError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
running: /usr/bin/python3 -mpytest --basetemp=/tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect2/runpytest-0 --mypy /tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect2/__init__.py
     in: /tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect2
============================= test session starts ==============================
platform linux -- Python 3.8.14, pytest-7.1.3, pluggy-1.0.0
rootdir: /tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect2
plugins: mypy-0.9.1, forked-1.4.0, xdist-2.5.0
collected 2 items

__init__.py ..                                                           [100%]
===================================== mypy =====================================

Success: no issues found in 1 source file
============================== 2 passed in 4.01s ===============================
______________________________________________________________________ test_mypy_indirect[False-good] ______________________________________________________________________

testdir = <Testdir local('/tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect3')>, xdist_args = [], module_name = 'good'

    @pytest.mark.parametrize(
        "module_name",
        [
            pytest.param(
                "__init__",
                marks=pytest.mark.xfail(
                    (3, 10) <= PYTEST_VERSION < (6, 2),
                    raises=AssertionError,
                    reason="https://github.com/pytest-dev/pytest/issues/8016",
                ),
            ),
            "good",
        ],
    )
    def test_mypy_indirect(testdir, xdist_args, module_name):
        """Verify that uncollected files checked by mypy cause a failure."""
        testdir.makepyfile(
            bad="""
                def pyfunc(x: int) -> str:
                    return x * 2
            """,
        )
        pyfile = testdir.makepyfile(
            **{
                module_name: """
                    import bad
                """,
            },
        )
        result = testdir.runpytest_subprocess("--mypy", *xdist_args, str(pyfile))
        mypy_file_checks = 1
        mypy_status_check = 1
>       result.assert_outcomes(passed=mypy_file_checks, failed=mypy_status_check)
E       AssertionError: assert {'errors': 0,...pped': 0, ...} == {'errors': 0,...pped': 0, ...}
E         Omitting 4 identical items, use -vv to show
E         Differing items:
E         {'passed': 2} != {'passed': 1}
E         {'failed': 0} != {'failed': 1}
E         Use -v to get more diff

/home/tkloczko/rpmbuild/BUILD/pytest-mypy-0.9.1/tests/test_pytest_mypy.py:310: AssertionError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
running: /usr/bin/python3 -mpytest --basetemp=/tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect3/runpytest-0 --mypy /tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect3/good.py
     in: /tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect3
============================= test session starts ==============================
platform linux -- Python 3.8.14, pytest-7.1.3, pluggy-1.0.0
rootdir: /tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect3
plugins: mypy-0.9.1, forked-1.4.0, xdist-2.5.0
collected 2 items

good.py ..                                                               [100%]
===================================== mypy =====================================

Success: no issues found in 1 source file
============================== 2 passed in 4.01s ===============================
____________________________________________________________________ test_mypy_indirect[None-__init__] _____________________________________________________________________

testdir = <Testdir local('/tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect4')>, xdist_args = ['-p', 'no:xdist'], module_name = '__init__'

    @pytest.mark.parametrize(
        "module_name",
        [
            pytest.param(
                "__init__",
                marks=pytest.mark.xfail(
                    (3, 10) <= PYTEST_VERSION < (6, 2),
                    raises=AssertionError,
                    reason="https://github.com/pytest-dev/pytest/issues/8016",
                ),
            ),
            "good",
        ],
    )
    def test_mypy_indirect(testdir, xdist_args, module_name):
        """Verify that uncollected files checked by mypy cause a failure."""
        testdir.makepyfile(
            bad="""
                def pyfunc(x: int) -> str:
                    return x * 2
            """,
        )
        pyfile = testdir.makepyfile(
            **{
                module_name: """
                    import bad
                """,
            },
        )
        result = testdir.runpytest_subprocess("--mypy", *xdist_args, str(pyfile))
        mypy_file_checks = 1
        mypy_status_check = 1
>       result.assert_outcomes(passed=mypy_file_checks, failed=mypy_status_check)
E       AssertionError: assert {'errors': 0,...pped': 0, ...} == {'errors': 0,...pped': 0, ...}
E         Omitting 4 identical items, use -vv to show
E         Differing items:
E         {'passed': 2} != {'passed': 1}
E         {'failed': 0} != {'failed': 1}
E         Use -v to get more diff

/home/tkloczko/rpmbuild/BUILD/pytest-mypy-0.9.1/tests/test_pytest_mypy.py:310: AssertionError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
running: /usr/bin/python3 -mpytest --basetemp=/tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect4/runpytest-0 --mypy -p no:xdist /tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect4/__init__.py
     in: /tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect4
============================= test session starts ==============================
platform linux -- Python 3.8.14, pytest-7.1.3, pluggy-1.0.0
rootdir: /tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect4
plugins: mypy-0.9.1, forked-1.4.0, xdist-2.5.0
collected 2 items

__init__.py ..                                                           [100%]
===================================== mypy =====================================

Success: no issues found in 1 source file
============================== 2 passed in 4.02s ===============================
______________________________________________________________________ test_mypy_indirect[None-good] _______________________________________________________________________

testdir = <Testdir local('/tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect5')>, xdist_args = ['-p', 'no:xdist'], module_name = 'good'

    @pytest.mark.parametrize(
        "module_name",
        [
            pytest.param(
                "__init__",
                marks=pytest.mark.xfail(
                    (3, 10) <= PYTEST_VERSION < (6, 2),
                    raises=AssertionError,
                    reason="https://github.com/pytest-dev/pytest/issues/8016",
                ),
            ),
            "good",
        ],
    )
    def test_mypy_indirect(testdir, xdist_args, module_name):
        """Verify that uncollected files checked by mypy cause a failure."""
        testdir.makepyfile(
            bad="""
                def pyfunc(x: int) -> str:
                    return x * 2
            """,
        )
        pyfile = testdir.makepyfile(
            **{
                module_name: """
                    import bad
                """,
            },
        )
        result = testdir.runpytest_subprocess("--mypy", *xdist_args, str(pyfile))
        mypy_file_checks = 1
        mypy_status_check = 1
>       result.assert_outcomes(passed=mypy_file_checks, failed=mypy_status_check)
E       AssertionError: assert {'errors': 0,...pped': 0, ...} == {'errors': 0,...pped': 0, ...}
E         Omitting 4 identical items, use -vv to show
E         Differing items:
E         {'passed': 2} != {'passed': 1}
E         {'failed': 0} != {'failed': 1}
E         Use -v to get more diff

/home/tkloczko/rpmbuild/BUILD/pytest-mypy-0.9.1/tests/test_pytest_mypy.py:310: AssertionError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
running: /usr/bin/python3 -mpytest --basetemp=/tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect5/runpytest-0 --mypy -p no:xdist /tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect5/good.py
     in: /tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect5
============================= test session starts ==============================
platform linux -- Python 3.8.14, pytest-7.1.3, pluggy-1.0.0
rootdir: /tmp/pytest-of-tkloczko/pytest-1000/test_mypy_indirect5
plugins: mypy-0.9.1, forked-1.4.0, xdist-2.5.0
collected 2 items

good.py ..                                                               [100%]
===================================== mypy =====================================

Success: no issues found in 1 source file
============================== 2 passed in 4.01s ===============================
========================================================================= short test summary info ==========================================================================
FAILED tests/test_pytest_mypy.py::test_mypy_indirect[True-__init__] - AssertionError: assert {'errors': 0,...pped': 0, ...} == {'errors': 0,...pped': 0, ...}
FAILED tests/test_pytest_mypy.py::test_mypy_indirect[True-good] - AssertionError: assert {'errors': 0,...pped': 0, ...} == {'errors': 0,...pped': 0, ...}
FAILED tests/test_pytest_mypy.py::test_mypy_indirect[False-__init__] - AssertionError: assert {'errors': 0,...pped': 0, ...} == {'errors': 0,...pped': 0, ...}
FAILED tests/test_pytest_mypy.py::test_mypy_indirect[False-good] - AssertionError: assert {'errors': 0,...pped': 0, ...} == {'errors': 0,...pped': 0, ...}
FAILED tests/test_pytest_mypy.py::test_mypy_indirect[None-__init__] - AssertionError: assert {'errors': 0,...pped': 0, ...} == {'errors': 0,...pped': 0, ...}
FAILED tests/test_pytest_mypy.py::test_mypy_indirect[None-good] - AssertionError: assert {'errors': 0,...pped': 0, ...} == {'errors': 0,...pped': 0, ...}
================================================================= 6 failed, 47 passed in 334.51s (0:05:34) =================================================================

Here is list of installed modules in build env

Package           Version
----------------- --------------
appdirs           1.4.4
attrs             22.1.0
Brlapi            0.8.3
build             0.8.0
codespell         2.2.1
cssselect         1.1.0
cycler            0.11.0
distro            1.7.0
execnet           1.9.0
extras            1.0.0
filelock          3.8.0
fixtures          4.0.0
fonttools         4.37.2
gpg               1.17.1-unknown
iniconfig         1.1.1
kiwisolver        1.4.4
libcomps          0.1.19
louis             3.23.0
lxml              4.9.1
matplotlib        3.5.3
mypy              0.971
mypy-extensions   0.4.3
numpy             1.23.1
olefile           0.46
packaging         21.3
pbr               5.9.0
pep517            0.12.0
pexpect           4.8.0
Pillow            9.2.0
pip               22.2.2
pluggy            1.0.0
ptyprocess        0.7.0
py                1.11.0
PyGObject         3.42.2
pyparsing         3.0.9
pytest            7.1.3
pytest-forked     1.4.0
pytest-xdist      2.5.0
python-dateutil   2.8.2
rpm               4.17.0
scour             0.38.2
setuptools        65.3.0
setuptools-scm    7.0.5
six               1.16.0
testtools         2.5.0
tomli             2.0.1
typing_extensions 4.3.0
wheel             0.37.1

[dmtucker]

Thanks for the report!

I am able to reproduce this on master with mypy 0.971 (only). It appears to be caused by python/mypy@2132036, and it does NOT reproduce if runpytest_subprocess is replaced with runpytest_inprocess: https://github.com/dbader/pytest-mypy/blob/9710ab44edb57f6e7048c5909351f629b19c832a/tests/test_pytest_mypy.py#L307

[kloczek]

Thank you 👍