StubtestMiscUnit.test_output fails during Debian package building

[mr-c]

mypy 0.770

Installing dmypy script to /build/mypy-0.770/debian/python3-mypy/usr/bin
Installing mypy script to /build/mypy-0.770/debian/python3-mypy/usr/bin
Installing stubgen script to /build/mypy-0.770/debian/python3-mypy/usr/bin
Installing stubtest script to /build/mypy-0.770/debian/python3-mypy/usr/bin
set -e; for v in 3.7 3.8; do \
	PATH=$PATH:/build/mypy-0.770/debian/mypy/usr/bin/ python$v -m pytest -n auto \
	-o testpaths=mypy/test -o python_files=test*.py \
	-o python_classes= -o python_functions=; \
done
============================= test session starts ==============================
platform linux -- Python 3.7.7, pytest-4.6.9, py-1.8.1, pluggy-0.13.0
rootdir: /build/mypy-0.770, testpaths: mypy/test
plugins: xdist-1.31.0, forked-1.1.3, cov-2.8.1
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I
gw0 [282] / gw1 [282] / gw2 [282] / gw3 [282] / gw4 [282] / gw5 [282] / gw6 [282] / gw7 [282]

.........................................s.............................. [ 25%]
............................................................s........... [ 51%]
..........................................................s....sss...... [ 76%]
.....................s.................F..........................       [100%]
=================================== FAILURES ===================================
_________________________ StubtestMiscUnit.test_output _________________________
[gw4] linux -- Python 3.7.7 /usr/bin/python3.7

self = <mypy.test.teststubtest.StubtestMiscUnit testMethod=test_output>

    def test_output(self) -> None:
        output = run_stubtest(
            stub="def bad(number: int, text: str) -> None: ...",
            runtime="def bad(num, text): pass",
            options=[],
        )
        expected = (
            'error: {0}.bad is inconsistent, stub argument "number" differs from runtime '
            'argument "num"\nStub: at line 1\ndef (number: builtins.int, text: builtins.str)\n'
            "Runtime: at line 1 in file {0}.py\ndef (num, text)\n\n".format(TEST_MODULE_NAME)
        )
>       assert remove_color_code(output) == expected
E       AssertionError: assert 'error: test_...um, text)\n\n' == 'error: test_m...um, text)\n\n'
E         Skipping 179 identical leading characters in diff, use -v to show
E         - 1 in file /tmp/tmppyozpeza/test_module.py
E         + 1 in file test_module.py
E           def (num, text)

mypy/test/teststubtest.py:587: AssertionError
=============== 1 failed, 274 passed, 7 skipped in 80.08 seconds ===============

[ethanhs]

Hm, @hauntsaninja I believe you wrote this code. It seems that the temp directory code is not doing what you expect.

[hauntsaninja]

I did write this code. The test passes for me locally as well as in CI:

~/dev/mypy master λ python3.7 -m pytest -k test_output 
====================================================== test session starts ======================================================
platform darwin -- Python 3.7.6, pytest-5.3.2, py-1.8.1, pluggy-0.13.1
rootdir: /Users/shantanu/dev/mypy, inifile: pytest.ini, testpaths: mypy/test, mypyc/test
plugins: forked-1.0.2, xdist-1.31.0, cov-2.8.1
gw0 [1] / gw1 [1] / gw2 [1] / gw3 [1]
.                                                                                                                         [100%]
====================================================== 1 passed in 21.85s =======================================================

StubtestMiscUnit.test_output is a basic test of the output of the stubtest script. It appears the issue is that it's printing the absolute path of the temporary python file in the error message, whereas StubtestMiscUnit.test_output expects it to print the file location relative to cwd (which we set in use_tmp_dir).

One easy way to fix this test failure, since it's pretty inconsequential, would just be to re.sub("/\S*" + TEST_MODULE_NAME, TEST_MODULE_NAME, output). Is there an easy way for me to repro?

[mr-c]

@hauntsaninja Here's a reproducer Dockerfile

FROM debian:testing-slim
RUN apt-get update && apt-get install -y mypy python3-pytest
WORKDIR /
RUN mkdir -p test-data/unit test-data/packages
ENV MYPY_TEST_PREFIX=/
RUN python3 -m pytest -k test_output -o testpaths=mypy/test -o python_files=test*.py --pyargs mypy

[mr-c]

and another reproducer using the latest mypy from PyPI:

FROM debian:testing-slim
RUN apt-get update && apt-get install -y python3-pip python3-pytest python3-lxml
RUN pip install mypy
WORKDIR /
RUN mkdir -p test-data/unit test-data/packages
ENV MYPY_TEST_PREFIX=/
RUN python3 -m pytest -k test_output -o testpaths=mypy/test -o python_files=test*.py --pyargs mypy

[hauntsaninja]

This was fixed a while back, I think by a562f0a