Repository navigation
Use incremental mode in python evaluation test cases #1412
Description
Activity
I looked into this for a few minutes and one problem is that currently each test is run in a clean directory. There currently is no way to change the name of the cache directory, and it's always
.mypy_cacherelative to the current directory. So we'd need something for this. (I've filed #1414 to track it.)I just tried making the pythoneval tests use a shared cache directory via this diff:
diff --git a/mypy/test/testpythoneval.py b/mypy/test/testpythoneval.py index cf4cbafd..9c207d5d 100644 --- a/mypy/test/testpythoneval.py +++ b/mypy/test/testpythoneval.py @@ -14,7 +14,7 @@ import os import os.path import re import sys - +from tempfile import TemporaryDirectory import pytest # type: ignore # no pytest in typeshed from typing import List @@ -37,12 +37,12 @@ class PythonEvaluationSuite(DataSuite): 'pythoneval-asyncio.test'] base_path = test_temp_dir optional_out = True - + cache_dir = TemporaryDirectory() def run_case(self, testcase: DataDrivenTestCase) -> None: - test_python_evaluation(testcase) + test_python_evaluation(testcase, os.path.join(self.cache_dir.name, '.mypy_cache')) -def test_python_evaluation(testcase: DataDrivenTestCase) -> None: +def test_python_evaluation(testcase: DataDrivenTestCase, cache_dir: str) -> None: """Runs Mypy in a subprocess. If this passes without errors, executes the script again with a given Python @@ -71,6 +71,7 @@ def test_python_evaluation(testcase: DataDrivenTestCase) -> None: with open(program_path, 'w') as file: for s in testcase.input: file.write('{}\n'.format(s)) + mypy_cmdline.append('--cache-dir={}'.format(cache_dir)) output = [] # Type check the program. out, err, returncode = api.run(mypy_cmdline)However I get a failure (consistently) with:
_______________ testCanConvertTypedDictToAnySuperclassOfMapping _______________ [gw8] win32 -- Python 3.6.5 C:\Python36\python.exe data: C:\Users\ethanhs\Documents\mypy\test-data\unit\pythoneval.test:1081: ..\..\..\..\Documents\mypy\mypy\test\testpythoneval.py:42: in run_case test_python_evaluation(testcase, os.path.join(self.cache_dir.name, '.mypy_cache')) ..\..\..\..\Documents\mypy\mypy\test\testpythoneval.py:92: in test_python_evaluation testcase.file, testcase.line)) ..\..\..\..\Documents\mypy\mypy\test\helpers.py:98: in assert_string_arrays_equal raise AssertionError(msg) E AssertionError: Invalid output (C:\Users\ethanhs\Documents\mypy\test-data\unit\pythoneval.test, line 1081) ---------------------------- Captured stderr call ----------------------------- Expected: _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: error: Incompatible... _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note: Following mem... _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note: Expected: _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note: def __iter__(self) -> Iterator[int] (diff) _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note: Got: _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note: def __iter__(self) -> Iterator[str] (diff) Actual: _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: error: Incompatible... _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note: Following mem... _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note: Expected: _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note: def () -> Iterator[int] (diff) _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note: Got: _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note: def () -> Iterator[str] (diff) Alignment of first line difference: E: ...y:11: note: def __iter__(self) -> Iterator[int] A: ...y:11: note: def () -> Iterator[int] ^this appears to be an incremental bug?
Yes, it looks like name of a method is lost during (de-)serialisation. If you have time and desire, you could fix this yourself, otherwise create a separate issue about this.
@ilevkivskyi that makes sense. I'm afraid I'm not familiar enough with the intricacies of incremental to debug this one. I'll bump the priority of that issue up however.
- added a commit that references this issue
on May 26, 2018 - added a commit that references this issue
on May 26, 2018
The Python evaluation test cases (which don't actually run the code if there are type errors...) use full stubs for
builtinsandtyping. Using incremental mode would likely speed them up significantly, as processing stubs probably takes the majority of time. These tests are the long pole in the full test suite, so this could give a nice speed boost.