Single threaded pytest spuriously fails for testcmdline and testpythoneval

[elazarg]

Reported by @gvanrossum in #3870 :

Sadly there's a problem with this. When I run the tests as follows:

pytest -n0 mypy/test/testcmdline.py

every test in that file fails. Apparently the pytest-xdist extension which runs tests in parallel does something that keeps the tests from failing. The error is essentially that every subprocess invoking scripts/mypy fails with the following traceback:

 Traceback (most recent call last):
   File "/Users/guido/src/mypy/scripts/mypy", line 4, in <module>
     from mypy.main import main
 ModuleNotFoundError: No module named 'mypy.main'

This makes sense as the current directory is set to the temporary directory where the test files are written, so the repo root is not on sys.path. I'm guessing that the xdist extension somehow retains the current directory?

The pythoneval tests fail in the same way.

(End quote)

On my machine most of the tests pass, except testCoberturaParser:

Expected:
  <coverage timestamp="$TIMESTAMP" version="$VERSION" line-rate="0.8000" branch-rate="0"> (diff)
  ...
Actual:
  <coverage timestamp="$TIMESTAMP" version="0.530" line-rate="0.8000" branch-rate="0"> (diff)
  ...

Apparently pytest-xdist adds mypy's base dir to PYTHONPATH.

This issue has two items:

• Fix the failures
• Add tests to prevent the reintroduction of such failures

[elazarg]

The following patch fixed the problem for me, but I can't reproduce the more serious issue, so it might not be the right solution (this diff is only about testcmdline; pythoneval should be similar):

diff --git a/mypy/test/testcmdline.py b/mypy/test/testcmdline.py
index 26f1780..ede8f30 100644
--- a/mypy/test/testcmdline.py
+++ b/mypy/test/testcmdline.py
@@ -12,7 +12,7 @@ import sys
 from typing import Tuple, List, Dict, Set

 from mypy.myunit import Suite, SkipTestCaseException, AssertionFailure
-from mypy.test.config import test_data_prefix, test_temp_dir
+from mypy.test.config import test_data_prefix, test_temp_dir, PREFIX
 from mypy.test.data import fix_cobertura_filename
 from mypy.test.data import parse_test_cases, DataDrivenTestCase, DataSuite
 from mypy.test.helpers import assert_string_arrays_equal, normalize_error_messages
@@ -58,10 +58,13 @@ def test_python_evaluation(testcase: DataDrivenTestCase) -> None:
     # Type check the program.
     fixed = [python3_path,
              os.path.join(testcase.old_cwd, 'scripts', 'mypy')]
+    env = os.environ.copy()
+    env['PYTHONPATH'] = PREFIX
     process = subprocess.Popen(fixed + args,
                                stdout=subprocess.PIPE,
                                stderr=subprocess.STDOUT,
-                               cwd=test_temp_dir)
+                               cwd=test_temp_dir,
+                               env=env)
     outb = process.stdout.read()
     # Split output into lines.
     out = [s.rstrip('\n\r') for s in str(outb, 'utf8').splitlines()]

[gvanrossum]

Thanks! I'll try it when I have more time. It may miraculously work.

[gvanrossum]

Yeah, this fixes it for me. I tried pythoneval.py as well and it passes with -n0 -- so it doesn't need this fix. This is because it doesn't fork a subprocess to run mypy -- it uses mypy.api.run(). Perhaps testcmdline.py could be modified to do that too? I don't think we gain much by forking a process. (Though we don't seem to lose much either.)

[emmatyping]

When I moved pythoneval to mypy.api.run() I also considered moving testcmdline.py to it as well but did not, as I worried about giving up testing the actual OS<->mypy interaction. On the other hand, I feel that it hasn't really ever been an issue, and there is not much that can go wrong without Python itself being broken.

[gvanrossum]

Maybe we can include 1-2 separate tests of the actual OS<->mypy interaction; one for a passing program and one for a failing program. (Maybe a few more to exercise the exceptional workflows that raise CompileError prematurely, e.g. missing file or syntax errors.)