Description
I was somewhat concerned that test builtins fixtures might not perfectly represent the production version of mypy, so I ran the tests without the fixtures.
The good news is that, as expected, fixtures make tests at least twice faster (pytest alone takes 212 sec without the fixtures, and 12 sec with).
The bad news is that 60 out of 2309 tests failed. Many of the failures were trivial (a slight change in the error message since the test was written, or not defining a type variables because it's already defined in the fixture). But a few were more serious.
For example this code (from check-class-namedtuple.test: testNewNamedTupleJoinTuple), instead of reporting the two revealed types, crashed mypy (under python 3.6):
class X(NamedTuple):
x: int
y: str
reveal_type([(3, 'b'), X(1, 'a')]) # E: Revealed type is 'builtins.list[Tuple[builtins.int, builtins.str]]'
reveal_type([X(1, 'a'), (3, 'b')]) # E: Revealed type is 'builtins.list[Tuple[builtins.int, builtins.str]]'
This code (from check-namedtuple.test: testNamedTupleMake) didn't report the second expected error message:
X = NamedTuple('X', [('x', int), ('y', str)])
reveal_type(X._make([5, 'a'])) # E: Revealed type is 'Tuple[builtins.int, builtins.str, fallback=__main__.X]'
X._make('a b') # E: Argument 1 to X._make has incompatible type "str"; expected Iterable[Any]
Not sure what the best solution is. Maybe it's worth having a daily run of all tests without fixtures using some special commit.
In any case, I made a branch that allows to easily run tests without fixtures because I couldn't find a command line parameter that would do it. I'll push a PR for it, although it's probably more appropriate for testing and discussion, not for merging.