provide tools for testing projects that use mypy (e.g. mypy.tests.data pytest plugin)

[chadrik]

I was very glad to see that mypy.tests was added into the mypy package a few releases back, because that opened up the possibility of using mypy.tests.data as a pytest plugin for testing other projects. I used it in the static typing tests for attrs: https://github.com/chadrik/attrs/blob/pyi_stubs/tests/test_stubs.py

Unfortunately commit e03780623 added an import of mypy.test.config which causes an error because the module expect to find a test-data/unit directory at the same level as the root mypy directory:

tests/test_stubs.py:8: in <module>
    from mypy.test.data import parse_test_cases, DataSuite
.tox/stubs/lib/python3.6/site-packages/mypy/test/data.py:14: in <module>
    from mypy.test.config import test_temp_dir
.tox/stubs/lib/python3.6/site-packages/mypy/test/config.py:14: in <module>
    'Test data prefix ({}) not set correctly'.format(test_data_prefix)

I'm pip installing mypy, so naturally it's not there.

I recognize that I've gone a bit out on a limb using mypy.tests.data since it's not a public API, but it's really really useful. I'd like to request that this become a public plugin, and I think a good first step towards this is to make the constants stored in mypy.tests.config configurable via pytest. If you like the idea, I'm glad to submit a PR.

[emmatyping]

There are plans to do some not insignificant refactoring to move mypy.test entirely to pytest (see #1673), but after that, I don't see why we couldn't expose the data api publicly. We might even consider splitting it out of the main mypy package (probably down the road) as the number of typed packages increases.

[gvanrossum]

I have to repeat what I've said before -- the mypy package is *not* a public API and we don't make any attempts at preserving compatibility across releases. Please don't build applications or libraries depending on mypy internals. And that includes the mypy.test subpackage. Please.

[chadrik]

I have to repeat what I've said before -- the mypy package is not a
public API

I get that. This is a feature request to make a pytest plugin that is public.

I think that making it easy to test statically type-checked projects is essential to fostering the mypy community. mypy.tests.data has no mypy dependencies other than mypy.myunit (which IIUC is migrating into the pytest plugin anyway) and now mypy.tests.config (which is just two constants). The plugin comprises code that every project that uses mypy needs, and should not be forced to copy-and-paste or reinvent.

[chadrik]

Updated the title to clarify the request

[gvanrossum]

I'm not sure I understand what you want to do with that plugin. mypy.test.data is mostly a parser for the kind of test files that are used in mypy's unit tests. How exactly do you think other statically type-checked benefit from this? More in particular, how does *your* project benefit from this? Are you writing such test files? You're welcome to copy the code and release it as a separate package, and maybe mypy can depend on your version if you maintain it well.

[chadrik]

mypy.test.data is mostly a parser for the kind of test files that are used
in mypy's unit tests. How exactly do you think other statically
type-checked benefit from this? More in particular, how does your project
benefit from this? Are you writing such test files?

Yes. All the problems that you faced when designing a way to test mypy, other developers will soon face when testing their stubs. I saw no reason to reinvent the wheel.

You're welcome to copy
the code and release it as a separate package, and maybe mypy can depend on
your version if you maintain it well.

Once #1673 is completed, I may take you up on that, though I'll admit I'm surprised that you don't think it would be easier to maintain within mypy.

[gvanrossum]

It still seems to me that testing mypy is very different from testing stubs, but I'm happy to agree to disagree.

…

On Dec 14, 2017 13:09, "Chad Dombrova" ***@***.***> wrote: mypy.test.data is mostly a parser for the kind of test files that are used in mypy's unit tests. How exactly do you think other statically type-checked benefit from this? More in particular, how does *your* project benefit from this? Are you writing such test files? Yes. All the problems that you faced when designing a way to test mypy, other developers will soon face when testing their stubs. I saw no reason to reinvent the wheel. You're welcome to copy the code and release it as a separate package, and maybe mypy can depend on your version if you maintain it well. Once #1673 <#1673> is completed, I may take you up on that, though I'll admit I'm surprised that you don't think it would be easier to maintain within mypy. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4366 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACwrMtBZKjSxA09Bqy3CmW-l6l-lvfEQks5tAY6cgaJpZM4RCj-r> .

[chadrik]

It still seems to me that testing mypy is very different from testing
stubs, but I'm happy to agree to disagree.

Not to belabor the point, but that's why mypy has several testing systems covering different strategies (e.g. some testing the code itself, and some testing the cli). mypy.tests.data answers the very general question: "does the mypy cli produce the expected errors when run against the the given modules?" That's quite useful.

[elazarg]

Just added #4369 which removes myunit, but will not be the last PR in the migration.

@chadrik I'm not sure I understand - do you want to assert something about mypy's error output? Because the I don't think the error messages themselves are (or should be) part of mypy's public API. Or maybe you are talking about something as abstract as # E: TypeError?

Note that your test code is already broken (take a look at the current PythonEvaluationSuite) and I believe it is likely to break again often in the near future.

[chadrik]

do you want to assert something about mypy's error output?

Yes. Here's our test suite: https://github.com/chadrik/attrs/blob/pyi_stubs/tests/test_stubs.test

Our goal is not to statically type check our library; the library itself contains no type comments. If that were the case we would simply assert that mypy returns 0.

Our goal is to test that the stubs, which we now consider part of our public API, behave as expected. Like any unit test, that means setting up specific scenarios and asserting that the the results are as expected.

e.g.

[case test_type_arg]
# cmd: mypy a.py
[file a.py]
import attr

@attr.s
class C(object):
    a = attr.ib(type=int)

c = C()
reveal_type(c.a)  # int
reveal_type(C.a)  # int
[out]
a.py:8: error: Revealed type is 'builtins.int*'
a.py:9: error: Revealed type is 'builtins.int*'

Most of the "errors" we care about are generated by reveal_type, but, again like any unit test, there are situations where we want to assert that an error does occur, such as with these validators:

[case test_validators]
# cmd: mypy a.py
[file a.py]
import attr
from attr.validators import in_, and_, instance_of

a = attr.ib(type=int, validator=in_([1, 2, 3]))
b = attr.ib(type=int, validator=[in_([1, 2, 3]), instance_of(int)])
c = attr.ib(type=int, validator=(in_([1, 2, 3]), instance_of(int)))
d = attr.ib(type=int, validator=and_(in_([1, 2, 3]), instance_of(int)))
e = attr.ib(type=int, validator=1)  # error

[out]
a.py:8: error: Argument 2 has incompatible type "int"; expected "Union[Callable[[Any, Attribute[Any], int], Any], List[Callable[[Any, Attribute[Any], int], Any]], Tuple[Callable[[Any, Attribute[Any], int], Any], ...]]"

Another feature that I would love is the ability to statically type-check doc snippets, like doctests for mypy. I whipped up a sphinx extension to do this, but it would also rely on string matching error output.

Because I don't think the error messages themselves are (or should be) part of mypy's public API.

This is a good point, and it's an annoyance that I was willing accept given the dearth of options (and my desire to focus on writing tests instead of writing a test runner). This problem could be mitigated (though not entirely avoided) by improving the mypy pytest plugin to support glob or regex matching.

Maybe a better solution to this problem is this other issue that I opened to support type checking assert statements that pertain to types: #4306.

Using assert instead of something like mypy.tests.data would solve several problems:

• restricts the kinds of tests possible to true/false type comparisons, removing reliance on formatting of error messages
• allows existing pytest unit tests to be statically type checked (the original motivation)

[emmatyping]

I think I agree with Guido that it doesn't make sense to have a public API. Some common test framework for stubs will have to evolve elsewhere.