Repository navigation
Third-party stubs: recommending a default path for installing stub files, overriding stubs #84
Description
Activity
- added a commit that references this issue
on Apr 17, 2015 Ended up recommending shared/typehints/python3.5, etc. since:
- a different Python version is effectively a diferent environment
- stub files for the same libraries on a different Python version might be different
@JukkaL, I've seen https://github.com/JukkaL/mypy/tree/master/stubs has a similar concept with distinct stubs per Python version. Do you see any problem with the suggestion?
My plan is actually to get rid of the separate stub directories for Python 3.4 etc. in mypy. The reason is that it makes stubs a more difficult to maintain with marginal benefits. Having separate stubs for Python 2 and 3 would be useful, however, since they are often significantly different. My plan is to only have
stubs/python2andstubs/python3(or similar) in mypy.Also, should the directory be
shared/python/typehints/...instead (i.e., with/python/), or maybeshared/pytypehints/...?Potentially we could recommend something like
__minpyversion__ = '3.4'in the stubs to specify the minimum supported Python version.Also, we discussed the possibility of having a single stub file that works in all Python versions (2.x and 3.x). It would be nice if we didn't have to maintain two copies of such a stub file, as these could easily get out of sync.
You could do something like the following which allows you to add a more specific stub file for a specific Python version
import os import sys class FSUnion(): def __init__(self, fsroot): self._dirs = [] vi = [str(elem) for elem in sys.version_info[:2]] self._dirs.append(os.path.join(fsroot, vi[0], vi[1])) self._dirs.append(os.path.join(fsroot, vi[0])) self._dirs.append(fsroot) def __getitem__(self, module_name:str) -> str: for d in self._dirs: stub_filename = os.path.join(d, '{}.pyi'.format(module_name)) if os.path.exists(stub_filename): return stub_filename raise KeyError('{}: Stub file for {} not found'.format(self.__class__.__name__, module_name)) fsu = FSUnion('typehints') print(fsu['datetime']) print(fsu['sys']) print(fsu['os']) print(fsu['notthere'])Which, with the following file structure
typehints os.pyi sys.pyi datetime.pyi /3 os.pyi /5 sys.pyiprints
typehints\datetime.pyi typehints\3\5\sys.pyi typehints\3\os.pyi Traceback (most recent call last): File "fsunion.py", line 26, in <module> print(fsu['notthere']) File "fsunion.py", line 21, in __getitem__ raise KeyError('{}: Stub file for {} not found'.format(self.__class__.__name__, module_name)) KeyError: 'FSUnion: Stub file for notthere not found'@JukkaL, so if we want to use PyPI and
pip, we have to have pythonX.Y. The reason for that is as follows:- a user has python3.3, python3.4 and python3.5 installed on his machine
- for some reason he doesn't use virtual environments, he is free to ignore them
- he uses a hypothetical package called "packagify" that since version 2.0 switched to "yield from", and since version 2.1 switched to type annotations in source files
- if he does
pip-3.3 install packagify==1.0 packagify-types==1.0and thenpip-3.4 install packagify==2.0 packagify-types==2.0and thenpip-3.5 install packagify==2.1(stubs not needed in the last case), he expects types to be valid for each version.
Moreover, as we talked about this with @vlasovskikh, if a construct is described in the stubs, we trust that it is correct. For instance, if functools.pyi has
def singledispatch(), then the type checker assumes it exists via some runtime magic, even if it's not there in the source. This would be incorrect for Python 3.3 and a nice bug to catch, actually. So you'd need to have the the hypothetical "stdlib-types" specify "minversion" in functools.pyi. But at this point the stubs stop being usable for Python 3.3 and lower. So you end up introducing versioning in the package name. Back to square one and with more hairy workarounds.I can't tell from this discussion if this requires PEP changes or not. I am labeling this as enhancement which I will interpret as "in the future, maybe", i.e. "no need to change the PEP or typing.py now".
(Sorry, didn't mean to close.)
FYI, I have thought a lot about this, and I think the best path forward is to extend the
if typing.PY3sort of logic.Currently mypy's stubs DTWT for python 3.2/3.3, because the stubs for modules that were present in 3.2 include functions that were only added in 3.4. And there are even cases where functions were removed in later python versions.
Actually, Mark Shannon made me remove typing.PY3 and other platform checks. Instead, type checkers should learn how typical programs check for platforms. So extending typing.PY3 is not an option.
(I think this issue is actually about something else, so I won't close it.)
In that case, a single stub file can write
if sys.version_info[:2] >= (3, 4)to expose different APIs and we can mandate that a checker supports that (as opposed to just a major version).You're right that it's not the direct focus of this PR, but the choosing in-file vs out-of-file multiversioning will affect the answer for what the paths should be.
That said, I am not looking forward to implementing the "calculate what
sys.pathwould be for a different python version than what we're currently using" logic. But since supporting in-file stubs is the ideal case, we can't avoid that.I'm interested in moving this issue forward, because it appears to be a somewhat common problem for mypy users that there is no standard place to install third-party stubs outside of typeshed. You can manually install stubs into some directory and set $MYPYPATH, but that is fragile and not portable. By fixing this issue, we could also fix python/typeshed#153, because third-party modules could now come with version-specific stubs.
I think Łukasz's approach of installing stubs using
setup(data_files=...)is basically sound, but there's one complication: Type checkers may not have access to the Python binary that is used to run the code they're checking, so they don't know where setup() installed the stubs. I think mypy can get by with something like this for getting 2.7 stubs:- Try to run
python2.7 -c 'import sys; print(sys.exec_prefix)'and use the output joined with shared/type_hinting/python2.7 as an additional search path for stubs. - If that doesn't work (e.g., because the code being checked runs in a virtualenv), it is the user's responsibility to use
$MYPYPATHor the mypy_path= config option to teach mypy to find the stubs. - Perhaps mypy could also provide an option like
--check-using-python /path/to/python/binary. If this is given, it can query the binary for its exec_prefix and find the stubs that way.
Other type checkers may provide additional implementation-specific ways to find the shared stub directory.
If we go with this approach, how should it be codified? I could write a new PEP, but perhaps this is small enough that it can just go as a new section into PEP 484.
I have a proof-of-concept implementation in https://github.com/JelleZijlstra/mypy/tree/stubdir and a library using the functionality at https://github.com/JelleZijlstra/sqlalchemy-stubs.
- Try to run
27 remaining items
But what if the author django-stub discovers that they made a mistake in their stubs for Django 1.11? Perhaps they could release django-stub 1.11.1, but then that would mean they'd have to maintain separate, mostly identical copies of the stubs for each Django version.
I think we'll have to support something like this straw man: stubs can do
if __version__ >= (1, 11), where__version__is a magical constant that the type checker evaluates to the version of the package that is being used. There's many details there that I haven't thought much about: what about namespace packages? how does the type checker know what library version you are using?Yeah, I think this proposal supports library versioning well enough.
[Clarification: This was written before Jelle's comment above, in response to @asvetlov's "I don't think versioning for stub libraries is an issue".]
But what if the author django-stub discovers that they made a mistake in their stubs for Django 1.11? Perhaps they could release django-stub 1.11.1, but then that would mean they'd have to maintain separate, mostly identical copies of the stubs for each Django version.
I think we'll have to support something like this straw man: stubs can do
if __version__ >= (1, 11), where__version__is a magical constant that the type checker evaluates to the version of the package that is being used. There's many details there that I haven't thought much about: what about namespace packages? how does the type checker know what library version you are using?This feels like worrying too much. We've never even tried any of this proposal, and we don't know if this scenario will occur frequently enough to worry about. But if we specify something to handle this now we'll never be able to remove that feature, no matter how unpopular (because it'll always break someone's workflow).
Also, I really don't like having to support library version comparisons in the checker -- unlike Python versions we don't have control over the ordering of libraries (they don't all use semver). The maintenance problem can be solved using branches. We're better off allowing some freedom in the naming of stub packages -- maybe the package name will end up including the django version (e.g. django-1.1-stubs) so the package version can be whatever the stub package author wants.
- added a commit that references this issue
on Aug 30, 2017 I've started writing up a PEP. Hopefully I can get a draft out tomorrow or the next day.
Things to think about which I leave open for discussion (because they should be discussed more):
- How should packages indicate they support typing? (My personal favorite is a new trove classifier, but other options exist)
- How should mixed stub/inline packages be dealt with? If we have stubs in a package, should we parse the Python files to check if there is type information in the files? That could be slow for large codebases. Should we have 3 states of metadata? Stubs, inline, mixed? That has its own drawbacks.
- What to do with the existing PEP 484 text on the matter? Some say we should delete it and replace it with the new PEP. Im not sure that is the best thing to do, but it sounds cleaner.
- Other issues with current designs.
-
I'm not sure I like Trove classifiers that much. They're almost free-form but here we need something machine parseable. Maybe there's something in PEP 459? Or maybe we can just add something else to the egg-info (but I don't know how any of that stuff works, really).
-
For packages that declare they support typing, we should use the standard search approach, which is .pyi first then .py, ignoring the .py if the .pyi is found.
Reacted by Emma Smith-
Okay, I have a rough draft here: https://github.com/ethanhs/peps/blob/typedist/pep-0561.rst
I also added a PR python/peps#415 if people prefer the Github review UI, but also feel free to leave comments in this issue.
I plan on making a POC implementation of the distutils extension when I have time either later today or tomorrow.
A couple points Im not sure about:
should the keyword be a boolean about whether the package is stubbed or not? Then
stubbed == Trueinline == Falseandtype unsafe == None.Mixed inline/stubbed packages - I originally wrote a version with an additional option for this, but it seemed to make things more complicated than needed.
Thanks!
- added a commit that references this issue
on Sep 25, 2017 The PEP has been posted to Python-dev and the latest live version can be found here: https://www.python.org/dev/peps/pep-0561/
Reacted by Serhii Khalymon- added a commit that references this issue
on Nov 3, 2017 I believe PEP 561 resolves this issue, now that it is accepted.
Reacted by Ivan Levkivskyi and Corey ColeYes!
I believe PEP 561 resolves this issue, now that it is accepted.
It was a long way. Thanks @ethanhs for all the work on this!
The ideal situation is where a file is annotated. The other obvious choice if there is a module.pyi stub file alongside module.py. However, since package maintainers are free not to add type hinting to their packages, there has to be support for third-party stubs installable by
pipfrom PyPI. This opens the following questions:data_files=('share/typehinting/', pathlib.Path(SRC_PATH).glob('**/*.pyi'))in setup.py. We are proposing to add a setup.cfg hook to do the right thing automatically."shared/typehinting/) where third-party packages can put their implementation of stubs for external packages. Also, tools like type checkers or IDEs might provide their own custom paths for stubs they themselves populate, etc. The type checker itself is expected to only load one *.pyi file per corresponding *.py module.