Third-party stubs: recommending a default path for installing stub files, overriding stubs

[ambv]

The ideal situation is where a file is annotated. The other obvious choice if there is a module.pyi stub file alongside module.py. However, since package maintainers are free not to add type hinting to their packages, there has to be support for third-party stubs installable by pip from PyPI. This opens the following questions:

• should there be a recommended naming scheme for those stubs? "No, a package maintainer can always trump third-party implementations by including types in his own module.".
• should there be a default install place for those stubs? "Yes, this makes the type checker's stub discovery much easier. We are proposing using data_files=('share/typehinting/', pathlib.Path(SRC_PATH).glob('**/*.pyi')) in setup.py. We are proposing to add a setup.cfg hook to do the right thing automatically."
• should third-party stubs be versioned? "Yes, they should be versioned as the lowest version of the source package that they are describing. Example: FooPackage has versions 1.1, 1.2, 1.3, 2.0, 2.1, 2.2, 2.3, 2.4. There are API changes in versions 1.1, 2.0 and 2.4. We need only those three versions of stubs, in which case the user should install the closest lower version of stubs available. If the user lives dangerously and does not pin versions, running both the source package and the stubs as latest will generally work (as long as the stubs are regularly updated)."
• as mentioned previously, the package itself might provide *.pyi files. There will be a directory (shared/typehinting/) where third-party packages can put their implementation of stubs for external packages. Also, tools like type checkers or IDEs might provide their own custom paths for stubs they themselves populate, etc. The type checker itself is expected to only load one *.pyi file per corresponding *.py module.

[ambv]

Ended up recommending shared/typehints/python3.5, etc. since:

• a different Python version is effectively a diferent environment
• stub files for the same libraries on a different Python version might be different

@JukkaL, I've seen https://github.com/JukkaL/mypy/tree/master/stubs has a similar concept with distinct stubs per Python version. Do you see any problem with the suggestion?

[JukkaL]

My plan is actually to get rid of the separate stub directories for Python 3.4 etc. in mypy. The reason is that it makes stubs a more difficult to maintain with marginal benefits. Having separate stubs for Python 2 and 3 would be useful, however, since they are often significantly different. My plan is to only have stubs/python2 and stubs/python3 (or similar) in mypy.

Also, should the directory be shared/python/typehints/... instead (i.e., with /python/), or maybe shared/pytypehints/...?

Potentially we could recommend something like __minpyversion__ = '3.4' in the stubs to specify the minimum supported Python version.

Also, we discussed the possibility of having a single stub file that works in all Python versions (2.x and 3.x). It would be nice if we didn't have to maintain two copies of such a stub file, as these could easily get out of sync.

[sffjunkie]

You could do something like the following which allows you to add a more specific stub file for a specific Python version

import os
import sys

class FSUnion():
    def __init__(self, fsroot):
        self._dirs = []
        vi = [str(elem) for elem in sys.version_info[:2]]
        self._dirs.append(os.path.join(fsroot, vi[0], vi[1]))
        self._dirs.append(os.path.join(fsroot, vi[0]))
        self._dirs.append(fsroot)

    def __getitem__(self, module_name:str) -> str:
        for d in self._dirs:
            stub_filename = os.path.join(d, '{}.pyi'.format(module_name))
            if os.path.exists(stub_filename):
                return stub_filename

        raise KeyError('{}: Stub file for {} not found'.format(self.__class__.__name__, module_name))

fsu = FSUnion('typehints')
print(fsu['datetime'])
print(fsu['sys'])
print(fsu['os'])
print(fsu['notthere'])

Which, with the following file structure

typehints
    os.pyi
    sys.pyi
    datetime.pyi
    /3
        os.pyi
        /5
            sys.pyi

prints

typehints\datetime.pyi
typehints\3\5\sys.pyi
typehints\3\os.pyi
Traceback (most recent call last):
  File "fsunion.py", line 26, in <module>
    print(fsu['notthere'])
  File "fsunion.py", line 21, in __getitem__
    raise KeyError('{}: Stub file for {} not found'.format(self.__class__.__name__, module_name))
KeyError: 'FSUnion: Stub file for notthere not found'

[ambv]

@JukkaL, so if we want to use PyPI and pip, we have to have pythonX.Y. The reason for that is as follows:

• a user has python3.3, python3.4 and python3.5 installed on his machine
• for some reason he doesn't use virtual environments, he is free to ignore them
• he uses a hypothetical package called "packagify" that since version 2.0 switched to "yield from", and since version 2.1 switched to type annotations in source files
• if he does pip-3.3 install packagify==1.0 packagify-types==1.0 and then pip-3.4 install packagify==2.0 packagify-types==2.0 and then pip-3.5 install packagify==2.1 (stubs not needed in the last case), he expects types to be valid for each version.

Moreover, as we talked about this with @vlasovskikh, if a construct is described in the stubs, we trust that it is correct. For instance, if functools.pyi has def singledispatch(), then the type checker assumes it exists via some runtime magic, even if it's not there in the source. This would be incorrect for Python 3.3 and a nice bug to catch, actually. So you'd need to have the the hypothetical "stdlib-types" specify "minversion" in functools.pyi. But at this point the stubs stop being usable for Python 3.3 and lower. So you end up introducing versioning in the package name. Back to square one and with more hairy workarounds.

[gvanrossum]

I can't tell from this discussion if this requires PEP changes or not. I am labeling this as enhancement which I will interpret as "in the future, maybe", i.e. "no need to change the PEP or typing.py now".

[gvanrossum]

(Sorry, didn't mean to close.)

[o11c]

FYI, I have thought a lot about this, and I think the best path forward is to extend the if typing.PY3 sort of logic.

Currently mypy's stubs DTWT for python 3.2/3.3, because the stubs for modules that were present in 3.2 include functions that were only added in 3.4. And there are even cases where functions were removed in later python versions.

[gvanrossum]

Actually, Mark Shannon made me remove typing.PY3 and other platform checks. Instead, type checkers should learn how typical programs check for platforms. So extending typing.PY3 is not an option.

(I think this issue is actually about something else, so I won't close it.)

[o11c]

@gvanrossum

In that case, a single stub file can write if sys.version_info[:2] >= (3, 4) to expose different APIs and we can mandate that a checker supports that (as opposed to just a major version).

You're right that it's not the direct focus of this PR, but the choosing in-file vs out-of-file multiversioning will affect the answer for what the paths should be.

That said, I am not looking forward to implementing the "calculate what sys.path would be for a different python version than what we're currently using" logic. But since supporting in-file stubs is the ideal case, we can't avoid that.