Repository navigation
Search procedure for inline annotations #1190
Description
Activity
I agree that it would be nice if mypy could do the right thing automatically when a library module has inline annotations. However, this proposal has a few problems:
- If a library is partially annotated, mypy could pick some files from the implementation and some from stubs. As these could refer to different versions of a library, this could be confusing or generate bogus errors.
- Some modules may only contain variable definitions without any functions, and for these it's not easy to see whether a module is "annotated", as all annotations are optional.
- Parsing a file to decide whether is has annotations would be slow, especially with the current parser implementation in mypy. We could cache the results but this is still not optimal.
- The target Python version/configuration may be different from the one running mypy. Thus determining
sys.pathis difficult in general. One potential way to do this is to give mypy a path to a Python interpreter and have mypy run/target/path/python -c 'import sys; print(sys.path)'or similar. - Even if a file has type annotations, there is no guarantee that mypy can process the file without errors. Maybe the annotations are only for PyCharm or human readers, and mypy would generate type errors.
Here are some further ideas:
- Maybe a library with inline annotations should ship with
.pyifiles automatically generated from the inline annotations. If we have a convention for where to look for them, they could always be considered as more authoritative than stubs from typeshed. - A library with inline annotations could ship with some marker file or other well-known flag that signals that mypy should look at the implementation. Also, this signals that the library author has verified that mypy can cleanly process the module implementation. This has the issue of potential mypy version conflicts. If this doesn't work for some reason, users should be able to not use automatic
sys.pathtraversal and manually decide where to look for annotations. - We could ask authors of libraries with inline annotations to generate stubs using the aforementioned (hypothetical) tool and contribute them to typeshed. This would have known challenges such as dealing with multiple library versions, and deciding what to do if typeshed is out of date for some reason (e.g. everyone who can process PRs is busy).
@bdarnell What do you think?
I'd be happy to include some sort of package-level marker to indicate that the package contains PEP 484 type annotations and these should be used instead of typeshed (and that these decisions should be made at package scope instead of file-by-file). I'd rather not make this mypy-specific since mypy is not supposed to be the only consumer of PEP 484 annotations. If mypy can't handle the annotations in a package then it should either fail or fall back to typeshed for that package.
It would be unfortunate if the only use of inline annotations was in a tool that extracted them to
.pyistub files. If that were the case I would probably write stub files by hand instead of wrestling with inline annotations (which require runtime dependencies ontypingand other intrusions on the production code).Requiring all stubs to be submitted to typeshed seems like the worst possible outcome - I was under the impression that typeshed was essentially a transitional thing, and the long-term goal was for type information to migrate from typeshed into the respective packages. I think centralizing everything in typeshed would discourage participation by package authors.
I haven't forgotten this (I've just been distracted). We just encountered an issue here where there's a discrepancy between the way PyCharm and mypy search for stubs. In this case it was about an extension module, for which the user had created a stub file (but not added it to typeshed). Mypy found it fine on the default module search path. But PyCharm would only find it if there was a corresponding .py file in the same directory. In their case the fix was to add a .py file that simple re-exported the extension module. But it would be nice to have some kind of standardization for this.
PEP 484 is not clear about several things about the search path for stubs. I've mentioned some of them in a comment to python/typing#184.
@bdarnell Shouldn't *.pyi files from typeshed override the *.py files from installed packages? (according to the PEP). If there is no installed top-level package for the module in typeshed, then I believe the type checker could skip the typeshed stub for it.
@vlasovskikh Why would typeshed annotations take precedence over annotations in the code itself? The files in typeshed may describe a different version of the package and the annotations closer to the code are more likely to be correct for that version. If typeshed takes precedence then it is impossible to transition from annotations in typeshed to annotations in the code itself, and packages are being added to typeshed without coordinating with the package authors.
@bdarnell Given the following variants (please correct me if I'm missing something):
- Typeshed annotations first, then the annotations inside the code (until the user changes the priority of annotations lookup)
- Only the annotations inside the code (until the user changes the priority of annotations lookup)
- The annotations inside the code first if there are any (may be time-consuming to detect), then typeshed annotations (until the user changes the priority of annotations lookup)
I would prefer the variant (1) here. Speaking of (3) it may take a while to determine if any files inside a package contain type hints, so it will slow the code analysis down making it hard to run it on-the-fly. Also (3) implies that any annotation added to a module (say, in the form of a
# type: ...comment) forces a type checker to switch from typeshed to the real code, this might confuse the user since all the sudden the code analysis for a slightly updated version of the library starts to miss many errors inside their code.You're assuming that typeshed annotations will be higher-quality than in-code annotations; I am assuming the opposite. More to the point, if I as the author of a package add type annotations to that package, that's how I want my users' code to be type-checked. I don't want people coming to my support forums with type-checking issues that are caused by version skew between typeshed and the actual code they have installed.
Typeshed currently contains near-useless stubs for my project which were added without my knowledge. The presence of these stubs stands in the way of me adding my own annotations to the code. This is frustrating and discourages me from exploring type annotations further.
Your option 3 is the only one that makes sense to me as a library author. I'm fine with adding some piece of metadata somewhere to say "this package uses PEP 484 annotations which should be used by type checkers" so that you don't have to actually load the code to find the annotations (and to address the possibility of incompletely-annotated projects that aren't ready to be checked without typeshed support).
Reacted by Mikhail Korobov and Dan Cardin@bdarnell I see your point. Forcing library authors to mark their libraries with a piece of metadata that tells that the annotations should override typeshed seems a bit bureaucratic. We might be better off with (2) + a type checker-dependent notification that typeshed has a stub for a particular library so the user might want to use the stub if they want to.
I like (2) as well. Can one of you formulate a PR to the PEP?
In #1372 (comment) @bdarnell wrote
I need to be able to specify that my .py files contain annotations that should be considered as valid as .pyi files. -s categorically considers .py files as inferior which undermines the rationale for inline annotations.
Actually mypy's search algorithm takes things one directory at a time. If the .py file occurs in a directory that's earlier in MYPYPATH than the .pyi file, the .py file always wins (with -s, it's marked as not found, but mypy doesn't fall back on the .pyi file if it's in a later directory).
When a .py and .pyi are in the same directory, the .pyi always wins (regardless of -s).
I personally find this a very reasonable algorithm. If you want to have independent control over whether your .py or your .pyi files win, put them in different directories, and flip the directory order in MYPYPATH.
It's perfectly reasonable that when both
.pyiand.pyfiles exist as peers, the.pyifile wins. When I said that-sconsiders.pyfiles as "inferior" I didn't just mean lower priority in the search process, I meant that it doesn't consider them at all.Here's my concern: I want to publish a library that contains PEP 484 annotations in the source code, and I want users of this library to be able to type check their code using my annotations. If
-sbecomes common (it is broadly encouraged in e.g. #1339), then I would need to also publish annotations in.pyiform, which I really don't want to do.If there were a tool to automate the creation of
.pyistubs from an annotated.pyfile then I could live with that, but in that case the PEP should be clear that the.pyifiles are the source of truth for type information (for public interfaces) and inline annotations are just an option for authors who prefer to write their annotations that way. The PEP currently gives the opposite impression: type hints are meant to be provided inline, and stubs are a workaround for when it is impractical to use inline annotations.[
-s] doesn't consider [.pyfiles] at allThat's not completely true. The presence of a
.pyfile is still noted and aborts the search right then (from the type analysis POV it's as if the import failed; the module is replaced with a dummy object of typeAny).The way to make it note them is to pass them explicitly on the command line -- or the directory containing the package. All
.pyfiles in the directory will be considered.If -s becomes common (it is broadly encouraged in e.g. #1339), then I would need to also publish annotations in .pyi form, which I really don't want to do.
OK, this is the crux of the matter. I think "encouraged" is too strong a word, and we should really document it in a way that doesn't let people fall in that trap. It's also possible that we'll have to change something, but first I'd like to explore how close we can get without changes to mypy first.
Today, without changes to mypy or typeshed, the way to arrange for what you want without passing the tornado package on the command line would be to download tornado and copy the files into a separate directory, and point MYPYPATH to that directory. The reason for the separate directory is that the version of tornado that's actually used is installed in
site-packages, and that presumably contains other packages that are not mypy-clean (maybe they have stubs in typeshed, maybe not).But just passing the tornado package on the command line (together with the app itself) would be much simpler! In that case pointing to the copy installed in site-packages works just fine, and does the right thing regardless of whether
-sis used. The problem with this solution is that it doesn't scale very well if an app uses lots of packages that use the same strategy as tornado -- and hopefully most packages will eventually adopt that strategy, because it's the ideal strategy!Another problem is that we'd like to have a solution that lets the user set an environment variable once, rather than having to pass things (other than their own app and the flags they like) on the command line. At Dropbox so far we side-step these issues by having a script that runs mypy with the right arguments checked into our repos, but I realize this is not an ideal solution for most use cases.
Perhaps a solution can be found using a central config file as suggested in #1249.
Another possible solution would be to extend the meaning of
MYPYPATHso that if you point it to a top-level package (instead of to a directory full of packages, like site-packages) it will do the right thing: trying to import it will just load the package rather than search it as a directory (which would be nonsensical) and.pyfiles in the package will be loaded regardless of-s. Actually this should probably be a separate environment variable (MYPYPACKAGES?) to avoid confusion -- Python 3 has a concept called implicit namespace packages (PEP 420) that would make it hard to distinguish between a package and a directory full of packages. Does that work for you?A problem with just passing the tornado package on the command line is that this forces mypy to analyze the entire package. Using the
-iflag (which caches the symbol tables of successfully type-checked modules) should speed that up the second time around, but it's still pointless.[-s] doesn't consider [.py files] at all
That's not completely true. The presence of a .py file is still noted and aborts the search right then (from the type analysis POV it's as if the import failed; the module is replaced with a dummy object of type Any).OK,
-sdoesn't consider annotations in.pyfiles at all.I believe that
-sshould be limited to suppressing errors, and if something would have succeeded without error in the absence of-s, it should work the same with it.Passing
tornadoon the command line at the same level as the code I'm working on seems like a non-starter - I wantmypy file_i_am_editing.pyto be fast and do no more work than necessary to find errors in that file.A
MYPYPACKAGESenvironment variable (or equivalent command-line flag that distinguishes these packages from the main input files would work), although it would be weird if this setting only had any effect if-swere used. If this setting were added I think I'd want to make-seffectively the default, and never consider packages unless they're either on the command line or inMYPYPACKAGES.This new setting still seems like a lot of unnecessary manual maintenance, though. I'm envisioning something like a line in
setup.py(a PEP 426 extension) to declare that a package contains PEP 484 annotations, and then mypy could be given asite-packagesdirectory and use this metadata to decide which packages it can look into.10 remaining items
--library-moduledoesn't address my concerns, because it still puts a burden on the application developer to know which of their dependencies have type information available. I want something that lets me point mypy at either a requirements.txt file or an installed environment and makes it figure out what type information it should use. I'm picturing a setting insetup.py(typehints="inline"?) which gets written out into thedist-infometadata.- Sorry about that. Unfortunately I know nothing about how setup.py works and you'll have to spell it out in more detail.
My knowledge of setup.py is thin as well; I was hoping that someone else would be able to fill in the gap behind my hand-waving. A verbose way to do what I'm talking about would be the
entry_pointsfeature. Packages can contain anentry_pointssection in their setup.py, and then other packages (in the linked example, sqlalchemy) can query the system for all installed entry points with a given name. Atypehintsentry point would be one way to do what I'm asking for. I'd like for there to be a less wordy way to do this, but that would be beyond my knowledge of thesetup.pyworld.Usually, entrypoint should be executable. There is some more obscure arguments, such as
package_datawhich can be useful here.Could we salvage option 1 by providing some minimal tooling in mypy (or some other common place) for packaging up
.pyifiles at package build time, and then providing a standard place for installing them (viapackage_data)? At that point, mypy could look at the installed packages for that those.pyifiles.Upsides I see to this approach:
- Library authors don't need to compile their own stub files (those would be built by a tool at package time)
- Mypy doesn't need to parse all the files during imports, it can just look for the pre-packaged
.pyifiles, which will be faster to parse.
I just re-read PEP-484 and found https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-files, which seems fairly close to what I'm already suggesting, but absent the additional tooling to auto-build
.pyifiles at package time.We could use http://setuptools.readthedocs.io/en/latest/setuptools.html#adding-setup-arguments, and add such an entrypoint for mypy, so that you can just add something like
typehints = Truetosetup(...), and it would compile the.pyfiles as.pyifiles into apackage_dataentry (say, in atypehintsdirectory, relative to the package), and then mypy could use thepkg_resourcesapi to find the relevant directory (http://peak.telecommunity.com/DevCenter/PythonEggs#accessing-package-resources).Alternately, perhaps the
typehints = Trueflag could just compile the.pyifiles adjacent to the.pyfiles, at which point the existing mypy lookup methods would Just Work.Hi. Sorry to bother you like this here but is there any official example with stub-only package?
- removed this from the milestone
on Mar 29, 2017 Fixed by #4693.
@ethanhs So now there is a way to make mypy use inline annotations without duplicating code to .pyi? How?
(moved from python/typeshed#52)
Currently mypy (AFAICT) looks for type information in the current directory/location of file being checked, directories on MYPYPATH, and typeshed, in that order, and takes the first file it finds. This will become problematic as more libraries adopt inline type annotations: the user will need to maintain a MYPYPATH that includes all such libraries. The easiest way to do that is to
pip install -r requirements.txtand use that environment'ssite-packagesas mypy's path, but this will include packages that do not have type annotations and these source files may mask stubs for those libraries in typeshed.I propose that rather than stopping the search at the first file that exists, mypy examine source files to see whether they have usable type annotations, and if not, continue the search. If the search concludes without any usable type information but it did find some source files, then the first one can be used (to at least get the list of top-level function names, etc)
Completely:
-mmode, the directory containing the given file if a file is given),$MYPYPATH,$PYTHONPATH, typeshed. I think it would be convenient if there were a shortcut to add thesite-packagesof a given virtualenv (perhaps defaulting to the environment in which mypy is running); this would come in between$PYTHONPATHand typeshed.2a. Look for a
.pyistub file. If one is found, we're done.2b. If no
.pyifile, look for a.pysource file. If it can be parsed and contains type information, we're done. If it can be parsed but does not contain types, and it's the first one we've seen, remember it.