Search procedure for inline annotations

[bdarnell]

(moved from python/typeshed#52)

Currently mypy (AFAICT) looks for type information in the current directory/location of file being checked, directories on MYPYPATH, and typeshed, in that order, and takes the first file it finds. This will become problematic as more libraries adopt inline type annotations: the user will need to maintain a MYPYPATH that includes all such libraries. The easiest way to do that is to pip install -r requirements.txt and use that environment's site-packages as mypy's path, but this will include packages that do not have type annotations and these source files may mask stubs for those libraries in typeshed.

I propose that rather than stopping the search at the first file that exists, mypy examine source files to see whether they have usable type annotations, and if not, continue the search. If the search concludes without any usable type information but it did find some source files, then the first one can be used (to at least get the list of top-level function names, etc)

Completely:

1. Construct a search path, mimicking python's sys.path initialization: an initial element based on the command line (the current directory in -m mode, the directory containing the given file if a file is given), $MYPYPATH, $PYTHONPATH, typeshed. I think it would be convenient if there were a shortcut to add the site-packages of a given virtualenv (perhaps defaulting to the environment in which mypy is running); this would come in between $PYTHONPATH and typeshed.
2. For each directory in this search path:
   2a. Look for a .pyi stub file. If one is found, we're done.
   2b. If no .pyi file, look for a .py source file. If it can be parsed and contains type information, we're done. If it can be parsed but does not contain types, and it's the first one we've seen, remember it.
3. If no type information was found but we remembered a file in 2b, use whatever we can squeeze out of it.

[JukkaL]

I agree that it would be nice if mypy could do the right thing automatically when a library module has inline annotations. However, this proposal has a few problems:

• If a library is partially annotated, mypy could pick some files from the implementation and some from stubs. As these could refer to different versions of a library, this could be confusing or generate bogus errors.
• Some modules may only contain variable definitions without any functions, and for these it's not easy to see whether a module is "annotated", as all annotations are optional.
• Parsing a file to decide whether is has annotations would be slow, especially with the current parser implementation in mypy. We could cache the results but this is still not optimal.
• The target Python version/configuration may be different from the one running mypy. Thus determining sys.path is difficult in general. One potential way to do this is to give mypy a path to a Python interpreter and have mypy run /target/path/python -c 'import sys; print(sys.path)' or similar.
• Even if a file has type annotations, there is no guarantee that mypy can process the file without errors. Maybe the annotations are only for PyCharm or human readers, and mypy would generate type errors.

Here are some further ideas:

• Maybe a library with inline annotations should ship with .pyi files automatically generated from the inline annotations. If we have a convention for where to look for them, they could always be considered as more authoritative than stubs from typeshed.
• A library with inline annotations could ship with some marker file or other well-known flag that signals that mypy should look at the implementation. Also, this signals that the library author has verified that mypy can cleanly process the module implementation. This has the issue of potential mypy version conflicts. If this doesn't work for some reason, users should be able to not use automatic sys.path traversal and manually decide where to look for annotations.
• We could ask authors of libraries with inline annotations to generate stubs using the aforementioned (hypothetical) tool and contribute them to typeshed. This would have known challenges such as dealing with multiple library versions, and deciding what to do if typeshed is out of date for some reason (e.g. everyone who can process PRs is busy).

@bdarnell What do you think?

[bdarnell]

I'd be happy to include some sort of package-level marker to indicate that the package contains PEP 484 type annotations and these should be used instead of typeshed (and that these decisions should be made at package scope instead of file-by-file). I'd rather not make this mypy-specific since mypy is not supposed to be the only consumer of PEP 484 annotations. If mypy can't handle the annotations in a package then it should either fail or fall back to typeshed for that package.

It would be unfortunate if the only use of inline annotations was in a tool that extracted them to .pyi stub files. If that were the case I would probably write stub files by hand instead of wrestling with inline annotations (which require runtime dependencies on typing and other intrusions on the production code).

Requiring all stubs to be submitted to typeshed seems like the worst possible outcome - I was under the impression that typeshed was essentially a transitional thing, and the long-term goal was for type information to migrate from typeshed into the respective packages. I think centralizing everything in typeshed would discourage participation by package authors.

[gvanrossum]

I haven't forgotten this (I've just been distracted). We just encountered an issue here where there's a discrepancy between the way PyCharm and mypy search for stubs. In this case it was about an extension module, for which the user had created a stub file (but not added it to typeshed). Mypy found it fine on the default module search path. But PyCharm would only find it if there was a corresponding .py file in the same directory. In their case the fix was to add a .py file that simple re-exported the extension module. But it would be nice to have some kind of standardization for this.

[vlasovskikh]

PEP 484 is not clear about several things about the search path for stubs. I've mentioned some of them in a comment to python/typing#184.

@bdarnell Shouldn't *.pyi files from typeshed override the *.py files from installed packages? (according to the PEP). If there is no installed top-level package for the module in typeshed, then I believe the type checker could skip the typeshed stub for it.

[bdarnell]

@vlasovskikh Why would typeshed annotations take precedence over annotations in the code itself? The files in typeshed may describe a different version of the package and the annotations closer to the code are more likely to be correct for that version. If typeshed takes precedence then it is impossible to transition from annotations in typeshed to annotations in the code itself, and packages are being added to typeshed without coordinating with the package authors.

[vlasovskikh]

@bdarnell Given the following variants (please correct me if I'm missing something):

1. Typeshed annotations first, then the annotations inside the code (until the user changes the priority of annotations lookup)
2. Only the annotations inside the code (until the user changes the priority of annotations lookup)
3. The annotations inside the code first if there are any (may be time-consuming to detect), then typeshed annotations (until the user changes the priority of annotations lookup)

I would prefer the variant (1) here. Speaking of (3) it may take a while to determine if any files inside a package contain type hints, so it will slow the code analysis down making it hard to run it on-the-fly. Also (3) implies that any annotation added to a module (say, in the form of a # type: ... comment) forces a type checker to switch from typeshed to the real code, this might confuse the user since all the sudden the code analysis for a slightly updated version of the library starts to miss many errors inside their code.

[bdarnell]

You're assuming that typeshed annotations will be higher-quality than in-code annotations; I am assuming the opposite. More to the point, if I as the author of a package add type annotations to that package, that's how I want my users' code to be type-checked. I don't want people coming to my support forums with type-checking issues that are caused by version skew between typeshed and the actual code they have installed.

Typeshed currently contains near-useless stubs for my project which were added without my knowledge. The presence of these stubs stands in the way of me adding my own annotations to the code. This is frustrating and discourages me from exploring type annotations further.

Your option 3 is the only one that makes sense to me as a library author. I'm fine with adding some piece of metadata somewhere to say "this package uses PEP 484 annotations which should be used by type checkers" so that you don't have to actually load the code to find the annotations (and to address the possibility of incompletely-annotated projects that aren't ready to be checked without typeshed support).

[vlasovskikh]

@bdarnell I see your point. Forcing library authors to mark their libraries with a piece of metadata that tells that the annotations should override typeshed seems a bit bureaucratic. We might be better off with (2) + a type checker-dependent notification that typeshed has a stub for a particular library so the user might want to use the stub if they want to.

[gvanrossum]

I like (2) as well. Can one of you formulate a PR to the PEP?

[gvanrossum]

In #1372 (comment) @bdarnell wrote

I need to be able to specify that my .py files contain annotations that should be considered as valid as .pyi files. -s categorically considers .py files as inferior which undermines the rationale for inline annotations.

Actually mypy's search algorithm takes things one directory at a time. If the .py file occurs in a directory that's earlier in MYPYPATH than the .pyi file, the .py file always wins (with -s, it's marked as not found, but mypy doesn't fall back on the .pyi file if it's in a later directory).

When a .py and .pyi are in the same directory, the .pyi always wins (regardless of -s).

I personally find this a very reasonable algorithm. If you want to have independent control over whether your .py or your .pyi files win, put them in different directories, and flip the directory order in MYPYPATH.

[bdarnell]

It's perfectly reasonable that when both .pyi and .py files exist as peers, the .pyi file wins. When I said that -s considers .py files as "inferior" I didn't just mean lower priority in the search process, I meant that it doesn't consider them at all.

Here's my concern: I want to publish a library that contains PEP 484 annotations in the source code, and I want users of this library to be able to type check their code using my annotations. If -s becomes common (it is broadly encouraged in e.g. #1339), then I would need to also publish annotations in .pyi form, which I really don't want to do.

If there were a tool to automate the creation of .pyi stubs from an annotated .py file then I could live with that, but in that case the PEP should be clear that the .pyi files are the source of truth for type information (for public interfaces) and inline annotations are just an option for authors who prefer to write their annotations that way. The PEP currently gives the opposite impression: type hints are meant to be provided inline, and stubs are a workaround for when it is impractical to use inline annotations.

[gvanrossum]

[-s] doesn't consider [.py files] at all

That's not completely true. The presence of a .py file is still noted and aborts the search right then (from the type analysis POV it's as if the import failed; the module is replaced with a dummy object of type Any).

The way to make it note them is to pass them explicitly on the command line -- or the directory containing the package. All .py files in the directory will be considered.

If -s becomes common (it is broadly encouraged in e.g. #1339), then I would need to also publish annotations in .pyi form, which I really don't want to do.

OK, this is the crux of the matter. I think "encouraged" is too strong a word, and we should really document it in a way that doesn't let people fall in that trap. It's also possible that we'll have to change something, but first I'd like to explore how close we can get without changes to mypy first.

Today, without changes to mypy or typeshed, the way to arrange for what you want without passing the tornado package on the command line would be to download tornado and copy the files into a separate directory, and point MYPYPATH to that directory. The reason for the separate directory is that the version of tornado that's actually used is installed in site-packages, and that presumably contains other packages that are not mypy-clean (maybe they have stubs in typeshed, maybe not).

But just passing the tornado package on the command line (together with the app itself) would be much simpler! In that case pointing to the copy installed in site-packages works just fine, and does the right thing regardless of whether -s is used. The problem with this solution is that it doesn't scale very well if an app uses lots of packages that use the same strategy as tornado -- and hopefully most packages will eventually adopt that strategy, because it's the ideal strategy!

Another problem is that we'd like to have a solution that lets the user set an environment variable once, rather than having to pass things (other than their own app and the flags they like) on the command line. At Dropbox so far we side-step these issues by having a script that runs mypy with the right arguments checked into our repos, but I realize this is not an ideal solution for most use cases.

Perhaps a solution can be found using a central config file as suggested in #1249.

Another possible solution would be to extend the meaning of MYPYPATH so that if you point it to a top-level package (instead of to a directory full of packages, like site-packages) it will do the right thing: trying to import it will just load the package rather than search it as a directory (which would be nonsensical) and .py files in the package will be loaded regardless of -s. Actually this should probably be a separate environment variable (MYPYPACKAGES?) to avoid confusion -- Python 3 has a concept called implicit namespace packages (PEP 420) that would make it hard to distinguish between a package and a directory full of packages. Does that work for you?