Find library stubs in __pypackages__

[pawamoy]

Feature

It seems mypy cannot find library stubs when developing a project using PEP582, for example through PDM. Not opening as a bug but rather as a feature request because PEP582 is still quite recent and therefore expected not to be supported everywhere.

$ pdm add -ds typing types-Jinja2
Adding packages to typing dev-dependencies: types-jinja2
✔ 🔒 Lock successful
Changes are written to pdm.lock.
Changes are written to pyproject.toml.
Synchronizing working set with lock file: 2 to add, 0 to update, 0 to remove

  ✔ Install types-Jinja2 2.11.0 successful
  ✔ Install types-MarkupSafe 1.1.0 successful

🎉 All complete!

$ ls __pypackages__/3.6/lib/jinja2-stubs/
bccache.pyi  compiler.pyi   debug.pyi     environment.pyi  ext.pyi      __init__.pyi  loaders.pyi    meta.pyi   optimizer.pyi  runtime.pyi  _stringdefs.pyi  utils.pyi
_compat.pyi  constants.pyi  defaults.pyi  exceptions.pyi   filters.pyi  lexer.pyi     METADATA.toml  nodes.pyi  parser.pyi     sandbox.pyi  tests.pyi        visitor.pyi

$ pdm run  mypy --config-file config/mypy.ini src/project tests duties.py
src/project/redacted.py:8: error: Library stubs not installed for "jinja2.sandbox" (or incompatible with Python 3.6)
src/project/redacted.py:8: note: Hint: "python3 -m pip install types-Jinja2"
src/project/redacted.py:8: note: (or run "mypy --install-types" to install all missing stub packages)
src/project/redacted.py:8: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 16 source files)

Pitch

Support running mypy in projects managed by tools like PDM that take advantage of PEP582.

[pawamoy]

I've tried setting MYPY_PATH to __pypackages__/3.6/lib but of course it did not work, as expected and documented.
Then I tried creating a temporary directory and symlinking only the -stubs packages inside that temp dir, and set MYPY_PATH to that temp dir, but again, Mypy does not acknowledge the stubs 😕
If anyone has another workaround idea, please share!

[pawamoy]

Is there a way to ignore such errors in the mean time?

UPDATE: the following configuration seems to do the trick.

[mypy-jinja2.*]
ignore_missing_imports = True

[blueyed]

The variable to set is MYPYPATH (without underscore).

But it still does not work:

% MYPYPATH=$PWD/__pypackages__/3.9/lib mypy proj
mypy: "__pypackages__/3.9/lib/typing_extensions.py" shadows library module "typing_extensions"
note: A user-defined top-level module with name "typing_extensions" is not supported

It appears to work then creating a separate directory and linking django-stubs and django into it, and then use that with MYPYPATH:

% ls -l MYPYPATH
… django -> ../__pypackages__/3.9/lib/django/
… django-stubs -> ../__pypackages__/3.9/lib/django-stubs/
% MYPYPATH=$PWD/MYPYPATH mypy proj

/cc @frostming

[simon-fothergill]

I would love it if mypy became PEP 582 compatible.

You can install things like mypy and black elsewhere (e.g. globally with pipx) so they are not in __pypackages__ and then set MYPYPATH to look in the __pypackages__ directory.

I'm not sure if this has other side effects, though. It certainly type checks everything unless you set it to skip following all imports, which seems equivalent to ignoring missing imports. But you can silence following imports to give added safety.

Please feel free to comment on this approach.

Best wishes!

[pawamoy]

Using @blueyed solution, I've been able to automate that temporary directory creation with symlinks to packages of interest. It's implemented as a duty (my task runner), but could easily be rewritten for other task runners.

UPDATE: also support .pth files (install_cache feature of PDM)

@duty
def check_types(ctx):
    # NOTE: the following code works around this issue:
    # https://github.com/python/mypy/issues/10633

    # compute packages directory path
    py = f"{sys.version_info.major}.{sys.version_info.minor}"
    pkgs_dir = Path("__pypackages__", py, "lib").resolve()

    # build the list of available packages
    packages = {}
    for package in pkgs_dir.glob("*"):
        if package.suffix not in {".dist-info", ".pth"} and package.name != "__pycache__":
            packages[package.name] = package

    # handle .pth files
    for pth in pkgs_dir.glob("*.pth"):
        with suppress(OSError):
            for package in Path(pth.read_text().splitlines()[0]).glob("*"):
                if package.suffix != ".dist-info":
                    packages[package.name] = package

    # create a temporary directory to assign to MYPYPATH
    with tempfile.TemporaryDirectory() as tmpdir:

        # symlink the stubs
        ignore = set()
        for stubs in (path for name, path in packages.items() if name.endswith("-stubs")):
            Path(tmpdir, stubs.name).symlink_to(stubs, target_is_directory=True)
            # try to symlink the corresponding package
            # see https://www.python.org/dev/peps/pep-0561/#stub-only-packages
            pkg_name = stubs.name.replace("-stubs", "")
            if pkg_name in packages:
                ignore.add(pkg_name)
                Path(tmpdir, pkg_name).symlink_to(packages[pkg_name], target_is_directory=True)

        # create temporary mypy config to ignore stubbed packages
        newconfig = Path("config", "mypy.ini").read_text()
        newconfig += "\n" + "\n\n".join(f"[mypy-{pkg}.*]\nignore_errors=true" for pkg in ignore)
        tmpconfig = Path(tmpdir, "mypy.ini")
        tmpconfig.write_text(newconfig)

        # set MYPYPATH and run mypy
        os.environ["MYPYPATH"] = tmpdir
        ctx.run(f"mypy --config-file {tmpconfig} {PY_SRC}", title="Type-checking", pty=PTY)

There's no --ignore-errors=module command line option so I have to create a temporary config file to ignore the stubbed packages, otherwise mypy shows me errors for those.

[JaviFuentes94]

Any updates on this? It would be really helpful

[phiresky]

Does anyone have a workaround for modules that ship their own types? I can't figure out how to get mypy to not either throw

error: Cannot find implementation or library stub for module named "torch"

or to throw tons of errors for the code in the torch package itself.

I've tried linking the torch package to a directory that I then set to MYPYPATH.

[iyume]

I'm working on PEP582 globally and confused about the following error message after adding mypy_path = $MYPY_CONFIG_FILE_DIR/__pypackages__/3.9/lib to mypy.ini.

mypy: "__pypackages__/3.9/lib/typing_extensions.py" shadows library module "typing_extensions"

maybe a side effects?

[martimors]

Facing this issue with PEP582 (pdm package manager).

[patrick91]

Do you have this package installed by any chance? https://pypi.org/project/types-typing-extensions/