"Source file found twice under different module names" with namespace package installed in virtualenv by "pip install -e ."

[gward]

Bug Report

At work, we have a standard project layout using namespace packages with dev/build/test dependencies declared in setup.py. It's a bit complex, but it worked well up until mypy 0.790. Since mypy 0.800, mypy is complaining: "Source file found twice under different module names".

This appears to be a rather obscure corner case requiring:

• implicit namespace packages
• dev dependencies declared in setup.py, rather than a requirements file
• use of pip install -e .[dev] to install those dependencies
• running mypy on directories and files (not packages or modules)

Remove any of those conditions and mypy behaves as expected (ie. back to what we had under 0.790). But since it worked nicely under 0.790, it sure feels like a regression to me.

Here's a simplified version of the project layout, for a package fn.common with one module and unit tests:

.
├── fn
│   └── common
│       ├── __init__.py
│       ├── py.typed
│       └── utils.py
├── setup.py
└── tests
    ├── __init__.py
    └── test_utils.py

fn is an implicit namespace package named after the company. We're Python 3 only, so we have no need for legacy namespace packages. Every project, whether library or application, puts all production code in fn/<something>. Nice and clean and tidy and consistent.

The source code of the package isn't important. (There are type hints and no type errors.)

What matters is the contents of setup.py, the way we initialize our virtualenv, and how we run mypy. Here's setup.py:

import os
from setuptools import setup

install_requires = []          # type: ignore
dev_requires = ["mypy==0.790"]

packages = ["fn.common"]
package_data = {"fn.common": ["py.typed"]}

setup(
    name="fnlib-common",
    description="",
    url="",
    author="",
    author_email="",
    packages=packages,
    package_data=package_data,
    install_requires=install_requires,
    extras_require={"dev": dev_requires},
)

This lets us create a virtualenv as follows:

$ /usr/local/bin/python3.8 -m venv venv
$ source venv/bin/activate
$ pip install -e '.[dev]'

Since I've pinned mypy to 0.790, it works fine:

$ mypy --ignore-missing-imports --check-untyped-defs fn tests *.py
Success: no issues found in 5 source files

If I upgrade to 0.812, it stops working:

$ pip install mypy==0.812
$ mypy --ignore-missing-imports --check-untyped-defs fn tests *.py
fn/common/__init__.py: error: Source file found twice under different module names: 'common' and 'fn.common'
Found 1 error in 1 file (errors prevented further checking)

The first release showing this behaviour was 0.800. I used git bisect to track down the responsible commit: e4131a5.

To Reproduce

1. get my source code:

$ wget https://www.gerg.ca/mypy/fnlib-common.tar.gz
$ tar -xzvf ~/fnlib-common.tar.gz
$ cd fnlib-common

2. setup virtualenv:

$ python3 -m venv venv
$ source venv/bin/activate
$ pip install -e '.[dev]'       # installs mypy 0.790

3. run with mypy 0.790: no problem

$ mypy --ignore-missing-imports fn tests *.py
Success: no issues found in 5 source files

4. upgrade to mypy >= 0.800 and see the problem

$ pip install mypy==0.800     # or later
$ mypy --ignore-missing-imports fn tests *.py
fn/common/__init__.py: error: Source file found twice under different module names: 'common' and 'fn.common'
Found 1 error in 1 file (errors prevented further checking)

Expected Behavior

Same output from mypy >= 0.800 as from mypy 0.790, i.e. no errors.

Actual Behavior

Errors from mypy >= 0.800.

Your Environment

host 1:

$ lsb_release -a
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.2 LTS
Release:	20.04
Codename:	focal

$ /usr/local/bin/python3.8 --version
Python 3.8.6+

host 2:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 10 (buster)
Release:	10
Codename:	buster

$ /usr/bin/python3.7 --version
Python 3.7.3

Same results on host 1 and host 2.

No mypy config files -- command-line only.

[gward]

I'm aware of several possible workarounds. Some of them might work, but I thought I should report the regression before implementing a workaround.

Workaround 1: run mypy twice

First, run it on packages, then run it on files:

mypy --ignore-missing-imports --check-untyped-defs -p fn.common -p tests && \
mypy --ignore-missing-imports --check-untyped-defs *.py

This is a bit ugly and bound to be slower. If it's just startup overhead, I can live with it. But if there is cache thrashing, no thank you!

Workaround 2: fake __init__ files

I cannot create fn/__init__.py, because that breaks namespace packages. But I can create fn/__init__.pyi to fool mypy without angering python. I think this is the least bad option, but it will be easy to forget in new projects.

Workaround 3: undo the "pip install -e ."

The easiest way to install the dev dependencies declared in setup.py is with pip install -e '.[dev]'. But that also installs a link to the current project, fnlib-common, right in the local virtualenv. Remove that and mypy is happy:

$ pip uninstall fnlib-common
$ mypy --ignore-missing-imports fn tests *.py
Success: no issues found in 5 source files

Verrry strange.

HOWEVER, I vaguely recall that we had a good reason for doing pip install -e . somewhere, somewhy, somehow. It might have been because of flask or uwsgi in one of our web apps, and I just did it everywhere for consistency. Need to investigate to see if this is a good option.

Workaround 4: use a dev-requirements.txt file

Stop declaring dev dependencies in setup.py, and put them in a requirements file. Then I don't need pip install -e '.[dev]' ... unless, of course, I do need it for other reasons that I do not remember right now!

[hauntsaninja]

I think https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-explicit-package-bases should work for you, see https://mypy.readthedocs.io/en/stable/running_mypy.html#mapping-paths-to-modules

[gward]

Thank you @hauntsaninja -- that seems to work, at least in my minimal test case. I still need to try it out on the real code, but this looks useful!