mypy doesn't recurse through directories, but -h says it does

[rvandegrift]

According to mypy -h:

Mypy is a program that will type check your Python code.

Pass in any files or folders you want to type check. Mypy will
recursively traverse any provided folders to find .py files:

    $ mypy my_program.py my_src_folder

But it doesn't recurse, it only checks what's explicitly provided on the command line:

$ python3 -m venv venv
$ . venv/bin/activate
(venv) $ pip install mypy
...
(venv) $ mkdir -p test/test/
(venv) $ touch test/test/a.py
(venv) $ mypy test
There are no .py[i] files in directory 'test'
(venv) $ touch test/b.py
(venv) $ mypy test
Success: no issues found in 1 source file
(venv) $ mypy test test/test
Success: no issues found in 2 source files

The fix might be to change the documentation. This is easy to workaround with find.

[Michael0x2a]

I believe this is because you're using namespace packages instead of regular packages. If you try running:

mypy --namespace-packages -p test

...mypy should detect that you have two source files. The command line docs should have more details about both flags.

Alternatively, you could do:

touch test/__init__.py
touch test/test/__init__.py

...to convert your package back into a regular one. Then running mypy test should detect four files, as expected.

[rvandegrift]

That might fix the above example. But my actual use case is to check any python files regardless of where in the directory hierarchy they appear. So I actually want the recursive behavior described in the help text.

[JukkaL]

@rvandegrift Can you give a more concrete example of a directory hierarchy where -p FOO doesn't work for you? You can give multiple -p FOO arguments.

[rvandegrift]

Well, there's no concrete directory hierarchy that lead me here.

I wanted to enable mypy on all of our repos via a single script in a shared CI pipeline. The script won't know which repo it's running for ahead of time. And since we're human and messy, we have different layouts in different repos. So I can't supply any -p FOO arguments, since I don't know what FOO is.

One of my team members put together a quick hack to find package roots and run once for each: https://github.com/jceresini/mypy_r

I'm not sure that this is a valuable use case. So maybe the only mypy issue is that the help text doesn't describe the actual operation.

[rjmorris]

The following directory hierarchy led me to this issue:

- setup.py
+ src
  + mypkg
    - __init__.py
    - module1.py
    - module2.py
+ tests
  - test_1.py
  - test_2.py

So this is a src layout, not a namespace package.

As @rvandegrift quoted from the help text:

Mypy will recursively traverse any provided folders to find .py files

Based on this statement, I expect that invoking mypy via mypy . will cause mypy to check all the .py files in my hierarchy. However, setup.py is the only one it checks. To check everything, I need to invoke it as mypy setup.py src/ tests/. And that's fine, I have no problem invoking it that way, but the behavior doesn't match what I'd expect given the documentation.

The description from the online docs is similar, although it doesn't specifically say it finds .py files:

In addition, mypy will recursively type check the entire contents of any provided directories.

I think the words "recursively" and "entire" are strong signals that it will descend down into every directory looking for every .py file, which isn't the case. A link to further details is helpfully provided right after this, where we learn how it actually works:

Directories that don’t represent Python packages (i.e. not directly containing an __init__.py[i] file) are checked as follows:

• All *.py[i] files contained directly therein are checked as toplevel Python modules;
• All packages contained directly therein (i.e. immediate subdirectories with an __init__.py[i] file) are checked as toplevel Python packages.

In the case of mypy ., the current directory doesn't contain an __init__.py, so the above logic applies. setup.py is contained directly therein, so that one is checked. But neither of the immediate subdirectories src or test contain an __init__.py, so they aren't checked. Stopping after one subdirectory level is hardly what I'd call recursive.

I understand that I can address the issue by invoking mypy differently. But like @rvandegrift suggested, I think the solution to this particular issue is to update the help text and docs so that they aren't misleading. I suggest removing the references to "recursive", keeping things concise by leaving the directory rules a bit vague, and pointing to where the procedure is described in greater detail.

For example, the help text could be changed from:

Pass in any files or folders you want to type check. Mypy will recursively traverse any provided folders to find .py files:

to:

Pass in any files or folders you want to type check:

Links to further documentation are already provided right after the sample command, so no need to add those.

And the online docs could be changed from:

The above command tells mypy it should type check all of the provided files together. In addition, mypy will recursively type check the entire contents of any provided directories.

For more details about how exactly this is done, see Mapping file paths to modules.

to:

The above command tells mypy it should type check all of the provided files together. In addition, mypy will type check the contents of any provided directories. For more details about how exactly this is done, see Mapping file paths to modules.

If you wanted to provide more details about the directory checking up front, maybe something like this:

For each provided directory, mypy will type check every package it finds in that directory and its immediate subdirectories, and it will type check all .py files it finds in that directory.

[ulope]

I find this behaviour highly surprising and unintuitive.

We just discovered that this caused a part of our source tree not to be checked for over a year...

[gvanrossum]

I am convinced. We just need to fix this. Someone please send a PR that makes this properly recursive. There is quite a difference between mypy <dir> and mypy -p <dir> because the latter takes a package name (a valid Python identifier).

If it breaks people they can fix it by writing something like mypy <dir>/*.py. (Hm, does the files config flag allow globs?)

[Eisbrenner]

my layout:

- pyproject.toml
- setup.cfg
+ src/
  + my_package_name/
    - __init__.py
    + subm/
      - __init__.py
+ tests/
   + test_group1/
      - test_*.py

with this layout, the following works (I have not yet found a way to combine the two into one command, for example using the configuration file)

$ MYPYPATH=src mypy -p my_package_name
$ mypy tests/**/*.py

However, this really is a nuisance compared to mypy . - which I expect to work from:

"Pass in any files or folders you want to type check. Mypy will recursively traverse any provided folders to find .py files"

$ mypy .                  
There are no .py[i] files in directory '.'

I agree with rjmorris above, that at least the help and docs need to be updated. I would also appreciate a less cumbersome method to use mypy for a src-layout package tho.

[gvanrossum]

There is no need to post more evidence to this issue. What we need is someone to volunteer a PR. The code is pretty straightforward (assuming you know how to iterate over directories recursively), it should all be contained in mypy/main.py.

[hauntsaninja]

I agree that this causes bad surprises and needs a solution.

I'll also mention that fixing this might be a good time to figure out / have a plan for #5759 and #8584 (assuming we want to be able to have some sort of support for checking namespace packages by passing directory arguments).