Warn about missing __init__.py

[JukkaL]

If a package doesn't have an __init__.py file, the error message from mypy isn't very helpful (e.g., Can't find package 'abc'), and sometimes users can't figure this out by themselves. If mypy can't find a package, it could see if the directory exists somewhere in the module search path without an __init__.py, and give a message that says this explicitly.

Examples of potential better messages:

$ mkdir foo
$ mypy -p foo
Can't find valid package 'foo' (foo/ has no __init__.py)
$ mypy -c "import foo"
<string>:1: error: Cannot find module named 'foo' (foo/ has no __init__.py)

[gvanrossum]

Not also that in Python 3, a directory without __init__.py can be a package (there's a PEP for that). But it's intended only for a certain type of "namespace" packages. There once was an issue requesting support for this, but I rejected it because it would break the "crawl up until root of package" algorithm that mypy uses when passed a file inside a package. Maybe though there should be a flag to change that behavior? (My current workaround is "add an __init__.pyi.)

[davidthornton]

I'm not seeing this message come through on release 0.521... I got stung by this, and wanted to prevent anyone else from having to go through this by reminding y'all 😀

[carljm]

At Instagram this behavior has been a common cause of new packages silently not being typechecked, and nobody realizing it until much later. Somebody will add a new package without an __init__.py, and it will work fine (because we are on Python 3, where __init__.py is not actually required), and they won't realize that their new code is not typechecked because mypy can't find it. We run with ignore-missing-imports, for other reasons we can't change at the moment, so it's totally silent.

It would be great to have an option to explicitly specify the package root and not require __init__.py files.

[gvanrossum]

I agree it would be nice to have a way to override this. The "crawl-up" behavior is only needed for files passed on the command line. @carljm -- do you pass files inside __init__-less packages on the command line? If not, maybe in PY3 mode we could just change the behavior when you set $MYPYPATH and implement a module search (when following imports) that's more similar to what Python 3 itself does.

BTW what exactly do you mean by "package root"? For a module m.py inside a package p, would that be the directory p or its parent?

[carljm]

BTW what exactly do you mean by "package root"? For a module m.py inside a package p, would that be the directory p or its parent?

I meant the parent; the directory that would be on sys.path at runtime. Probably "package root" was a bad choice of term; maybe "import root" is better. Finding this directory is the reason mypy crawls up looking for __init__.py files, right?

do you pass files inside init-less packages on the command line?

Not as part of any automated flow, but it's likely that a developer may do this locally when debugging some type errors; it would be confusing if it didn't work correctly. For our automated full typecheck we just run mypy . from our import root. (To be clear, we have a single "import root" directory with multiple top-level packages inside it. We don't currently set MYPYPATH or expect mypy to find any code outside that directory.)

maybe in PY3 mode we could just change the behavior when you set $MYPYPATH and implement a module search (when following imports) that's more similar to what Python 3 itself does.

Not sure I fully understand what you're suggesting here, but it seems like it might be possible to leverage MYPYPATH in a way that would work even if mypy is passed a file within a package; the logic would be something like "if there is a directory on MYPYPATH that is an ancestor of the file we were given, assume the directory on MYPYPATH is the "import root" and don't crawl up directories looking for __init__.py." Then we would just add our import root to MYPYPATH. This isn't 100% ideal, since it would require setting MYPYPATH where currently we don't need to, but that tradeoff would be worth it to solve this problem.

Another option that comes to mind is to assume (maybe opt-in based on a config flag) that the directory containing mypy.ini is the import root. This would work great for our case, but I'm sure there are projects it wouldn't work for.

[JukkaL]

What about something like this in the config file:

[mypy]
import_roots = .

[gvanrossum]

I don't think "root" is a common part of Python's terminology about imports (though it seems to be when talking about distributions). We usually talk about "import path", and it's always thought of as a list of directories; for individual directories we say "path entry".

I like to just add a flag (command line and in mypy.ini) that changes the behavior so that it doesn't try to crawl up the filename to find additional path entries, but instead searches the search path (current directory plus MYPYPATH) for the file and complains loudly if a file isn't on those. And perhaps the flag would also make it so that if you pass any directories on the command line if will treat those directories as path entries.

[carljm]

I like to just add a flag (command line and in mypy.ini) that changes the behavior so that it doesn't try to crawl up the filename to find additional path entries, but instead searches the search path (current directory plus MYPYPATH) for the file and complains loudly if a file isn't on those.

Presuming "searches the search path" implements Py3-compatible behavior (willingness to import packages without __init__.py) in PY3 mode, that sounds fine.

And perhaps the flag would also make it so that if you pass any directories on the command line if will treat those directories as path entries.

Not sure that makes sense. E.g. in our case, say the correct import path entry is /opt/instagram-server. If that's my cwd and I run mypy some/package/, that doesn't mean that I want to add /opt/instagram-server/some/package/ to the import path (doing so would be wrong), it just means I want to typecheck the files inside that particular package.

[gvanrossum]

OK, sounds good. Let's name the flag --namespace-packages (in mypy.ini, namespace_packages = True). I'm pretty busy with other stuff but I'd be willing to look at a PR.

[carljm]

Given the conclusion here, it seems this is a dupe of #1645.

[ethanhs]

This is a duplicate of #1645.

[cclauss]

I have to say that I find this __init__.py requirement completely baffling and it is making it extremely difficult to adopt mypy in projects. Why are we forced to create unneeded files to make mypy run?
https://stackoverflow.com/questions/62831486

--namespace-packages was implemented but it does not seem to do anything to alleviate mypy's __init__.py issues.

https://github.com/TheAlgorithms/Python is a (100k+ GitHub stars) repo of algorithms in Python. Most files are standalone implementations of algorithms with few if any imports. I don't believe that we need packages or namespace packages. I would like to gradually introduce type checking and have determined which directories currently have no mypy issues.
% mypy --ignore-missing-imports arithmetic_analysis backtracking bit_manipulation blockchain boolean_algebra cellular_automata computer_vision digital_image_processing fuzzy_logic genetic_algorithm geodesy knapsack networking_flow scheduling sorts
works perfectly on my local machine returning Success: no issues found in 138 source files
Awesome, let's put that into a pre-commit hook... TheAlgorithms/Python#4272

digital_image_processing/filters/median_filter.py: error: Duplicate module named
    'digital_image_processing.filters.median_filter' (also at 'digital_image_processing/filters/median_filter.py')
digital_image_processing/filters/median_filter.py: note: Are you missing an __init__.py?
    Alternatively, consider using --exclude to avoid checking one of them.

1. Why does this error only appear in a mypy pre-commit job but not when mypy is run on the command line?
2. --namespace-packages seems to have no effect on mypy's results.
3. Could the error message please state the full pathname of the __init__.py file it needs and/or
4. Could the error message please state the actual --exclude value?
5. Why does mypy require an __init__.py when digital_image_processing.filters.median_filter is only defined once in the file digital_image_processing/filters/median_filter.py? The translation between these two should not be rocket science.

I comment out the digital_image_processing line and then mypy has the same complaint about knapsack
I comment out knapsack and mypy complains about computer_vision.
I comment out computer_vision and mypy complains about sort...

Is there a tool that I can run that will create mypy's required __init__.py files throughout the repo?

I know that type checking will help us find bugs but I am unclear how to implement it in projects. Thanks for all the great work!

[hauntsaninja]

It's at best confusing to post on dead, years old issues. Pretty much everything talked about here is out of date (including your stackoverflow link, as it happens). In case other people stumble upon this, you're best served by reading https://mypy.readthedocs.io/en/latest/running_mypy.html#mapping-file-paths-to-modules

In your case, it's clear the issue is with pre-commit. I'm guessing what's happening is pre-commit is using git magic to additionally pass in a bunch of files in the command. This obviously isn't what you want, since you want typing enforced only on a couple folders and it's obviously not what mypy wants, since mypy is complaining about the duplication.

It looks like the relevant pre-commit incantation is setting pass_filenames to false. In general, I recommend doing this when using pre-commit with mypy.

[anikolaienko]

Definitely read comment the most recent comment above.
But long story short just add namespace_packages = True to your mypy config.