Document new source finding behaviour

docs/source/command_line.rst

@@ -110,10 +110,18 @@ imports.
     passed on the command line, the ``MYPYPATH`` environment variable,
     and the :confval:`mypy_path` config option.
 
-    Note that this only affects import discovery -- for modules and
-    packages explicitly passed on the command line, mypy still
-    searches for ``__init__.py[i]`` files in order to determine the
-    fully-qualified module/package name.
+    This flag affects how mypy finds modules and packages explicitly passed on
+    the command line. It also affects how mypy determines fully qualified module
+    names for files passed on the command line. See :ref:`Mapping file paths to
+    modules <mapping-paths-to-modules>` for details.
+
+.. option:: --explicit-package-bases
+
+    This flag tells mypy that top-level packages will be based in either the
+    current directory, or a member of the ``MYPYPATH`` environment variable or
+    :confval:`mypy_path` config option. This option is only useful in
+    conjunction with :option:`--namespace-packages`. See :ref:`Mapping file
+    paths to modules <mapping-paths-to-modules>` for details.
 
 .. option:: --ignore-missing-imports
 

docs/source/running_mypy.rst

@@ -24,8 +24,11 @@ actual way mypy type checks your code, see our
 Specifying code to be checked
 *****************************
 
-Mypy lets you specify what files it should type check in several
-different ways.
+Mypy lets you specify what files it should type check in several different ways.
+
+Note that if you use namespace packages (in particular, packages without
+``__init__.py``), you'll need to specify :option:`--namespace-packages <mypy
+--namespace-packages>`.
 
 1.  First, you can pass in paths to Python files and directories you
     want to type check. For example::
@@ -336,58 +339,79 @@ while this option can be quite powerful, it can also cause many
 hard-to-debug errors.
 
 
-
 .. _mapping-paths-to-modules:
 
 Mapping file paths to modules
 *****************************
 
-One of the main ways you can tell mypy what files to type check
-is by providing mypy the paths to those files. For example::
+One of the main ways you can tell mypy what to type check
+is by providing mypy a list of paths. For example::
 
     $ mypy file_1.py foo/file_2.py file_3.pyi some/directory
 
 This section describes how exactly mypy maps the provided paths
 to modules to type check.
 
-- Files ending in ``.py`` (and stub files ending in ``.pyi``) are
-  checked as Python modules.
+- mypy will check all paths provided that correspond to files.
+
+- mypy will recursively discover and check all files ending in ``.py`` or
+  ``.pyi`` in directory paths provided.
+
+- For each file to be checked, mypy will attempt to associate the file (e.g.
+  ``project/foo/bar/baz.py``) with a fully qualified module name (e.g.
+  ``foo.bar.baz``). The directory the package is in (``project``) is then
+  add to mypy's module search paths.
 
-- Files not ending in ``.py`` or ``.pyi`` are assumed to be Python
-  scripts and checked as such.
+How mypy determines fully qualified module names depends on if the options
+:option:`--namespace-packages <mypy --namespace-packages>` and
+:option:`--explicit-package-bases <mypy --explicit-package-bases>` are set.
 
-- Directories representing Python packages (i.e. containing a
-  ``__init__.py[i]`` file) are checked as Python packages; all
-  submodules and subpackages will be checked (subpackages must
-  themselves have a ``__init__.py[i]`` file).
+1. If :option:`--namespace-packages <mypy --namespace-packages>` is off,
+   mypy will rely solely upon the presence of ``__init__.py[i]`` files to
+   determine the fully qualified module name. That is, mypy will crawl up the
+   directory tree for as long as it continues to find ``__init__.py`` (or
+   ``__init__.pyi``) files.
 
-- Directories that don't represent Python packages (i.e. not directly
-  containing an ``__init__.py[i]`` file) are checked as follows:
+   For example, if your directory tree consists of ``pkg/subpkg/mod.py``, mypy
+   would require ``pkg/__init__.py`` and ``pkg/subpkg/__init__.py`` to exist in
+   order correctly associate ``mod.py`` with ``pkg.subpkg.mod``
 
-  - All ``*.py[i]`` files contained directly therein are checked as
-    toplevel Python modules;
+2. If :option:`--namespace-packages <mypy --namespace-packages>` is on, but
+   :option:`--explicit-package-bases <mypy --explicit-package-bases>` is off,
+   mypy will allow for the possibility that directories without
+   ``__init__.py[i]`` are packages. Specifically, mypy will look at all parent
+   directories of the file and use the location of the highest
+   ``__init__.py[i]`` in the directory tree to determine the top-level package.
 
-  - All packages contained directly therein (i.e. immediate
-    subdirectories with an ``__init__.py[i]`` file) are checked as
-    toplevel Python packages.
+   For example, say your directory tree consists solely of ``pkg/__init__.py``
+   and ``pkg/a/b/c/d/mod.py``. When determining ``mod.py``'s fully qualified
+   module name, mypy will look at ``pkg/__init__.py`` and conclude that the
+   associated module name is ``pkg.a.b.c.d.mod``.
 
-One more thing about checking modules and packages: if the directory
-*containing* a module or package specified on the command line has an
-``__init__.py[i]`` file, mypy assigns these an absolute module name by
-crawling up the path until no ``__init__.py[i]`` file is found.
+3. You'll notice that the above case still relies on ``__init__.py``. If
+   you can't put an ``__init__.py`` in your top-level package, but still wish to
+   pass paths (as opposed to packages or modules using the ``-p`` or ``-m``
+   flags), :option:`--explicit-package-bases <mypy --explicit-package-bases>`
+   provides a solution.
 
-For example, suppose we run the command ``mypy foo/bar/baz.py`` where
-``foo/bar/__init__.py`` exists but ``foo/__init__.py`` does not.  Then
-the module name assumed is ``bar.baz`` and the directory ``foo`` is
-added to mypy's module search path.
+   With :option:`--explicit-package-bases <mypy --explicit-package-bases>`, mypy
+   will locate the nearest parent directory that is a member of the ``MYPYPATH``
+   environment variable, the :confval:`mypy_path` config or is the current
+   working directory. mypy will then use the relative path to determine the
+   fully qualified module name.
 
-On the other hand, if ``foo/bar/__init__.py`` did not exist, ``foo/bar``
-would be added to the module search path instead, and the module name
-assumed is just ``baz``.
+   For example, say your directory tree consists solely of
+   ``src/namespace_pkg/mod.py``. If you run the command following command, mypy
+   will correctly associate ``mod.py`` with ``namespace_pkg.mod``::
 
-If a script (a file not ending in ``.py[i]``) is processed, the module
-name assumed is ``__main__`` (matching the behavior of the
-Python interpreter), unless :option:`--scripts-are-modules <mypy --scripts-are-modules>` is passed.
+       $ MYPYPATH=src mypy --namespace-packages --explicit-package-bases .
+
+If you pass a file not ending in ``.py[i]``, the module name assumed is
+``__main__`` (matching the behavior of the Python interpreter), unless
+:option:`--scripts-are-modules <mypy --scripts-are-modules>` is passed.
+
+Passing :option:`-v <mypy -v>` will show you the files and associated module
+names that mypy will check.
 
 
 .. _finding-imports:
@@ -407,7 +431,7 @@ This is computed from the following items:
   (a colon-separated list of directories).
 - The :confval:`mypy_path` config file option.
 - The directories containing the sources given on the command line
-  (see below).
+  (see :ref:`Mapping file paths to modules <mapping-paths-to-modules>`).
 - The installed packages marked as safe for type checking (see
   :ref:`PEP 561 support <installed-packages>`)
 - The relevant directories of the
@@ -418,11 +442,6 @@ This is computed from the following items:
     You cannot point to a :pep:`561` package via the ``MYPYPATH``, it must be
     installed (see :ref:`PEP 561 support <installed-packages>`)
 
-For sources given on the command line, the path is adjusted by crawling
-up from the given file or package to the nearest directory that does not
-contain an ``__init__.py`` or ``__init__.pyi`` file. If the given path
-is relative, it will only crawl as far as the current working directory.
-
 Second, mypy searches for stub files in addition to regular Python files
 and packages.
 The rules for searching for a module ``foo`` are as follows: