Due to limitation, static analysis don't support import hook used in editable install v64+

[JacobHayes]

setuptools version

setuptools==64.0.2

Python version

3.10

OS

macOS, Linux

Additional environment information

No response

Description

After installing editable packages (with a pyproject.toml) with setuptools >=64, mypy is no longer able to find the py.typed files when checking a codebase importing/referencing the editable installed package.

This was opened as a mypy issue in python/mypy#13392, but they recommended we open an issue here. I'm guessing there will be some further discussion on both sides, but just getting the ball rolling.

Expected behavior

Editable installs are still discoverable by static type checkers.

How to Reproduce

See the README in https://github.com/JacobHayes/editable-install-hooks-repro

Output

$ mypy --namespace-packages --explicit-package-bases src
src/org/pkg2/__init__.py:1: error: Cannot find implementation or library stub for module named "org.pkg1"
src/org/pkg2/__init__.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)
$ pylint src/
************* Module org.pkg2
src/org/pkg2/__init__.py:2:0: E0611: No name 'pkg1' in module 'org' (no-name-in-module)

-----------------------------------
Your code has been rated at 0.00/10

[abravalheri]

Hi @JacobHayes thank you very much for bringing this topic up.

A different way of describing the issue would be stating that tools doing static analysis do not support import hooks (which is a very fair limitation, it makes total sense). Please note that this is not specific to setuptools. For example other backend using the editables might also rely on import hooks. I would also not classify it as a bug.

The links you shared in your example repository are very relevant (specially the email thread). In addition to that I would also mention:

• https://discuss.python.org/t/help-testing-pep-660-support-in-setuptools/16904/41
• https://discuss.python.org/t/pep-660-support-and-ides/13878

Looking on a way forward on how to handle these circumstances, I believe that first we need to design a solution having inputs and buy-in from both sides (packaging and static analysis).

One of the solutions proposed in the email thread (a .json file with metadata pointing out the locations for each package) seems to be a good start point. I am very happy to support a similar solution that would enhance the experience with import hooks.

To pursue this topic further in the setuptools side we would need consensus on such mechanism. This should probably come as a packaging PEP.

I don't know if the setuptools issue tracker is the right forum for proceeding with this discussion, since there is no clear action point for setuptools to take.
I recommend anyone interested in such a feature to engage in the mailing list discussion, or on the thread on discourse. We probably need a champion to push it forward :)

For the time being, if you need to integrate type checking and editable installations, you can try out the strict mode, or if you own the project to swap to a src-layout.

[JacobHayes]

Why move forward with the install hooks if it was known there was no consensus yet? setuptools (v64 and used ~everywhere) seems quite different than editables (v0.3 and used in a couple spots / where?).

try out the strict mode

I'll test this out within the next couple days. Is there any reason this isn't the default, to reduce the number of breaking workflows by default?

if you own the project to swap to a src-layout.

Can you elaborate on this? I think the example repro repo I linked is in src layout, but still does not work. Is there a part I'm missing there?

--

Thanks for your work on setuptools, I know packages like this in particular are in a hard spot in regards to feature evolution.

[abravalheri]

Why move forward with the install hooks if it was known there was no consensus yet? setuptools (v64 and used ~everywhere) seems quite different than editables (v0.3 and used in a couple spots / where?).

There is consensus in terms of packaging. This is expressed in terms of PEP 660, which was debated over and over again in the community. The PEP's text recognizes import hooks as perfectly valid implementation strategy. Import hooks have been part of Python's feature list for a long time. There are even other built-in behaviours (e.g. zip imports) that rely on these dynamics. Just because they are features that static analysis tools have problems to deal with, it does not mean that they are invalid.

As an anecdote: hasattr() is an extremely useful feature of Python and until the other day, code analysers/type checkers had problems to deal with it. It does not mean that everyone should stop using it...

The lack of consensus that I describe is in what Python's package ecosystem can do to make it easier for static analysis tools to deal such kind of implementation.

Is there any reason this isn't the default, to reduce the number of breaking workflows by default?

The strict mode potentially is not what users expect to find by default. When using editable installs, users expect to be able to create new files, or delete old ones and have them automatically picked up without requiring a new installation. The strict mode does not offer that capability.

To offer a "minimal surprise" experience, the default editable installation tries to be a bit more "lenient"...

Can you elaborate on this? I think the example repro repo I linked is in src layout, but still does not work. Is there a part I'm missing there?

It is very definitely almost what I would expect for a src-layout... However, there is a very subtle customization that you are passing to setuptools, which requires special ad-hoc handling.

When you type find_namespace_packages(include=["org.*"], ...), you are effectively instructing setuptools to take any sub-package of org (e.g. org.pkg1 or org.pkg2) from your project directory and put that in the distribution, but not the org package itself (org does not match org.*). This makes setuptools' life harder, since it has to guarantee that the immediate contents of the org folder other than subpackages like pkg1 or pkg2 don't end up in the distribution by mistake. The way setuptools replicates this behaviour in the editable installation is by using import hooks (or file links in the case of the strict mode).

The following change may make what you want possible:

diff --git i/pkg1/setup.py w/pkg1/setup.py
index 406504e..5097881 100644
--- i/pkg1/setup.py
+++ w/pkg1/setup.py
@@ -3,7 +3,7 @@ from setuptools import find_namespace_packages, setup
 setup(
     name="org-pkg1",
     version="0.0.1",
-    packages=find_namespace_packages(include=["org.*"], where="src"),
+    packages=find_namespace_packages(include=["org*"], where="src"),
     package_dir={"": "src"},
     package_data={"org.pkg1": ["py.typed"]},
     python_requires=">=3.9",
diff --git i/pkg2/setup.py w/pkg2/setup.py
index 5ee4648..a48c94f 100644
--- i/pkg2/setup.py
+++ w/pkg2/setup.py
@@ -3,7 +3,7 @@ from setuptools import find_namespace_packages, setup
 setup(
     name="org-pkg2",
     version="0.0.1",
-    packages=find_namespace_packages(include=["org.*"], where="src"),
+    packages=find_namespace_packages(include=["org*"], where="src"),
     package_dir={"": "src"},
     package_data={"org.pkg2": ["py.typed"]},
     python_requires=">=3.9",

(or just removing include, or just removing packages= entirely and relying on auto-discovery, or using include=["org", "org.*"]).

[ofek]

For example other backend using the editables might also rely on import hooks. I would also not classify it as a bug.

Hatch does not by default for this reason https://hatch.pypa.io/latest/config/build/#dev-mode