[BUG] recent versions of setuptools broke editable mode

[dvarrazzo]

setuptools version

65.2

Python version

3.9.13

OS

Ubuntu

Additional environment information

No response

Description

Since some recent releases (likely 64, when PEP-660 was implemented), installing psycopg with -e results in a broken package, where psycopg is an empty namespace.

This is an example of broken run. This is a similar run whose difference is installing psycopg without the -e flag, in order to test a depending package.

Expected behavior

See above

How to Reproduce

A more complete reproduction with stable git reference can be found in tox-dev/tox#2479.

Output

See above

[abravalheri]

Thank you very much @dvarrazzo for reporting this issue.

Could you please try to create a minimal/standalone reproducer (that does not involve using tox or pytest)?

I tried to do the following:

sudo apt update -y
sudo apt install -y libpq5 libpq-dev
cd /tmp && rm -rf /tmp/psycopg /tmp/empty_dir
git clone https://github.com/psycopg/psycopg.git /tmp/psycopg
cd /tmp/psycopg
python3.8 -m venv .venv
.venv/bin/python -m pip install -U pip
.venv/bin/python -m pip install -e ./psycopg
.venv/bin/python -m pip install -e ./psycopg_c
mkdir /tmp/empty_dir
cd /tmp/empty_dir  # <-- To avoid any problems with automatic injection of CWD into `sys.path`
cat <<EOS | /tmp/psycopg/.venv/bin/python -
import psycopg, psycopg_c
print(psycopg.Cursor)
print(psycopg.AsyncCursor)
print(dir(psycopg_c))
EOS
# Output ==>
# <class 'psycopg.Cursor'>
# <class 'psycopg.AsyncCursor'>
# ['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_psycopg', 'pq', 'sys', 'version']

And I can see that the module members mentioned in the tox issue seem to exist.

Is there any chance this happens because Python automatically adds the current working dir to sys.path? (This way, the import system could mistake "/tmp/psycopg/psycopg" as the root of the pyscopg package). I tried to run the following example to investigate this theory:

# Continuation of the previous script
# still inside /tmp/psycopg
mkdir psycopg
echo "raise SystemError('kaboom!')" > psycopg/__init__.py
/tmp/psycopg/.venv/bin/python -c 'import psycopg'
# Output ==>
# Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#  File "/tmp/empty_dir/psycopg/__init__.py", line 1, in <module>
#    raise SystemError('kaboom!')
# SystemError: kaboom!

[dvarrazzo]

I thought that, and in my investigation with tox I tried to chdir away, but it didn't fix the problem.

Anyway, does it mean that it is now mandatory to adopt a "src" style project organization?

[abravalheri]

Anyway, does it mean that it is now mandatory to adopt a "src" style project organization?

I don't think so...

In terms of "plain Python" usage", I would say things should not change a lot. The flat-layout should have the same flaws as before, right?
Specifically, if you have a folder under CWD with a name coinciding with the installed package, this folder might be imported (but that was already a well-known risk ...).

In terms of pytest things may change. I know that they try some magic to compensate the flat-layout flaws... I have not received any report about that yet (maybe this error is the first?).

---

The limitations of the implementation are documented here. So far the biggest impact is for old-style namespace packages using pkgutil and pkg_resources, but some workarounds are suggested in the docs (e.g., the strict mode should work).

[abravalheri]

I thought that, and in my investigation with tox I tried to chdir away, but it didn't fix the problem.

So do you confirm that the package itself works on an editable install, but things start to get weird when a 3rd tool is added to the mix? (Maybe tox, maybe pytest?)

Could it be the case one of these tools relies on a specific assumption about the editable installation? (I know that some static analysis tools will try to read a .egg-link file, which is gone in setuptools>=64...)

[dvarrazzo]

I can reproduce the problem with:

git clone git@github.com:psycopg/psycopg.git psycopg-test
cd psycopg-test
git checkout a7d39bf740d0d97dbe230d07177463668327431a

python3 -m venv .venv
source .venv/bin/activate
pip install "pip==22.2.2" "setuptools==65.2.0" "wheel==0.37.1"
pip install -e ./psycopg
python -c "import psycopg; psycopg.ProgrammingError" # fails
python -c "import psycopg; print(psycopg)"
# <module 'psycopg' (namespace)>

changing cwd works. Using pip install psycopg works too:

pip install ./psycopg
python -c "import psycopg; print(psycopg)"
# <module 'psycopg' from '/home/piro/dev/psycopg-test/.venv/lib/python3.8/site-packages/psycopg/__init__.py'>

Using a rather older set of libraries, editable mode works as expected:

pip install -U "setuptools==44.0.0" "pip==20.0.2" "wheel==0.37.1"
pip install -e ./psycopg
python -c "import psycopg; psycopg.ProgrammingError" # works
python -c "import psycopg; print(psycopg)"
# <module 'psycopg' from '/home/piro/dev/psycopg-test/psycopg/psycopg/__init__.py'>

Tested on my laptop using Python 3.8.10 (Ubuntu 20.04 package)

[abravalheri]

changing cwd works. Using pip install psycopg works too:

Thank you very much @dvarrazzo. So it is confirmed that it is happening because of the existence of the package CWD...

I can see that a regular installation does not have the same problem. If I had to guess I would say that Python starts by creating a namespace package, but then when it finds the other folder (the actual package) on sys.path and realises it contains a __init__.py file, it "regrets" the decision of having a namespace package and "collapses" it into a regular module.

This behaviour does not seem to be very well defined/coherent though...
In theory a folder in CWD should get priority during the import. We can do a series of experiments showing that if we don't mix namespace packages and traditional packages (the ones with __init__.py). I need to seek for some advice regarding this behaviour.

For the time being let's consider these "accidental namespace packages" as a limitation of the editable mode.

[dvarrazzo]

Ok, but what does it mean? Is this a regression that you will address (why would a random directory in cwd be more meaningful that an explicit entry in the pythonpath?) Or is this going to stay broken?

Should we pin a dependency on setuptools < 64 for development? Should we rename our code directory and make all git history operations more painful in order to restore editable mode?

Please let us know.