PEP 668 support breaks --user/--editable

[stefanor]

Description

@obfusk filed Debian bug #1030335 about the implementation of PEP 668 support.

I think it's best if this is discussed here, with upstream involvement.
cc @geofft @doko42 @FFY00 @dstufft @uranusjr @ehashman @pradyunsg @kitterma who were involved in the PEP or pip in Debian.

Hi!

I just updated python3-pip and was greeted with a NEWS message about
PEP 668 support:

This version of pip introduces PEP 668 support. Debian's python3.11
interpreter will soon (>= 3.11.1-3) declare the installation to be
EXTERNALLY-MANAGED, instructing pip to disallow package installation outside
virtualenvs.

See: https://peps.python.org/pep-0668/

Practically, this means that you can't use pip to install packages outside a
virtualenv, on a Debian system, any more.

See /usr/share/doc/python3.11/README.venv for more details.
If that isn't available yet, check:
https://salsa.debian.org/cpython-team/python3/-/blob/master/debian/README.venv

Not being able to install packages system-wide seems like a good idea,
I fully support that.

But I often install the Python packages I'm developing under my own
user account with "pip install -e", which is also made impossible by
this change.

And I intentionally do not use virtualenvs for this because I want to
have all dependencies provided by Debian packages, not downloaded from
PyPI.

And as far as I can tell there is not even an option I can give to pip
to tell it to allow me to install to ~/.local anyway. PEP 668
mentions a --break-system-packages (as an example), but such an option
doesn't seem to actually exist.

• FC

This bug is about a Debian-user-specific issue (not wanting to depend on PyPI), but I don't think this is something Debian should try to solve, alone.

When the PEP was drafted, we were assuming there'd be some sort of mechanism to override it (allow the user to break their system and keep both pieces), I quote:

The installer should have a way for the user to override these rules, such as a command-line flag --break-system-packages. This option should not be enabled by default and should carry some connotation that its use is risky.

That didn't make it into the implementation, so I suggested deleting the EXTERNALLY-MANAGED file in the documentation Debian is providing about this. It's not great, but it's about the only suggestion I can make:

https://salsa.debian.org/cpython-team/python3/-/blob/master/debian/README.venv

Does pip have any thoughts on providing an override mechanism?

There is some value in not having an override (we don't need to keep testing that all these schemes work, for one). But as we transition, we're going to be breaking use-cases...

Expected behavior

No response

pip version

23.0

Python version

3.11.1

OS

Debian GNU/Linux

How to Reproduce

1. apt install python3-pip
2. check out a python source package.
3. pip install -e .

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[obfusk]

FWIW, I can use e.g. python3 -mvenv --system-site-packages --without-pip ~/.venv as a workaround (and even automatically activate that venv in ~/.bashrc).

But that's more work, and not compatible with what's already installed in ~/.local.

I don't see how that is any better than continuing to allow --user.

I'm still using the system site packages like before, so I don't think it "breaks the system" any less than --user does.

[obfusk]

I suggested deleting the EXTERNALLY-MANAGED file in the documentation Debian is providing about this. It's not great, but it's about the only suggestion I can make:

Which is only an option for admins, not regular users on e.g. a shared university server.

[uranusjr]

Disallowing --user is intentional because user-site package installations affect the system Python the same way system-site ones do (except they only affect the single user). Using a virtual environment with --system-site-packages is a much better solution.

[obfusk]

Disallowing --user is intentional

According to the PEP, so is having an override mechanism. Which there isn't.
So the current unconditional breakage of --user is not in accordance with the PEP.

Using a virtual environment with --system-site-packages is a much better solution.

Why?
I can understand requiring some kind of "I know what I'm doing" option (as the PEP mentions), after all --system-site-packages is also opt-in.
But why is is better to be forced to use a virtual environment when I always want that to be active?

[uranusjr]

Since the override mechanism is backed by the PEP, contribution is always welcomed.

[actualben]

If you're breaking PEP 370 shouldn't its status be marked as Superseded as part of this change?
Sorry I've just realized that this isn't quite the right place for this question.

[Omarabdul3ziz]

Thank you for the quick report and fix. but is the fix published yet? I tried to use the flag --break-system-packages but got error no such option

I am working on alpine:edge container

[uranusjr]

@actualben Quick answer is no. Marking PEP 370 as Superceded would be mean the mechanism no longer works, but what PEP 668 specifies is a way to disable user-site by default. Note that per-user site-packages can already be disabled via various ways, the new PEP only adds one additional way and encourages distros to set it by default.

[pradyunsg]

Thank you for the quick report and fix. but is the fix published yet?

No, it is not.