Unexpected --user behavior since version 23.1

[flowmeadow]

Description

Hey everyone,
I'm using pip typically in virtual environments (managed by PyCharm). Since version 23.1 I get an error message when installing packages.

(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip --version
pip 23.1 from c:\users\florian.wiese\pycharmprojects\guimanager\venv\lib\site-packages\pip (python 3.9)

(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip install numpy
ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

A little research told me, that a --user flag within a venv does not really make sense. However, it seems that some kind of default --user is set since 23.1.

A workaround I found is to add the --no-user flag.

(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip install numpy --no-user
Collecting numpy
   Using cached numpy-1.24.2-cp39-cp39-win_amd64.whl (14.9 MB)
Installing collected packages: numpy
Successfully installed numpy-1.24.2

For completeness, downgrading to 23.0.1 with the --no-user flag gives the me the behavior I'm used to.

(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip --version
pip 23.1 from c:\users\florian.wiese\pycharmprojects\guimanager\venv\lib\site-packages\pip (python 3.9)

(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>python.exe -m pip install --upgrade pip==23.0.1 --no-user
Collecting pip==23.0.1
  Using cached pip-23.0.1-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 23.1
    Uninstalling pip-23.1:
      Successfully uninstalled pip-23.1
Successfully installed pip-23.0.1

(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip install numpy
Collecting numpy
  Using cached numpy-1.24.2-cp39-cp39-win_amd64.whl (14.9 MB)
Installing collected packages: numpy
Successfully installed numpy-1.24.2

[notice] A new release of pip is available: 23.0.1 -> 23.1
[notice] To update, run: python.exe -m pip install --upgrade pip

Not sure if this is a bug or a feature. Maybe I'm also missing something very obvious. But at least it is documented now for anyone experiencing the same issue :)

Expected behavior

No response

pip version

23.1

Python version

3.9

OS

Windows 10

How to Reproduce

create a venv with pip 23.1
try to install a package with pip install <any-package-name>

Output

ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

Pip 23.1 does not have a default to --user. However it's possible you are picking up a user config of some sort. Please can you provide the output of pip config debug in both the failing and the successful cases?

[flowmeadow]

Sure, thanks in advance.

Pip 23.0.1 (successful case):

(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip config debug
env_var:
env:
global:
  C:\ProgramData\pip\pip.ini, exists: False
site:
  C:\Users\florian.wiese\PycharmProjects\GUIManager\venv\pip.ini, exists: False
user:
  C:\Users\florian.wiese\pip\pip.ini, exists: False
  C:\Users\florian.wiese\AppData\Roaming\pip\pip.ini, exists: False

Pip 23.1 (Failing case):

(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip config debug
env_var:
base:
  C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\pip.ini, exists: True
    install.user: yes
    uninstall.user: yes
    freeze.user: yes
    list.user: yes
env:
global:
  C:\ProgramData\pip\pip.ini, exists: False
site:
  C:\Users\florian.wiese\PycharmProjects\GUIManager\venv\pip.ini, exists: False
user:
  C:\Users\florian.wiese\pip\pip.ini, exists: False
  C:\Users\florian.wiese\AppData\Roaming\pip\pip.ini, exists: False

[pfmoore]

So that's your issue. In the failing case, you have a base config that sets user to true. This looks like the Windows Store installation of Python. I'm not sure why they set a pip config (I think it's a workaround for a permission issue with Windows Store apps) but IMO setting user globally like this is a bad idea. But it's what we have to deal with, unfortunately.

This was caused by

In the case of virtual environments, configuration files are now also included from the base installation. (#9752)

That issue was extensively discussed, but it appears that the case of the Windows Store distribution wasn't known (or wasn't considered). IMO this is a sufficiently serious problem that it requires a bugfix. I'm not sure it's possible to simply revert the change (reverting a merge commit is a bit beyond my git skills!) so we'll need someone to work on a fix ASAP. @pelson as the author of the original change, are you able to look at this? Alternatively, if anyone can tell me the right git incantation to revert #11487, I can do that and we can revisit the feature for a future pip version.

Thanks for reporting this @flowmeadow!

[pfmoore]

As a quick proof of concept, I've created #11983 to patch out the BASE config. This might form the basis of a minimal fix if we can't find a better solution.

[pfmoore]

I also opened python/cpython#103646 to discuss with CPython whether there's a better way than a pre-installed pip.ini to achieve what they are trying to do with the Store installation.

[flowmeadow]

Thanks a lot for the analysis. I can confirm that its the Windows Store installation. I remember having a very bad feeling using it. Well, at least this was the first issue that arose in almost two years.

[pfmoore]

Generally, I think the Store install should be fine. There's some quirks like this, but they mostly seem to work. And having people use the distribution and report issues like you have is the only way we'll know what needs improving, so your help is appreciated.

[abhinavsingh]

jFYI, this is also impacting proxy.py workflows e.g. https://github.com/abhinavsingh/proxy.py/actions/runs/4717368971/jobs/8450593742

[pfmoore]

@abhinavsingh that's nothing to do with this issue. That's the deprecation of pkg_resources, which is #11975 (and which is now fixed in main, and awaiting a bugfix release which will come once we've established what fixes we need to include).