Providing pip configuration in `sys.base_prefix`

[pelson]

pip version

main

Python version

all

OS

all

Additional information

When a pip config is installed in an installation prefix $PREFIX/pip.conf, $PREFIX/bin/python -m pip correctly picks up the config. When one makes a virtual environment with $PREFIX/bin/python -m venv ./my-venv then ./my-venv/bin/pip does not pick up the config.

It is questionable if this is a bug or a feature request, but essentially, I believe that pip should be looking in sys.base_prefix as well as sys.prefix for a config file.

Description

No response

Expected behavior

No response

How to Reproduce

Starting with a non-virtual environment (e.g. a conda environment):

$ touch ./env/pip.conf
$ pip config debug
env_var:
env:
global:
  /etc/xdg/xdg-ubuntu/pip/pip.conf, exists: False
  /etc/xdg/pip/pip.conf, exists: False
  /etc/pip.conf, exists: False
site:
  /media/important/github/pypa/pip/env/pip.conf, exists: True
user:
  /home/pelson/.pip/pip.conf, exists: False
  /home/pelson/.config/pip/pip.conf, exists: False

$ python -m venv ./venv

## BEWARE THAT YOUR VENV HAS THE BUNDLED PIP, SO INSTALL A NEWER PIP FOR DEBUGGING
$ ./venv/bin/pip install -e /path/to/pip/repo

$ ./venv/bin/pip config debug
env_var:
env:
global:
  /etc/xdg/xdg-ubuntu/pip/pip.conf, exists: False
  /etc/xdg/pip/pip.conf, exists: False
  /etc/pip.conf, exists: False
site:
  /media/important/github/pypa/pip/venv/pip.conf, exists: False
user:
  /home/pelson/.pip/pip.conf, exists: False
  /home/pelson/.config/pip/pip.conf, exists: False

What I want to see for a venv is that the base_prefix is searched as well as the prefix.

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct

[pradyunsg]

Hi! Thanks for filing this issue and PR!

Could you please elaborate on why you want this? What use case does this help with?

[uranusjr]

I think this is by design? A virtual environment is supposed to isolated things from the global environment, so configuration from sys.base_prefix should not be picked up.

[pelson]

Could you please elaborate on why you want this? What use case does this help with?

Sure, I'll try with my specific case, and then hopefully we can draw parallels elsewhere.

I an the responsible for a Python distribution to a controlled network. There are other Python distributions on the controlled network (both accessible from the same machine) for which I am not responsible, and which need specific pip configurations. I too have a specific pip configuration (e.g. index-url), I therefore configure them in the pip.conf of the distribution. It is designed that users (developers) take virtual environments from this distribution in order to extend it as needed - pip is a fundamental part of that work flow to allow them to extend as required. I therefore want those users to have the appropriate pip configuration within their virtual environments (out of the box).

To re-iterate, it is not acceptable to have the config be global (e.g. in /etc/pip.conf) nor user enabled (e.g. in ~/pip.conf) as they need to also be able to use another Python distribution with different pip configurations (i.e. the pip config needs to be isolated to the environment).

Environment variables defining the config aren't acceptable because the two Python distributions wouldn't be able to inter-operate (you can only have one PIP_INDEX_URL.

To me the idea that a virtual environment should inherit the pip configuration from the base environment is natural. After all, a virtual environment inherits the Python and its standard library from the base environment - if you change the config of how Python is built such config propagates/leaks to the virtual environment.

[pfmoore]

IMO, the only sort of environment that should inherit is a --use-system-site-packages one, as that's explicitly not isolated from the system environment. Certainly changing the default behaviour to inherit would be a significant backward incompatibility, and would likely cause problems for people who rely on global settings not leaking into a virtualenv (e.g., testing).

[pelson]

Thank you for your input so far.

would likely cause problems for people who rely on global settings not leaking into a virtualenv (e.g., testing)

We should try to be more precise with the terminology here. global settings do leak into a virtualenv:

$ cat /etc/pip.conf 
[global]
no-cache-dir = true

$ cat ./venv/pip.conf 
[global]
no-dependencies = yes

$ pip config list
global.no-cache-dir='true'
global.no-dependencies='yes'

What I'm suggesting is, just like Python config (e.g. compile flags) leaks into a virtual environment, so too should the pip config. i.e. site configuration should leak into a virtual env.

To turn this discussion around a little, perhaps I could ask you how you would control a virtual environment's pip configuration if global, user, and environment settings are not acceptable (because (a) a pip config doesn't apply to all Python installations on a machine, (b) it isn't a user configuration to define how a Python installation should interact with the distribution)?

I'm trying to not be hypothetical here, so my concrete scenario:

• 1 machine
• No internet, but internal repositories exist
• 2 completely different Python distributions
• Each python distribution has different pip configuration requirements (e.g. index urls)
• Users use virtual environments to give them extensibility

The problem occurs fairly quickly as users take a virtual environment from one of the distributions and they immediately have no pip configuration. Even worse, they try to use a PEP517 tool such as build to create a wheel python -m build ./ using the correctly (pip) configured Python distribution, but because the build isolation creates a new virtual environment the pip isn't configured and you get an error.

[piotr-dobrogost]

@pelson
I have to admit the scenario you brought up is interesting. :)

To turn this discussion around a little, perhaps I could ask you how you would control a virtual environment's pip configuration if global, user, and environment settings are not acceptable (because (a) a pip config doesn't apply to all Python installations on a machine, (b) it isn't a user configuration to define how a Python installation should interact with the distribution)?

After taking a quick look at docs of venv it looks like you could extend EnvBuilder class and define post_setup() method.

From https://docs.python.org/3/library/venv.html#venv.EnvBuilder.post_setup

post_setup(context)
A placeholder method which can be overridden in third party implementations to pre-install packages in the virtual environment or perform other post-creation steps. (emphasis mine)

The post-creation step in your case could be creation of appropriate pip configuration file in the newly created virtual environment.

[gaborbernat]

After taking a quick look at docs of venv it looks like you could extend EnvBuilder class and define post_setup() method.

That's my recommendation for using build too, to extend virtualenv via its plugins- pypa/build#270 (comment).

[pelson]

Thanks for the suggestion of extending EnvBuilder. Indeed this does work to automatically configure a venv with pip.conf, and I've been running with such a tweak for quite some time. Unfortunately the python -m venv and python -m build invocations aren't something that you can override in this way (without monkeypatching during Python startup 🙊).

Perhaps I could have been a bit clearer here: I'm not looking for a workaround per se (I have one, though I do appreciate ideas of other potential workaround solutions), I'm looking to address what I believe is a genuine shortfall (i.e. unexpected behaviour) in the way pip behaves with virtual environments.

That's my recommendation for using build too, to extend virtualenv via its plugins

I'm not using virtualenv, I'm using venv, the standard library way to make virtual environments. So too does build (https://github.com/pypa/build/blob/0.3.1/src/build/env.py#L188), no? There is no equivalent plugin system for venv.

the only sort of environment that should inherit is a --use-system-site-packages one

I disagree: I am not proposing that you inherit a single extra package with this change, merely that a locally (site) scoped pip.conf from a parent environment should be inherited in the child (virtual) one. I think considering configuration inheritance as the same as package inheritance (system site packages) would be a conflation of the --use-system-site-packages flag.

I'm trying to figure out real-world use cases where you wouldn't want to inherit the pip.conf from the parent environment. The one provided by @pfmoore "isolation testing" is a reasonable one, but I don't believe it should necessarily trump the one in which a user wants to create a virtual environment on a Python distribution which has been correctly pip configured and expect that their virtual env pip to "just work™".

In my PR (#9753) I implemented the easier approach of blending together the pip.conf in sys.base_prefix and sys.prefix (just like all of the other config search path items). It would totally solve my use case if, instead of doing a blend, it simply didn't look in sys.base_prefix if sys.prefix / pip.conf existed. This would solve the "isolated testing" usecase, as you would simply touch sys.prefix / pip.conf to avoid sys.base_prefix / pip.conf being considered. This may be best of both worlds - you can easily enough have isolated testing, but still get working behaviour out of the box.