Option to look for pip.conf in conda environment

[jakirkham]

Related to issue ( conda-forge/conda-forge.github.io#518 ).

Noted that there was some special handling with virtualenv to figure out the pip.conf file to use related to that environment. Was wondering if the same thing might be possible for a conda environment. Perhaps something like $CONDA_PREFIX/pip/pip.conf or something similar for both Unix and Windows (guessing with .ini as the extension). Though others may have thoughts on where this should live and how it should be named.

cc @kalefranz @msarahan @pradyunsg

[jakirkham]

Related point that we may want to consider is using a different configuration file for when conda-build is running vs. not.

[jakirkham]

Should add the conda package building considerations are independent from the active conda environment considerations.

[kalefranz]

I'm probably fine with it being at os.path.join(sys.prefix, 'pip.conf'), just like with virtualenv.

FYI, maybe not relevant here, but the canonical way to tell if a directory is a conda environment (and the way conda itself uses) is os.path.isfile(os.path.join(sys.prefix, 'conda-meta', 'history')).

[jakirkham]

Thoughts @pfmoore? Would this be something pip would be interested in and/or accept a patch for?

[pfmoore]

@jakirkham I assume you're referring to the running_under_virtualenv function in pip? But I'd appreciate a more explicit pointer to the code that you mean.

Having sys.base_prefix differ from sys.prefix is defined by Python as the way to expose the fact that you're in a virtual environment (see here). The use of sys.real_prefix is historically how virtualenv did this, and we handle that because prior to core Python support for virtual environments, virtualenv was the only solution around. IMO virtualenv should now switch to the standard base_prefix approach, and in the longer term, pip could drop checking for real_prefix.

I'd suggest that conda environments follow the core Python standard and set base_prefix, and then there would be no need for pip to special-case them, and other tools should also work without any special considerations.

[jakirkham]

Honestly don't really know the pip codebase well enough to propose where it should go. Was reading the docs, which sparked the question. :)

Interesting. Not sure to what extent using base_prefix is possible, but agree that sharing an API is a good idea if at all workable. This might work in some cases (e.g. base_prefix points at the python used by conda and prefix points to the python in the current environment if there is one). Not sure what to do when we are talking about the environment conda lives in (i.e. root later base) where both of these are the same though 😕 (added to the list below).

Can think of a few tricky cases though. There may be more. Not totally sure the implications off the top of my head, but we'd have to be sure we understand what base_prefix and prefix mean in these cases.

1. Activating virtualenv installed into a conda environment.
2. Environment stacking. ( support stacking environments conda/conda#5159 )
3. Using build/host in conda-build.
4. Using pip.conf/pip.ini in the root/base environment of conda.

[pfmoore]

Honestly don't really know the pip codebase well enough to propose where it should go. Was reading the docs, which sparked the question. :)

Cool, I thought you were referring to what's in the code. I assume you were looking in the docs here. Pip doesn't look at $VIRTUAL_ENV, but what it does is basically equivalent - the virtual environment config file is identified by pip as:

        # finally virtualenv configuration first trumping others
        if running_under_virtualenv():
            yield kinds.VENV, [venv_config_file]

venv_config_file is simply {sys.prefix}/pip.conf, and running_under_virtualenv simply tests if sys.prefix differs from sys.base_prefix (or if sys.real_prefix is set).

[jakirkham]

That sounds right.

Ah ok thanks for the info.

What does pip do on Python 2 to detect a virtualenv?

So if we just drop a config file in sys.prefix, should that be enough for pip? Is there a way we can verify what config file it is using? Does being in a virtualenv actually affect this?

[pfmoore]

What does pip do on Python 2 to detect a virtualenv?

sys.real_prefix. As I say, virtualenv sets that so pip will use that. But for conda, it's better to set sys.base_prefix as that's more forward compatible (you can set sys.base_prefix from user code in Python, both 2 and 3).

So if we just drop a config file in sys.prefix, should that be enough for pip? Is there a way we can verify what config file it is using? Does being in a virtualenv actually affect this?

From the code I quoted, such a config file will only be used if you're in a virtualenv (i.e. sys.base_prefix is set and is not equal to sys.prefix). You can't verify "what config file pip is using", because it uses all of them - there's a hierarchy of files and config settings are taken from the highest-priority place the specific setting you're looked at is defined in. From the docs:

If multiple configuration files are found by pip then they are combined in the following order:

1. Firstly the site-wide file is read, then
2. The per-user file is read, and finally
3. The virtualenv-specific file is read.

Each file read overrides any values read from previous files, so if the global timeout is specified in both the site-wide file and the per-user file then the latter value is the one that will be used.

[jakirkham]

So I think it is important to point out that conda and virtualenv are coming at this from two different angles. Sorry if what follows is already obvious, want to establish a baseline before asking more questions.

AFAIK virtualenv allows one to create a separate environment for Python packages that links in an external Python (with batteries included) for use in this otherwise clean environment. Please correct me if I'm getting any of that wrong.

OTOH conda allows users to install Conda packages into an environment. Those may be (and a lot of them are) Python packages as well, but they are more similar to Python packages from a system package manager (e.g. apt-get). However a user can actually install a Conda package with a different Python interpreter altogether. In fact I have several environments with different Python interpreters and this is a common use case. It can also install dependencies of Python (e.g. OpenSSL) or things completely unrelated (e.g. Boost).

As a result of this, it's quite easy to have a version of Python installed in the base environment in conda that does not match the one in my active environment. Honestly one could be CPython and the other PyPy. So one of the things I'm trying to wrap my head around is what happens when base_prefix and prefix are not somehow related to the same Python. Namely what if base_prefix and prefix point to totally different, possibly incompatible Pythons? Related what happens when we mess with things like base_prefix under the hood give site.py seems to try and configure it (amongst other things)?

On a different point, the base environment in conda is technically just another environment. There's some special things that go into other environments to make sure that the conda CLI can be run when other environments are active (@kalefranz feel free to correct me if I'm missing anything here). Otherwise the base environment is no different than the other ones. The definitions of base_prefix and prefix really breakdown here as they should be the same. However we wouldn't want base to be treated as a somehow special environment from the other conda environments as it would break things for users not expecting this difference. How can we solve this issue?

Looking at the search priority might help to highlight the conceptual differences. Conda users typically think of each environment as being its own siloed /usr (or /usr/local if you would rather). The Anaconda and conda-forge stacks are even built up that way. For example, we provide our own certificates, which our copies of OpenSSL, curl, git, etc. are taught to look for. Users are free to build their own certificate package and replace ours, but it would still be expected to inhabit the same location relative to the prefix environment. So different environments could even have different certificates. Digression aside, we'd ideally want pip.conf/pip.ini in a conda environment to be treated as a site-wide file. Is that something we could achieve with the virtualenv approach or could we try another approach?

[dstufft]

If I understand conda correctly, I don't think that the requested feature is really specific to conda (even though the original ask suggests that it is). I think it might be better worded as adding a config file that is specific to the actual Python install. On a Windows machine with the Python.org installers, this might be located at C:\Python36\pip.conf, on a Linux machine it might be /usr/lib/python3.6/pip.conf, in a Conda environment it'd be relative to that conda environment.

[jakirkham]

Thanks @dstufft. Agree that Python installers (batteries included) seems like an overlapping use case with similar end goals ( even if I wasn't thinking of that initially ;).