Warning about unhelpful interaction between conda, pip and Python

[wwarriner]

Notes

Strange interaction between python, pip and conda we encountered on Cheaha.

Researcher wants to install conda environment from a YML file. Unable to do so due to an error (with package mypy_extensions) when installing pip dependencies. I am unable to reproduce no matter how closely I align our environments, module reset, conda clean --all. Same error every time.

Today I noticed that the error is from a python file in ~/.local/ rather than the expected ~/.conda/. Digging further, we uncover that it is possible for pip packages to be placed in ./local/lib/python#.#/site-packages even when using pip from within an Anaconda environment. It is not clear yet why this happens.

By default, the CPython interpreter uses ~/.local/lib/python#.#/site-packages as the first location to find imports, then other PYTHONPATH locations (like Anaconda). We had the researcher clear out those directories and the environment installed correctly.

What we don't know is the exact conditions that will cause a package to be placed in ~/.local instead of ~/.conda. From my own heavily-used environments, it appears the last version in ~/.local is python3.9, so I suspect at some point between 3.9 and 3.10, the issue was resolved. What we also don't know is which part of the ecosystem causes the issue.

• Is it pip being unaware of Anaconda?
• Is it python version specific?
• Is it conda not informing pip of what path to use?
• Is it related to order of package installation when using multiple conda install commands?

At any rate, it appears that recent versions of Anaconda, pip and Python don't reproduce this issue. I'll see if I can do a bit more digging to find the boundaries.

[wwarriner]

We did figure out what causes packages to be placed in ~/.local. pip will install to this location whenever it can't install to the desired location. On Cheaha, this means installing using system pip, i.e., with no Anaconda environment activated, or with the base environment activated, or with a custom environment and pip is not installed (i.e., using system pip).

[wwarriner]

Bottom line: make sure you have your Anaconda environment activated and have pip installed in it before using pip to install any packages.