Description
Environment
- pip version: 20.1.1
- Python version: 3.8.4
- OS: Mac OS Catalina 10.15.5 (Homebrew Python)
Description
Packages installed using pip into a venv created with --system-site-packages and with a Homebrew base interpreter have shebangs written in their scripts that point to the system Python, causing these scripts to crash with ModuleNotFoundError.
To clarify, this might not be a bug in pip - I am opening this issue to get feedback on my workaround and learn about where this issue should actually be opened, as well as to provide information for anyone who encounters this in the future (at first glance this seemed to me to be a bug in pip). If this is indeed not a bug in pip and belongs elsewhere, this issue can be closed and I can re-open it where it "belongs".
Expected behavior
When installing a package into a venv, I expect pip to write the shebang of the scripts included by that package so that they point to the venv's interpreter, not the system Python.
MInimal reproducible example
% python3 -m venv --system-site-packages venv
% source venv/bin/activate
(venv)% pip install tox
(venv)% tox --version
Traceback (most recent call last):
File ".../venv/bin/tox", line 5, in <module>
from tox import cmdline
ModuleNotFoundError: No module named 'tox'
I use tox as an example package here, but I have seen this same behavior with every other Python package I have tested.
My understanding/workaround
This happens because the shebang in venv/bin/tox incorrectly points to the system Python, which in turn seems to be because sys.executable is set to the system Python, even though python is executed from inside the venv:
(venv)% python
Python 3.8.4 (default, Jul 14 2020, 02:58:48)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'/usr/local/opt/python@3.8/bin/python3.8'
Passing the absolute path to the venv's Python interpreter does not change this.
This is not the case on Linux (python:3.7 Docker image)
$ python3 -m venv --system-site-packages venv
$ source venv/bin/activate
(venv)$ python
Python 3.7.8 (default, Jun 30 2020, 18:27:23)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'/venv/bin/python'
or if --system-site-packages is turned off
% python3 -m venv venv-nossp
% source venv-nossp/bin/activate
(venv-nossp)% python
Python 3.8.4 (default, Jul 14 2020, 02:58:48)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'.../venv-nossp/bin/python'
This github comment by @chbrown (thank you!) pointed me in the direction of /usr/local/lib/python3.8/site-packages/sitecustomize.py, a file generated by Homebrew that appears to be responsible for setting sys.executable to point to /usr/local/opt/python@3.8/bin/python3.8 if PYTHONEXECUTABLE is not set.
If my venv's all include "venv" in their folder name, I can write something like
for p in sys.path:
if 'venv' in p:
sys.executable = p.split('lib')[0] + 'bin/python'at the end of this file, which re-overwrites sys.executable to the venv's interpreter whenever sys.path appears to contain a venv. (There is probably a better way to identify the correct value of sys.executable when running Python from inside a venv.)
After making this change, everything works as expected. However, this feels like a quite fragile/"wrong" way to address the problem.
There are many related github issues (especially pypa/virtualenv#845) that lead me to think that a change in how __PYVENV_LAUNCHER__ is interpreted (or not interpreted) by pip's vendored distlib here is somehow related to this, but in the end I am not sure I understand the problem well enough to say that this is an issue in pip, virtualenv, or Homebrew, or if there's just a "better" way for me to apply my workaround, or if I just have something configured incorrectly on my end (I am quite new to Mac OS).