FileNotFoundError for pipx_shared.pth with `pipx install --python` and pyenv

[bhrutledge]

Describe the bug

I have pipx installed from Homebrew (yay!), as well as pyenv. When I run pipx install black, the resulting venv using the python3.7 interpreter from Homebrew. When I run pipx install --python python3.8 black (or any other version, with any other package), I get the error mentioned in the issue title and reproduced below.

How to reproduce

$ pipx install --verbose --python python3.8 black
pipx > (_package_name_from_spec:93): Determined package name: black
pipx > (_package_name_from_spec:94): Package name determined in 0.0s
pipx > (run_subprocess:108): running python3.8 -m venv --without-pip /Users/brian/.local/pipx/venvs/black

pipx > (rmdir:18): removing directory /Users/brian/.local/pipx/venvs/black
Traceback (most recent call last):
  File "/usr/local/Cellar/pipx/0.15.1.1/libexec/bin/pipx", line 11, in <module>
    load_entry_point('pipx==0.15.1.1', 'console_scripts', 'pipx')()
  File "/usr/local/Cellar/pipx/0.15.1.1/libexec/lib/python3.7/site-packages/pipx/main.py", line 590, in cli
    return run_pipx_command(parsed_pipx_args)
  File "/usr/local/Cellar/pipx/0.15.1.1/libexec/lib/python3.7/site-packages/pipx/main.py", line 172, in run_pipx_command
    include_dependencies=args.include_deps,
  File "/usr/local/Cellar/pipx/0.15.1.1/libexec/lib/python3.7/site-packages/pipx/commands/commands.py", line 62, in install
    venv.create_venv(venv_args, pip_args)
  File "/usr/local/Cellar/pipx/0.15.1.1/libexec/lib/python3.7/site-packages/pipx/venv.py", line 139, in create_venv
    pipx_pth.write_text(f"{shared_libs.site_packages}\n", encoding="utf-8")
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py", line 1235, in write_text
    with self.open(mode='w', encoding=encoding, errors=errors) as f:
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py", line 1203, in open
    opener=self._opener)
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py", line 1058, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/brian/.local/pipx/venvs/black/lib/python3.7/site-packages/pipx_shared.pth'

$ pipx --version
0.15.1.1

$ pyenv version
3.8.0 (set by /Users/brian/.pyenv/version)
2.7.15 (set by /Users/brian/.pyenv/version)
3.7.4 (set by /Users/brian/.pyenv/version)
3.6.9 (set by /Users/brian/.pyenv/version)

$ pyenv which python3.8
/Users/brian/.pyenv/versions/3.8.0/bin/python3.8

Expected behavior

pipx should install a package using the python3.8 interpreter from pyenv.

I get the same error using an absolute path with --python $(pyenv which python3.8), but --python $(brew --prefix python@3.8)/bin/python3 works.

[itsayellow]

Looks like a bad interaction with pyenv and pipx_shared.pth . This reminds me a little of #294, in that it seems like a problem involving both pyenv and pipx_shared.pth.

My guess is that this has something to do with detecting which python we're using, and that is getting messed up because of the use of pyenv. It's interesting to note that even though --python python3.8 is used, it is looking for pipx_shared.pth in path /Users/brian/.local/pipx/venvs/black/lib/python3.7/site-packages/pipx_shared.pth (note the 3.7.)

[itsayellow]

The first problem that happens when using --python python3.8 in install and pyenv, is that for some reason the venv is getting installed with the system python, and ending up (in my case) with python3.7.

I can successfully

• use python3.8 via pyenv from the command-line to create a python3.8 venv
• run python3.8 -V from a subprocess in python3.7.6 and get back the right answer ("python3.8.1")

What never works (even in the pipx code)

• ❌ building a python3.8 venv from python3.7.6

Something very weird is going on. Observe me trying to use subprocess in python3.7.6 to make a venv with python3.8:

carvel:pipx python3
Python 3.7.6 (default, Dec 30 2019, 19:38:26) 
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.run(["python3.8", "-V"])
Python 3.8.1
CompletedProcess(args=['python3.8', '-V'], returncode=0)
>>> subprocess.run(["python3.8", "-m", "venv", "venv"])
CompletedProcess(args=['python3.8', '-m', 'venv', 'venv'], returncode=0)
>>> subprocess.run(["ls", "venv/bin"])
Activate.ps1		easy_install		pip3.7
activate		easy_install-3.7	python
activate.csh		pip			python3
activate.fish		pip3
CompletedProcess(args=['ls', 'venv/bin'], returncode=0)
>>> 

On macOS, there's some kind of voodoo with the environment variable __PYVENV_LAUNCHER__. This variable is set and used for Framework builds of python. The Framework binary (as I understand the explanation) is a stub, and it starts a subprocess and communicates which version it is through this (temporary?) environment variable. I notice on my system (macOS) it is set to /usr/local/Cellar/python/3.7.6_1/bin/python3.7 during pipx's run_subprocess() function before the invocation of subprocess.run(). This variable has caused people trouble before:

• Scripts installed with wrong shebang under subprocess virtualenv#845
• https://stackoverflow.com/questions/26323852/whats-the-meaning-of-pyvenv-launcher-environment-variable
• https://bugs.python.org/issue22490#msg227579
• bpo-22490: Remove __PYVENV_LAUNCHER__ from environment during launch python/cpython#9516

I'm wondering if running pipx from a macOS Framework python build, and using a subprocess to start a pyenv build of python (probably NOT a Framework build) is confusing which python gets built in the venv. This could also be why @bhrutledge is able to succeed when he uses the 3.8 homebrew build, as the 3.8 homebrew build WILL be a Framework build.

[itsayellow]

Well I made progress. If I remove __PYVENV_LAUNCHER__ from env in run_subprocess() (like we do for PYTHONPATH) then I can successfully build a venv really based on python3.8

Afterwards it cannot find pip, but still, progress.

(Not that I exactly know what I'm doing by removing __PYVENV_LAUNCHER__)

[itsayellow]

Hm. Now the problem is that it thinks that it should install the pipx_shared.pth file to my system pyenv python3.8.1 directory, and not to the venv directory.

Which is why it is not finding pip.

[itsayellow]

Ah it turns out that util.get_site_packages() was using get_script_output(), another function that uses subprocess.run(). When I deleted __PYVENV_LAUNCHER__ from the env of this subprocess.run() command as well, everything worked.

So I can fix this bug, if every time we use a subprocess.run() we delete __PYVENV_LAUNCHER__ from its environment.

I have no idea what side effects that might cause.

[bhrutledge]

So I can fix this bug, if every time we use a subprocess.run() we delete __PYVENV_LAUNCHER__ from its environment. I have no idea what side effects that might cause.

Thanks for for the investigation, @itsayellow. I'm glad it's not just me. :)

This reminded me of a chat I had at PyCon with a current maintainer of virtualenv. He said one of the reasons for its continued existence is that it facilitates creating virtual environments using different

Might there be any other workarounds?

If not, instead of deleting __PYVENV_LAUNCHER__ and PYTHONPATH from a copied environment, would it make any sense to start with a blank environment, and explicitly set the variables needed by pipx?

[bhrutledge]

Possibly relevant? https://virtualenv.pypa.io/en/latest/reference/#compatibility-with-the-stdlib-venv-module

Has there been any discussion of pipx using virtualenv instead of venv?

[itsayellow]

👍 thanks for the link, that's a lot of useful information. (Some I'm still trying to parse in my head!)

Has there been any discussion of pipx using virtualenv instead of venv?

Not really, I think we were trying to stay with the built-in venv because pipx is a python3.6+ tool, and so far venv has served well.

If not, instead of deleting PYVENV_LAUNCHER and PYTHONPATH from a copied environment, would it make any sense to start with a blank environment, and explicitly set the variables needed by pipx?

It's a fair question. Originally I believe that all env variables were implicitly used. Later, we specifically removed PYTHONPATH because homebrew was manipulating it for its own uses that conflicted with pipx. So the subject of environment variables might need some conscious thought.

As far as __PYVENV_LAUNCHER__, I believe we can safely remove it, because it was never really meant to be a user variable, it was only meant to be created by the main python binary to allow it to communicate to a subprocess. Since we are starting a fresh python as if from a shell, we shouldn't have this set beforehand.

[bhrutledge]

I just reinstalled using brew install --HEAD pipx, and confirmed that the fix resolves my issue. Thanks for the prompt fix!

[itsayellow]

Glad it works for you! Thanks for your help on this issue.