Use the base system executable as the default Python interpreter if running in venv

[TomiBelan]

If pipx is running inside a venv, e.g. sys.executable is "/path/to/some_venv/bin/python3", it should use the original ancestor Python (e.g. "/usr/bin/python3") as its default interpreter for creating venvs, not the some_venv python it's running in.

How would this feature be useful?

pipx usually does not run in a venv, but it could be running in a venv in these cases:

• The user activated a venv of some unrelated project they're working on, and ran python3 pipx.pyz ...
• The user activated a venv of some unrelated project they're working on, and ran pipx ... if the user moved pipx.pyz to ~/.local/bin/pipx and made it executable (thanks to its shebang #!/usr/bin/env python3)
• The user installed pipx itself into a venv, and symlinked ~/my_pipx_venv/bin/pipx to ~/.local/bin/pipx (although, the pipx documentation does not recommend to install it like this)

If pipx determines the default interpreter to be "/path/to/some_venv/bin/python3", it can have unintended consequences. The venvs created by pipx (in shared/, venvs/, .cache/) may accidentally become linked to some_venv, and will stop working if some_venv is deleted or significantly changed.

Normal installation with python3 -m pip install --user pipx does not have this problem because it writes a fixed shebang in ~/.local/bin/pipx, pointing to the python3 used at installation time. System distro package manager installation is also OK. I think this is a significant disadvantage of pipx.pyz compared to pip install --user.

The details are platform and Python version dependent. In Python 3.10 on Linux, if you run /usr/bin/python3 -m venv venv1 ; venv1/bin/python3 -m venv venv2, then venv2 will depend on venv1. I.e. venv2/bin/python3 will be a symlink pointing to venv1/..., and venv2/pyvenv.cfg will have home = /path/to/venv1/bin.

There was an undocumented change between Python 3.10 and 3.11. In Python 3.11 on Linux, if you run the same thing, venv2 will have home = /usr/bin, but it'll still have some traces of venv1: command = /.../venv1/bin/python3 -m venv .... As far as I know, nothing reads that field. So users of pipx.pyz on Python 3.11+ are (probably) (mostly) (kind of) fine.

Describe the solution you'd like

I don't know how to find the base executable...

This SO question mentions sys.base_prefix, but pipx doesn't need the ancestor directory, it needs the python3 binary.

I thought os.readlink(sys.executable) or os.path.realpath(sys.executable) could work, but then I remembered venvs can also use copies, not just symlinks.

I thought sys._base_executable must be it (although it is undocumented). Or perhaps getattr(sys, "_base_executable", None) or sys.executable. But on Python 3.10 Linux, sys._base_executable == sys.executable even in a venv.

Side note: sys._base_executable was previously discussed as a potential fix for #638 with brew on macOS.

And there are so many special ways to install Python. On macOS I've seen mentions of something called a "framework build". On Windows there is embedded Python, and the Windows Store build. The __PYVENV_LAUNCHER__ variable might be related somehow.

It might be possible to use some clever combination of these techniques. I was planning to just send a PR but then I realized it's much more complex than I thought. I hope at least I managed to explain the problem.

[gaborbernat]

Duplicate of #982

[TomiBelan]

I don't think this is a duplicate of #982. I knew about #982 before I filed this issue (I even commented in it earlier).

Summarizing:
#982 asks "what should we document as the official recommended installation method for this user segment?"
(users on recent Linux distros who want the newest pipx or who don't have root access)
#1074 is "here is a specific scenario how pipx misbehaves when installed like this and used like that."
(you activate a throwaway project venv, you run "python3 pipx.pyz ..." not realizing it's using that venv, you delete your venv, the tool you installed breaks)

Updating the documentation does not automatically fix this problem.

But, as I mentioned, in Python >=3.11 it became mostly harmless. Python 3.10 is still popular, but over time the problem will go away on its own. I don't mind too much if you close this bug, I'm just making sure you understand it.