When --python option is placed after the subcommand name in pip install, it gets ignored

[richardxia]

Description

Apologies if this is a duplicate issue, but I cannot for the life of me figure out how to use either GitHub or Google's search to search for the string "--python", since they either ignore the -- or treat it as a negative search, and the string "python" is obviously not specific enough to identify just the CLI option.

From reading the documentation on the --python option, it sounds like it exactly matches my use case of wanting to use pip to install a package into a different virtual environment. However, the commands in the example code block on that page don't work for me when I try to use a pip that was itself installed into a different virtual environment, since instead of installing to the virtual environment pointed to by the --python option, it installs to the virtual environment that pip itself is installed to.

I can't use the pip that is bundled with my python installation, since that version of pip is too old (22.2.1) to have the --python option. In my environment, we are running RHEL 8.7, but we are using a version of Python 3.10.6 that was compiled from source, and we cannot upgrade the version of pip that we installed at the time for reproducibility reasons. For all intents and purposes, you can treat this Python and pip as part of the "system", even if they were not packaged by Red Hat.

My use case is that I would like the ability to create minimal virtual environments that don't have pip installed and still be able to tell a separate installation of pip to install packages into that virtual environment. Since my "system" pip is too old to have the --python option, I would like to use a pip that was installed in one virtual environment to install packages into a different virtual environment.

Expected behavior

I expected that the packages would be installed to the virtual environment pointed to by the --python option, not the virtual environment that pip itself is installed to. I would also at least expect a warning message if it is just going to ignore the --python CLI option.

pip version

23.1.2

Python version

3.10.6

OS

RHEL 8.7

How to Reproduce

# First create a virtual environment to just hold the specific version of pip
$ python3 -m venv venv-pip
$ ./venv-pip/bin/pip install pip==23.1.2
# Confirm that the virtual environment pip was correctly updated
$ ./venv-pip/bin/pip --version
# Then create a new virtual environment without pip
$ python3 -m venv venv-app --without-pip
$ ./venv-pip/bin/pip install --python ./venv-app/bin/python3 black
# venv-app doesn't have black installed
$ ls venv-app/bin/
# But venv-pip _does_ have black installed
$ ls venv-pip/bin/

Output

# First create a virtual environment to just hold the specific version of pip
$ python3 -m venv venv-pip
$ ./venv-pip/bin/pip install pip==23.1.2
Collecting pip==23.1.2
  Using cached pip-23.1.2-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.2.1
    Uninstalling pip-22.2.1:
      Successfully uninstalled pip-22.2.1
Successfully installed pip-23.1.2
# Confirm that the virtual environment pip was correctly updated
$ ./venv-pip/bin/pip --version
pip 23.1.2 from /home/rxia/tmp/venv-pip/lib/python3.10/site-packages/pip (python 3.10)

# Then create a new virtual environment without pip
$ python3 -m venv venv-app --without-pip
$ ./venv-pip/bin/pip install --python ./venv-app/bin/python3 black
Collecting black
  Using cached black-23.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)
Collecting click>=8.0.0 (from black)
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting mypy-extensions>=0.4.3 (from black)
  Using cached mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)
Collecting packaging>=22.0 (from black)
  Using cached packaging-23.1-py3-none-any.whl (48 kB)
Collecting pathspec>=0.9.0 (from black)
  Using cached pathspec-0.11.1-py3-none-any.whl (29 kB)
Collecting platformdirs>=2 (from black)
  Using cached platformdirs-3.5.1-py3-none-any.whl (15 kB)
Collecting tomli>=1.1.0 (from black)
  Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
Installing collected packages: tomli, platformdirs, pathspec, packaging, mypy-extensions, click, black
Successfully installed black-23.3.0 click-8.1.3 mypy-extensions-1.0.0 packaging-23.1 pathspec-0.11.1 platformdirs-3.5.1 tomli-2.0.1

# venv-app doesn't have black installed
$ ls venv-app/bin/
activate  activate.csh  activate.fish  Activate.ps1  python  python3  python3.10

# But venv-pip _does_ have black installed
$ ls venv-pip/bin/
activate  activate.csh  activate.fish  Activate.ps1  black  blackd  python  python3  python3.10

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[richardxia]

Sorry, user error. I realized that the --python option has to come before the word install.

But I think it's still misleading for pip to silently ignore the --python option when it appears after the word install but not before. Should I leave this issue open to address that issue, or should I close this and file a new issue that more correctly describes the bug?

[pfmoore]

This sounds like a bug. I would expect the global --python option to be picked up anywhere on the command line (that's how other global options like --verbose work). But pip's option parsing is complex, and it looks like the --python option isn't getting recognised in that position. The problem is that we have to process --python before we identify the subcommand, but at that point we don't parse any arguments after the subcommand name.

I don't know how best to fix this. One option is to simply document the behaviour (and if you have any suggestions on how we could do that in a way that would have helped you understand what was going on here, that would be great). Or we could raise an error if --python is found in the subcommand options¹. But I'm not sure we could recognise the --python argument in the subcommand options without a significant restructuring of our parsing code.

I've retitled this issue to reflect the actual problem here.

Footnotes

1. A brief outline of how we might do this, based on my initial investigations. I'm adding this here mainly so I don't forget it - I don't have time to make a PR now, but I don't want to lose the information. The --python option is recognised by subcommand parsers because it's in the general_options listing, so we need to remove it from there. Then, in order to have it recognised by the main parser, it needs to be added in as a special case in pip._internal.cli.main_parser.create_main_parser, just after the code marked "add the general options". I think that should do it, we'd need tests and probably still some documentation of this special case. I'm also not clear how this would get displayed in the help - that's something that would need to be checked. ↩

[pfmoore]

The outlined solution doesn't work because the initial parse stops (by design) when it encounters the subcommand name. So I think the best we can do is warn (or error?) when the --python option is specified after the subcommand name. I've created #12068 for this - it still needs tests and documentation.

[richardxia]

Thanks for the quick reply! I agree that if it's difficult to fix, then at least adding a warning is a reasonable short-term step to take, since silently ignoring is definitely not good.

One option is to simply document the behaviour (and if you have any suggestions on how we could do that in a way that would have helped you understand what was going on here, that would be great).

Yes, I think it would also be reasonable to add some text to the documentation to describe the limitation in addition to printing a warning. Here is my attempt at wording it:

Note: Due to how pip currently processes command-line options, the --python option must appear before the subcommand name such as install. The --python option will be ignored if it appears after the subcommand name.

Feel free to take, adapt, or ignore that suggestion as you like.