Choosing a prerelease if no stable versions are available to satisfy a requirement

[bersbersbers]

What's the problem this feature will solve?
This may be an issue or not - I have looked through PEP440 and haven't found anything. Basically, when installing package>1.0a0, I'd expect 1.0rc1 to be installed if it's the only prerelease version available. This is not the case.

Describe the solution you'd like

I'd like pip to understand that rc > a. Unless I am mistaken, we have all of these

2.0a0>1.0a0
1.1a0>1.0a0
1.0a1>1.0a0

but we do not have

1.0rc1>1.0a0

Some console output:

# see what's installed
> pip freeze | grep ^tensorflow=
tensorflow==2.3.1

# see what's available
> pip install tensorflow==10
ERROR: Could not find a version that satisfies the requirement tensorflow==10 (from versions: 2.2.0rc1, 2.2.0rc2, 2.2.0rc3, 2.2.0rc4, 2.2.0, 2.2.1, 2.3.0rc0, 2.3.0rc1, 2.3.0rc2, 2.3.0, 2.3.1, 2.4.0rc0, 2.4.0rc1)
ERROR: No matching distribution found for tensorflow==10

> pip install "tensorflow>2.4a0"
ERROR: Could not find a version that satisfies the requirement tensorflow>2.4a0 (from versions: 2.2.0rc1, 2.2.0rc2, 2.2.0rc3, 2.2.0rc4, 2.2.0, 2.2.1, 2.3.0rc0, 2.3.0rc1, 2.3.0rc2, 2.3.0, 2.3.1, 2.4.0rc0, 2.4.0rc1)
ERROR: No matching distribution found for tensorflow>2.4a0

> pip install "tensorflow>=2.4rc0"
Collecting tensorflow>=2.4rc0
...

> pip freeze | grep ^tensorflow=
tensorflow==2.4.0rc1

[pfmoore]

Just a quick note, your analysis isn't quite right, as 1.0rc1>1.0a0 is true:

>>> Version("1.0rc1") > Version("1.0a0")
True

Do you get the same result if you specify --pre to explicitly allow pre-releases?

Also, your console output appears inconsistent:

> pip install "tensorflow>=2.4rc0"
Collecting tensorflow>=2.4rc0
...

> pip freeze | grep ^tensorflow=
tensorflow==2.4.0rc1

You installed 2.4.0rc0, but you have 2.4.0rc1. Assuming you copy and pasted this from an actual session, there's something going on in the "..." that might be relevant here...

[bersbersbers]

Alright, thanks for checking the Versions.

I re-did the whole thing (on Windows this time), with complete output and --pre as well.

Regarding the apparent inconsistency, it says Collecting tensorflow>=2.4rc0 (note >=), and it is indeed downloading 2.4.0rc1 (see also below). This looks consistent to me.

It seems to be working with --pre, see below.

I had expected that tensorflow>2.4a0 would imply --pre (through the "explicitly requested by the user" alternative in PEP 440) - maybe this assumption is wrong. Still, >=2.4a0 should install 2.4.0rc0 in my opinion. Yes, it is not "the only available version that satisfies the version specifier", but closely related: all available versions that satisfy the version specifier are pre-releases. It feels to me this is how PEP 440 should be read.

By the way, I am on pip==20.2.4.

>pip freeze | grep "^tensorflow="
tensorflow==2.3.1

>pip install tensorflow==10
ERROR: Could not find a version that satisfies the requirement tensorflow==10 (from versions: 2.2.0rc1, 2.2.0rc2, 2.2.0rc3, 2.2.0rc4, 2.2.0, 2.2.1, 2.3.0rc0, 2.3.0rc1, 2.3.0rc2, 2.3.0, 2.3.1, 2.4.0rc0, 2.4.0rc1)
ERROR: No matching distribution found for tensorflow==10

>pip install "tensorflow>2.4a0"
ERROR: Could not find a version that satisfies the requirement tensorflow>2.4a0 (from versions: 2.2.0rc1, 2.2.0rc2, 2.2.0rc3, 2.2.0rc4, 2.2.0, 2.2.1, 2.3.0rc0, 2.3.0rc1, 2.3.0rc2, 2.3.0, 2.3.1, 2.4.0rc0, 2.4.0rc1)
ERROR: No matching distribution found for tensorflow>2.4a0

>pip install --pre "tensorflow>2.4a0"
Collecting tensorflow>2.4a0
  Downloading tensorflow-2.4.0rc1-cp38-cp38-win_amd64.whl (370.7 MB)
  [canceled]

>pip install --pre "tensorflow>=2.4rc0"
Collecting tensorflow>=2.4rc0
  Downloading tensorflow-2.4.0rc1-cp38-cp38-win_amd64.whl (370.7 MB)
  [canceled]

>pip install "tensorflow>=2.4rc0"
Collecting tensorflow>=2.4rc0
  Downloading tensorflow-2.4.0rc1-cp38-cp38-win_amd64.whl (370.7 MB)
  Downloading tensorflow-2.4.0rc1-cp38-cp38-win_amd64.whl (370.7 MB)
     |████████████████████████████████| 370.7 MB 1.9 kB/s
Collecting tensorflow-estimator<2.5.0,>=2.4.0rc0
  Downloading tensorflow_estimator-2.4.0rc0-py2.py3-none-any.whl (462 kB)
     |████████████████████████████████| 462 kB 6.4 MB/s
Requirement already satisfied: keras-preprocessing~=1.1.2 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from tensorflow>=2.4rc0) (1.1.2)
Requirement already satisfied: astunparse~=1.6.3 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from tensorflow>=2.4rc0) (1.6.3)
Requirement already satisfied: opt-einsum~=3.3.0 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from tensorflow>=2.4rc0) (3.3.0)
Requirement already satisfied: absl-py~=0.10 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from tensorflow>=2.4rc0) (0.11.0)
Requirement already satisfied: flatbuffers~=1.12.0 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from tensorflow>=2.4rc0) (1.12)
Requirement already satisfied: wrapt~=1.12.1 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from tensorflow>=2.4rc0) (1.12.1)
Collecting numpy~=1.19.2
  Downloading numpy-1.19.4-cp38-cp38-win_amd64.whl (13.0 MB)
     |████████████████████████████████| 13.0 MB 3.3 MB/s
Requirement already satisfied: gast==0.3.3 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from tensorflow>=2.4rc0) (0.3.3)
Requirement already satisfied: six~=1.15.0 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from tensorflow>=2.4rc0) (1.15.0)
Requirement already satisfied: tensorboard~=2.3 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from tensorflow>=2.4rc0) (2.3.0)
Requirement already satisfied: h5py~=2.10.0 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from tensorflow>=2.4rc0) (2.10.0)
Requirement already satisfied: typing-extensions~=3.7.4 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from tensorflow>=2.4rc0) (3.7.4.3)
Requirement already satisfied: termcolor~=1.1.0 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from tensorflow>=2.4rc0) (1.1.0)
Collecting grpcio~=1.32.0
  Using cached grpcio-1.32.0-cp38-cp38-win_amd64.whl (2.6 MB)
Requirement already satisfied: wheel~=0.35 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from tensorflow>=2.4rc0) (0.35.1)
Requirement already satisfied: protobuf~=3.13.0 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from tensorflow>=2.4rc0) (3.13.0)
Requirement already satisfied: google-pasta~=0.2 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from tensorflow>=2.4rc0) (0.2.0)
Requirement already satisfied: markdown>=2.6.8 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from tensorboard~=2.3->tensorflow>=2.4rc0) (3.3.3)
Requirement already satisfied: google-auth<2,>=1.6.3 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from tensorboard~=2.3->tensorflow>=2.4rc0) (1.23.0)
Requirement already satisfied: setuptools>=41.0.0 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from tensorboard~=2.3->tensorflow>=2.4rc0) (50.3.2)
Requirement already satisfied: werkzeug>=0.11.15 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from tensorboard~=2.3->tensorflow>=2.4rc0) (1.0.1)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from tensorboard~=2.3->tensorflow>=2.4rc0) (0.4.2)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from tensorboard~=2.3->tensorflow>=2.4rc0) (1.7.0)
Requirement already satisfied: requests<3,>=2.21.0 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from tensorboard~=2.3->tensorflow>=2.4rc0) (2.24.0)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from google-auth<2,>=1.6.3->tensorboard~=2.3->tensorflow>=2.4rc0) (4.1.1)
Requirement already satisfied: pyasn1-modules>=0.2.1 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from google-auth<2,>=1.6.3->tensorboard~=2.3->tensorflow>=2.4rc0) (0.2.8)
Requirement already satisfied: rsa<5,>=3.1.4; python_version >= "3.5" in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from google-auth<2,>=1.6.3->tensorboard~=2.3->tensorflow>=2.4rc0) (4.6)
Requirement already satisfied: requests-oauthlib>=0.7.0 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.3->tensorflow>=2.4rc0) (1.3.0)
Requirement already satisfied: idna<3,>=2.5 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from requests<3,>=2.21.0->tensorboard~=2.3->tensorflow>=2.4rc0) (2.10)
Requirement already satisfied: chardet<4,>=3.0.2 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from requests<3,>=2.21.0->tensorboard~=2.3->tensorflow>=2.4rc0) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from requests<3,>=2.21.0->tensorboard~=2.3->tensorflow>=2.4rc0) (1.25.11)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from requests<3,>=2.21.0->tensorboard~=2.3->tensorflow>=2.4rc0) (2020.6.20)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard~=2.3->tensorflow>=2.4rc0) (0.4.8)
Requirement already satisfied: oauthlib>=3.0.0 in c:\users\bers\appdata\local\programs\python\python38\lib\site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.3->tensorflow>=2.4rc0) (3.1.0)
Installing collected packages: tensorflow-estimator, numpy, grpcio, tensorflow
  Attempting uninstall: tensorflow-estimator
    Found existing installation: tensorflow-estimator 2.3.0
    Uninstalling tensorflow-estimator-2.3.0:
      Successfully uninstalled tensorflow-estimator-2.3.0
  Attempting uninstall: numpy
    Found existing installation: numpy 1.18.5
    Uninstalling numpy-1.18.5:
      Successfully uninstalled numpy-1.18.5
  Attempting uninstall: grpcio
    Found existing installation: grpcio 1.33.2
    Uninstalling grpcio-1.33.2:
      Successfully uninstalled grpcio-1.33.2
  Attempting uninstall: tensorflow
    Found existing installation: tensorflow 2.3.1
    Uninstalling tensorflow-2.3.1:
      Successfully uninstalled tensorflow-2.3.1
Successfully installed grpcio-1.32.0 numpy-1.19.4 tensorflow-2.4.0rc1 tensorflow-estimator-2.4.0rc0

> pip freeze | grep "^tensorflow="

[bersbersbers]

Extending my arguments above, I have noticed that pip install "pip>20.2.4" does not install 20.3b1, although "the only available version that satisfies the version specifier is a pre-release". Shouldn't it be installed then?

(I added the pip example because different from tensorflow, which has 2, pip only has a single pre-release version at the top as of now, and so that should be installed even from a literal reading of PEP 440.)

[uranusjr]

This is basically the reverse example of #7579 (there’s an example similar to this toward the end of the thread), pip’s implementation of PEP 440 logic is faulty.

[bersbersbers]

Another example (that I am not sure of if it's covered by the cases above - it may!):
pip install "tensorflow>2.4.0rc3" does not install 2.4.0rc4:

I think this is an easier case to think about, without the a vs. rc confusion that I introduced above.

[pradyunsg]

Can confirm, on pip 20.3:

$ pip install "tensorflow>2.4.0rc3"                                 
ERROR: Could not find a version that satisfies the requirement tensorflow>2.4.0rc3
ERROR: No matching distribution found for tensorflow>2.4.0rc3
$ pip install "tensorflow>2.4.0rc3" --use-deprecated=legacy-resolver
ERROR: Could not find a version that satisfies the requirement tensorflow>2.4.0rc3 (from versions: 2.2.0rc3, 2.2.0rc4, 2.2.0, 2.2.1, 2.3.0rc0, 2.3.0rc1, 2.3.0rc2, 2.3.0, 2.3.1, 2.4.0rc0, 2.4.0rc1, 2.4.0rc2, 2.4.0rc3, 2.4.0rc4)
ERROR: No matching distribution found for tensorflow>2.4.0rc3
$ pip install "tensorflow>2.4.0rc3" --pre                           
Collecting tensorflow>2.4.0rc3
  Downloading tensorflow-2.4.0rc4-cp38-cp38-macosx_10_11_x86_64.whl (175.5 MB)
     |█                               | 5.2 MB 5.3 MB/s eta 0:00:32^C
ERROR: Operation cancelled by user
$ pip install "tensorflow>2.4.0rc3" --use-deprecated=legacy-resolver --pre
Collecting tensorflow>2.4.0rc3
  Downloading tensorflow-2.4.0rc4-cp38-cp38-macosx_10_11_x86_64.whl (175.5 MB)
     |█▊                              | 9.6 MB 4.4 MB/s eta 0:00:39^C
ERROR: Operation cancelled by user

[sv158]

Confirm with Python 3.8.5 + pip 20.1.1, which specifier check works well in v20.1.1 but work wrong in v20.3.3.

To reproduce the right case:

1. using Python 3.7+ and an older version of pip, like 20.1.1; Then create an empty venv and activate it
2. try pip install snooty~=0.8.2.dev0, and then it will abort when try to install watchdog~=0.10.3
3. because watchdog~=0.10.3 requires Python'>=2.7, <3.6', pip can not install it and abort the progress of installing snooty

To reproduce the wrong case:

1. using Python 3.7+ and a newer version of pip, like 20.3.3; Then create another empty venv and activate it
2. try pip install snooty~=0.8.2.dev0
3. because the specifier check works wrong, it will successfully install the dependency watchdog~=0.10.3(like version 0.10.5, which theoretically only support Python<=3.5 🤨), finally snooty was fully installed

See also: vscode-restructuredtext/vscode-restructuredtext#257 (comment)

[notatallshaw]

Carefully reading through this discussion again I'm a little unsure what specifically it's about as it goes over a lot of different topics

But, reading just the title, I think this will be fixed by pypa/packaging#872