Unhelpful errors with --only-binary :all: when binaries are not available

[zackw]

Description

If you try to install a package in --only-binary :all: mode, and dependency resolution fails because some of the dependencies are only available as sdists, you may get unhelpful error messages that make it look like the problem is a dependency conflict.

A good example right now is the pdr package. It has a transitive dependency on cytoolz, which is a compiled-code package. Currently there are no prebuilt wheels of cytoolz for the just-released Python 3.13 (pytoolz/cytoolz#209). Here is a transcript of what happens if you try to install pdr right now (technically yesterday) in a Python 3.13 virtualenv, using --only-binary :all:

$ python3.13 -m venv /tmp/demo-bad-errors-venv
$ . /tmp/demo-bad-errors-venv/bin/activate
(venv) $ python3 --version
Python 3.13.0
(venv) $ pip --version
pip 24.2 from /tmp/demo-bad-errors-venv/lib/python3.13/site-packages/pip (python 3.13)
(venv) $ pip install --only-binary :all: pdr
Collecting pdr
  Downloading pdr-1.2.3-py3-none-any.whl.metadata (23 kB)
Collecting dustgoggles (from pdr)
  Downloading dustgoggles-0.7.5-py3-none-any.whl.metadata (594 bytes)
Collecting more-itertools (from pdr)
  Downloading more_itertools-10.5.0-py3-none-any.whl.metadata (36 kB)
Collecting multidict (from pdr)
  Downloading multidict-6.1.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.0 kB)
Collecting numpy (from pdr)
  Downloading numpy-2.1.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)
Collecting pandas>=2.0.0 (from pdr)
  Downloading pandas-2.2.3-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (89 kB)
Collecting rms-vax (from pdr)
  Downloading rms_vax-1.0.4-py3-none-any.whl.metadata (6.7 kB)
Collecting python-dateutil>=2.8.2 (from pandas>=2.0.0->pdr)
  Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting pytz>=2020.1 (from pandas>=2.0.0->pdr)
  Downloading pytz-2024.2-py2.py3-none-any.whl.metadata (22 kB)
Collecting tzdata>=2022.7 (from pandas>=2.0.0->pdr)
  Downloading tzdata-2024.2-py2.py3-none-any.whl.metadata (1.4 kB)
INFO: pip is looking at multiple versions of dustgoggles to determine which version is compatible with other requirements. This could take a while.
Collecting dustgoggles (from pdr)
  Downloading dustgoggles-0.7.4-py3-none-any.whl.metadata (594 bytes)
  Downloading dustgoggles-0.7.3-py3-none-any.whl.metadata (594 bytes)
  Downloading dustgoggles-0.7.2-py3-none-any.whl.metadata (564 bytes)
  Downloading dustgoggles-0.7.1-py3-none-any.whl.metadata (564 bytes)
  Downloading dustgoggles-0.7.0-py3-none-any.whl.metadata (564 bytes)
  Downloading dustgoggles-0.6.0-py3-none-any.whl.metadata (564 bytes)
  Downloading dustgoggles-0.5.0-py3-none-any.whl.metadata (496 bytes)
INFO: pip is still looking at multiple versions of dustgoggles to determine which version is compatible with other requirements. This could take a while.
  Downloading dustgoggles-0.4.2-py3-none-any.whl.metadata (496 bytes)
  Downloading dustgoggles-0.4.1-py3-none-any.whl.metadata (496 bytes)
  Downloading dustgoggles-0.4.0-py3-none-any.whl.metadata (496 bytes)
  Downloading dustgoggles-0.3.1-py3-none-any.whl.metadata (557 bytes)
  Downloading dustgoggles-0.3.0-py3-none-any.whl.metadata (418 bytes)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
ERROR: Cannot install pdr because these package versions have conflicting dependencies.

The conflict is caused by:
    dustgoggles 0.7.5 depends on cytoolz
    dustgoggles 0.7.4 depends on cytoolz
    dustgoggles 0.7.3 depends on cytoolz
    dustgoggles 0.7.2 depends on cytoolz
    dustgoggles 0.7.1 depends on cytoolz
    dustgoggles 0.7.0 depends on cytoolz
    dustgoggles 0.6.0 depends on cytoolz
    dustgoggles 0.5.0 depends on cytoolz
    dustgoggles 0.4.2 depends on cytoolz
    dustgoggles 0.4.1 depends on cytoolz
    dustgoggles 0.4.0 depends on cytoolz
    dustgoggles 0.3.1 depends on cytoolz
    dustgoggles 0.3.0 depends on cytoolz

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
ERROR: Cannot install pdr because these package versions have conflicting dependencies.

If you don't use --only-binary :all:, it downloads cytoolz-1.0.0.tar.gz and builds it from source and everything is fine. (Or possibly the installation fails because you don't have a C compiler available, but that's not a problem with pip, it's just why you were using --only-binary in the first place 😉 .)

Expected behavior

The error message in this case should say something like

ERROR: Cannot install pdr because cytoolz cannot be installed.
INFO: pdr depends on dustgoggles depends on cytoolz
INFO: No binary packages of cytoolz are available for any compatible ABI tag.
INFO: You have instructed pip not to install packages from source.
INFO: Try removing `--only-binary :all:` from the command line.

pip version

24.2

Python version

3.13.0

OS

linux, macos

How to Reproduce

1. Set up an environment based on Python 3.13.
2. Attempt to install any PyPI package that has a transitive dependency on a compiled-code package for which Python 3.13 wheels are not yet available, using --only-binary :all:
3. The error messages will make it look like the problem is a dependency conflict, not a lack of wheels.

Output

see above

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[notatallshaw]

I agree the messaging could be improved.

But I'm not convinced it is sensible to detect that --only-binary :all: has been used in the command, there are several flags you can pass to pip to restrict what it will resolve on, the assumption is the user chose those flags for a reason. It would probably be convoluted to determine if those flags are different from default, and if they had some bearing on this particular conflict.

But I would generally be supportive of improving conflict messages and I would happily assist in a PR that did that.

If you don't use --only-binary :all:, it downloads cytoolz-1.0.0.tar.gz and builds it from source and everything is fine. (Or possibly the installation fails because you don't have a C compiler available, but that's not a problem with pip, it's just why you were using --only-binary in the first place 😉 .)

FYI, the flag you might be looking for is --prefer-binary, which will select an older wheel instead over a newer sdist, but still selects an sdist if that's the only option for you.

[zackw]

It's understandable that it would be hard to detect whether the issue is due to command line flags. The "try not using --only-binary" part of the suggested improved error message isn't nearly as important as the "cytoolz cannot be installed because no compatible binary packages are available" part. And the part where it doesn't confuse the issue by spending a lot of vertical screen space talking about the intermediate dustgoggles package. dustgoggles did nothing wrong here :)

I cannot promise to develop a PR for this myself. I know nothing about the guts of pip.

the flag you might be looking for is --prefer-binary

Thanks for the tip; the situation that prompted this report is complicated enough that it might not work for me but I'll give it a try.

[notatallshaw]

The "try not using --only-binary" part of the suggested improved error message isn't nearly as important as the "cytoolz cannot be installed because no compatible binary packages are available" part

Yeah, I think I have an existing issue (although I can't find it right now) where I suggest improving it to something like this:

...
   dustgoggles 0.7.5 depends on cytoolz (no versions available)
...

But I've never had time to go and make this PR.