Can we define per-dependency sources? Some PyTorch and Transformers dependency resolving issues

[Tremeschin]

Update: This issue is known and being tracked on uv/171 and partially explained on uv/PIP_COMPATIBILITY.md

Would probably still need integration in pyproject.toml; take my report below as a unwanted behavior causing a "deadlock" of dependencies, in how uv decides and always prioritizes custom indexes

Old comment

Hi, I'll try to give enough context and reasoning, didn't find similar issues after some searching, bit of complex problem

~

I'm porting all my infra to a Rye Workspaces setup instead of nested Poetry venvs, as that is being the single most reason of user error, SSD abuse, overengineered micro management code and development frustration when changing dependencies

I'm installing PyTorch CPU 'flavor', some projects use it, as that is more platform agnostic and the smallest:

[[tool.rye.sources]]
name = "torch"
url  = "https://download.pytorch.org/whl/cpu"

To my understanding adding this source with implied type="index" makes packages to take precedence for the custom source over PyPI, globally

I've probably stumbled on two issues due this, the main one being that when installing torch and transformers together, dependency resolving for huggingface-hub fails due fsspec package:

❯ rye add transformers
Added transformers>=4.39.1 as regular dependency
Reusing already existing virtualenv
Generating production lockfile: /home/tremeschin/BrokenSource/requirements.lock
  × No solution found when resolving dependencies:
  ╰─▶ Because any of:
          huggingface-hub>=0.19.3,<=0.19.4
          huggingface-hub>=0.20.0,<=0.20.3
          huggingface-hub>=0.21.0,<=0.21.4
          huggingface-hub>=0.22.0
      depends on fsspec>=2023.5.0 and only the following versions of huggingface-hub are available:
          huggingface-hub<=0.19.4
          huggingface-hub>=0.20.0,<=0.20.3
          huggingface-hub>=0.21.0,<=0.21.4
          huggingface-hub>=0.22.0
      we can conclude that huggingface-hub>=0.19.3 depends on fsspec>=2023.5.0.
      And because transformers==4.39.1 depends on huggingface-hub>=0.19.3 and only fsspec<2023.5.0 is available, we can
      conclude that transformers==4.39.1 cannot be used.
      And because only transformers<=4.39.1 is available and you require transformers>=4.39.1, we can conclude that the
      requirements are unsatisfiable.

Previously, I had similar issues with dependency resolving for Pillow and moderngl-window, even got in touch with the developers there to maybe relax pillow requirements, and I was getting weird issues like Pillow==10.0.1 doesn't exist, even when clearly all the wheels are on PyPI and Python 3.11 is not an exception

So, after noticing similar stuff with transformers, I went to investigate a bit

At first, when inspecting the METADATA file of the lastest huggingface_hub-0.22.1-py3-none-any.whl, it points to Requires-Dist: fsspec >=2023.5.0, and there's clearly no version of that on PyPI currently, I thought it was their error on dependency specification and was preparing to write a bug report, but I wasn't ultimately convinced because how come Poetry, pip-tools resolve the dependencies right and uv fail it, it's NP hard but not NP impossible 😅

I was about to write reproduction steps and I rye init, rye add transformers and rye add torch, to my happy accident, I forgot to add the sources to the pyproject.toml, and everything installed correctly, so I'm writing here now

It looks like uv is prioritizing https://download.pytorch.org/whl/fsspec and https://download.pytorch.org/whl/pillow over PyPI's ones, and PyTorch's wheels are clearly either slightly outdated or missing older versions like Pillow==10.0.1

~

Long story short; is there a way to only define the PyTorch CPU source for the torch package itself?

With Poetry we simply do-

[tool.poetry.dependencies]
torch = {version="2.2.1", source="torch-cpu"}
torchvision = {version="^0.17.1", source="torch-cpu"}

-on a dependency basis to specify the source, not globally

I can just slap regular torch for now but that's quite a GBs of data for just running any project. This is kind of a breaking feature I need, and I really want to use rye, it solves so many issues with packaging and monorepo in general

Thanks !

[dawgster]

I am facing the exact same issue. Once this is improved it will be possible to address a very important part of the ML ecosystem