can pip-compile handle multiple extras of the same package properly

[zyxue]

Environment Versions

1. OS Type: Mac OS or Linux
2. Python version: 3.8.2
3. pip version: 21.3
4. pip-tools version: 6.4.1.dev1+geb9162f

Steps to replicate

1. put the following two top-level deps in requirements.in

cat requirements.in 
ray[default]==1.4.0
tune-sklearn==0.4.0

2. pip-compile requirements.in --rebuild
3. the output is wrong

My debugging leads to the following:

tune-sklearn depends on ray[tune] as specified in its setup.py, then pip-compile groups constraints into ray[default,tune], which leads to zero dependencies

jq .dependencies.ray ~/Library/Caches/pip-tools/depcache-cp3.8.json
{
  "1.4.0[default,tune]": [],
  "1.4.0[default]": [
    "aiohttp",
    "aiohttp-cors",
    "aioredis",
    "click>=7.0",
    "colorama",
    "colorful; extra == \"default\"",
    "filelock",
    "gpustat",
    "grpcio>=1.28.1",
    "jsonschema",
    "msgpack<2.0.0,>=1.0.0",
    "numpy>=1.16",
    "opencensus",
    "prometheus-client>=0.7.1",
    "protobuf>=3.15.3",
    "py-spy>=0.2.0",
    "pydantic>=1.8",
    "pyyaml",
    "redis>=3.5.0",
    "requests"
  ]
}

Expected result

"1.4.0[default,tune]" should have the deps of both default and tune.

Actual result

"1.4.0[default,tune]" has zero deps.

[atugushev]

Hello @zyxue ,

Thanks for the report. That's because pip-tools doesn't yet have a proper dependency resolver. See #693 and the other related issues for details.

[zyxue]

Are you sure that's really the cause? I thought pip-tools depends on pip to do dep resolution, no?

I find this change makes the resolution behaving more as expected, https://github.com/jazzband/pip-tools/pull/1512/files, any comment?

Or what's the currently suggested work around, please?

[atugushev]

pip-tools internally uses pip's legacy resolver which is not a complete resolver, see pypa/pip#988.

However, I was wrong and the issue was really in the requirements combiner. Sorry, my bad. I've checked your PR and it looks like it fixes the issue. Thanks for the nice catch and the concise solution!

[zyxue]

pip-tools internally uses pip's legacy resolver which is not a complete resolver, see pypa/pip#988.

Doesn't pip have a new solver now since pip-20.3 (https://pip.pypa.io/en/stable/user_guide/#changes-to-the-pip-dependency-resolver-in-20-3-2020), is pip-tools planning to use that instead?

Thanks, I'll try to add tests to the PR.

[atugushev]

#1539 adds support for the new resolver. Could you give it a try?

[zyxue]

tried, still the same, it doesn't fix the example I have at the top

[di]

This may be due to pypa/pip#9644.

[richafrank]

Hi @zyxue , would you check again on master, now that #1519 has merged and covers this (instead of #1512) ?

[zyxue]

@richafrank , tried

• rebase onto e2d0fae
• run pip-compile requirements.in --rebuild --annotation-style=line
• output:

jq .dependencies.ray ~/Library/Caches/pip-tools/depcache-cp3.8.json

{
  "1.4.0[default,tune]": [
    "aiohttp",
    "aiohttp-cors",
    "aioredis",
    "click>=7.0",
    "colorama",
    "colorful; extra == \"default\"",
    "filelock",
    "gpustat",
    "grpcio>=1.28.1",
    "jsonschema",
    "msgpack<2.0.0,>=1.0.0",
    "numpy>=1.16",
    "opencensus",
    "pandas; extra == \"tune\"",
    "prometheus-client>=0.7.1",
    "protobuf>=3.15.3",
    "py-spy>=0.2.0",
    "pydantic>=1.8",
    "pyyaml",
    "redis>=3.5.0",
    "requests",
    "tabulate; extra == \"tune\"",
    "tensorboardX; extra == \"tune\""
  ],
  "1.4.0[default]": [
    "aiohttp",
    "aiohttp-cors",
    "aioredis",
    "click>=7.0",
    "colorama",
    "colorful; extra == \"default\"",
    "filelock",
    "gpustat",
    "grpcio>=1.28.1",
    "jsonschema",
    "msgpack<2.0.0,>=1.0.0",
    "numpy>=1.16",
    "opencensus",
    "prometheus-client>=0.7.1",
    "protobuf>=3.15.3",
    "py-spy>=0.2.0",
    "pydantic>=1.8",
    "pyyaml",
    "redis>=3.5.0",
    "requests"
  ]
}

so it appears to be working now.

[richafrank]

Wonderful. I'll close this then.

Based on the above chatter, @atugushev feel free to reopen if you think there's something else to be done here.