New resolver: Failure when package only specified with extras is not available from index

[wchill]

What did you want to do?

pip install --use-feature=2020-resolver -r requirements.txt --no-deps
pip install --use-feature=2020-resolver -r requirements.txt

requirements.txt only contains a list of packages to be installed in editable mode, with some depending on each other.

This is in a fresh miniconda environment and occurred on both 20.2.2 and master.

I ran pip using --no-deps first since in my experience, installing multiple editable mode packages with dependencies on each other fails otherwise. However, running just the normal install command directly in a fresh environment still fails with the new resolver, as below.

ERROR: Could not find a version that satisfies the requirement azureml-dataset-runtime[fuse]~=0.1.0.0 (from azureml-defaults)
ERROR: No matching distribution found for azureml-dataset-runtime[fuse]~=0.1.0.0

Output

This output is after running the 2nd pip install command, to actually install the package dependencies after installing the editable mode packages using --no-deps. Output has been slightly edited to remove full file paths.

ERROR: Cannot install azureml-dataset-runtime 0.1.0.0 (from src\azureml-dataset-runtime), -r requirements.txt (line 9), -r requirements.txt (line 16) and azureml-dataset-runtime[fuse] 0.1.0.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested azureml-dataset-runtime 0.1.0.0 (from src\azureml-dataset-runtime)
    azureml-automl-core 0.1.0.0 depends on azureml-dataset-runtime~=0.1.0.0
    azureml-train-automl-client 0.1.0.0 depends on azureml-dataset-runtime~=0.1.0.0
    azureml-dataset-runtime[fuse] 0.1.0.0 depends on azureml-dataset-runtime 0.1.0.0 (Installed)

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

Additional information

The requirements file looks like this (all below packages should be available from pypi as well):

-e "src\azureml-core\."
-e "src\azureml-dataset-runtime\."
-e "src\azureml-defaults\."
-e "src\azureml-telemetry\."
-e "src\azureml-opendatasets\."
-e "src\azureml-pipeline\."
-e "src\azureml-pipeline-core\."
-e "src\azureml-pipeline-steps\."
-e "src\azureml-automl-core\."
-e "src\azureml-automl-runtime\."
-e "src\azureml-interpret\."
-e "src\azureml-explain-model\."
-e "src\azureml-train-restclients-hyperdrive\."
-e "src\azureml-train-core\."
-e "src\azureml-train\."
-e "src\azureml-train-automl-client\."
-e "src\azureml-train-automl-runtime\."

[uranusjr]

Can you provide either setup.py for all the listed projects (or at least azureml-dataset-runtime and azureml-train-automl-client), or a reduced setup that can reproduce the same error? It is impossible to tell what is going on without any context.

[wchill]

Sure. The below is a trimmed down version of setup.py for azureml-dataset-runtime.

azureml-train-automl-client takes a dependency on this via this entry for install_requires:
'{}~={}'.format('azureml-dataset-runtime', VERSION)

# ---------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# ---------------------------------------------------------

from setuptools import setup, find_packages
import os
import shutil

SELFVERSION = '0.1.0.0'
DATAPREP_VERSION = '>=2.0.1a,<2.1.0a'

REQUIRES = [
    'azureml-dataprep{}'.format(DATAPREP_VERSION),
    'pyarrow>=0.17.0,<2.0.0'
]

with open('README.md', 'r', encoding='utf-8') as f:
    long_description = f.read()
with open('../.inlinelicense', 'r', encoding='utf-8') as f:
    inline_license = f.read()

setup(
    name="azureml-dataset-runtime",
    version=SELFVERSION,
    description='',
    long_description=long_description,
    long_description_content_type='text/markdown',
    author='Microsoft Corp',
    license=inline_license,
    url='https://docs.microsoft.com/python/api/overview/azure/ml/?view=azure-ml-py',
    packages=find_packages(exclude=["*tests*"]),
    install_requires=REQUIRES,
    extras_require={
        'pandas': ['numpy>=1.14.0,<2.0.0', 'pandas>=0.23.4,<2.0.0'],
        'parquet': [],  # Keeping as no-op to avoid breaking scenarios where this extra is used.
        'pyspark': ['pyspark==2.3.0'],
        'fuse': ['fusepy>=3.0.1,<4.0.0'],
        'scipy': ['scipy>=1.1.0,<2.0.0']
    }
)

[uranusjr]

I was able to reproduce the error with the following:

# a/setup.py
from setuptools import setup
setup(name="a", version="0.1.0.0", extras_require={"z": ["c"]})

# b/setup.py
from setuptools import setup
setup(name="b", version="0.1.0.0", install_requires=["a[z]~=0.1.0.0"])

# c/setup.py
from setuptools import setup
setup(name="c", version="0.1.0.0")

$ pip install --use-feature=2020-resolver --no-deps ./a ./b
(snip, this works)

$ pip install --use-feature=2020-resolver ./a ./b
Processing file:///.../a
Processing file:///.../b
Requirement already satisfied: a[z]~=0.1.0.0 in .../a (from b==0.1.0.0) (0.1.0.0)
ERROR: Cannot install a 0.1.0.0 (from .../a) and a[z] 0.1.0.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested a 0.1.0.0 (from .../a)
    a[z] 0.1.0.0 depends on a 0.1.0.0 (Installed)

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

---

A few observations. First, editable-ness does not matter.

Second, the extra indirection is needed to trigger this. pip install ./a ./b would work if b depends on a directly (instead of a[z]).

Third, the error if you don’t install --no-deps first is

ERROR: Could not find a version that satisfies the requirement a[z]~=0.1.0.0 (from b)
ERROR: No matching distribution found for a[z]~=0.1.0.0

Again, the extra indirection is needed to trigger the error. pip install ./a ./b works if b depends on a directly (instead of a[z]).

So the problem is that we’re doing something wrong when handling dependencies to an extra-ed requirement.

[uranusjr]

OK, I think I know what’s going on here. When a package requests dependencies, the resolver pass all the currently known specifiers for the provider to find candidates with. If b requests a (no extras), the resolver would pass ['a', 'a @ ./a'] since it knows that there is an additional specifier provided by the user.

When b requests a[z], however, the resolver does not know that a @ ./a also provides a[z], and thus only passes ['a[z]'] to the provider, resulting it failing to find a match.

The provider API will need to be tweaked slightly for it to be able to “tell” the resolver that a actually provides candidates for both a and a[z]. I will update resolvelib and come back to this afterwards.

---

The ResolutionImpossible exception triggered when a was first installed before hand is triggered by the same cause. But instead of not being able to find any candidates, the provider now is able to find exactly one match—the previously-installed distribution—but then rejects it since the installed distribution does not satisfy the direct link requirement. How this should be fixed (or not fixed) would depend on our resolution to #5780. Factory.find_candidates() will need to be updated to relect the decision and make an AlreadyInstalledCandidate be able to satisfy a SpecifierRequirement if the URL matches.

[pradyunsg]

I will update resolvelib and come back to this afterwards.

@uranusjr Any updates on this? I am not 100% sure how this would work and whether we have to make changes for this issue, prior to the new-resolver-is-default release.

[uranusjr]

I have a couple of plans to make this work (not yet decided which is best; the plans I came up with thus far are all a bit ugly). It’s not the end of the world if the release is made without fixing this, but we should try to get a fix in if possible.