Unnecessary hard failure of pip-compile when a required package is already present in output with an unavailable version

[AndydeCleyre]

This was encountered in #1529

When a needed package is already pinned in the output file,
but has an invalid or at least unavailable version there,
the compilation will fail.

Environment Versions

1. Linux
2. Python version: 3.9.5
3. pip version: 21.3.1
4. pip-tools version: 6.4.0

Steps to replicate

$ echo requests >reqs.in
$ pip-compile reqs.in
$ sed -i 's/==2021/==2041/' reqs.txt
$ pip-compile reqs.in

Expected result

I'd expect that the pre-existing content of the output file is considered "low priority" -- that if something there doesn't fit the needs of the input files, it will be discarded. In other words, it should ultimately behave in this case as if certifi isn't in the output file to begin with, and so generate a valid output file again.

Actual result

  ERROR: Could not find a version that satisfies the requirement certifi==2041.10.8 (from requests==2.26.0->-r reqs.in (line 1)) (from versions: 0.0.1, 0.0.2, 0.0.3, 0.0.4, 0.0.5, 0.0.6, 0.0.7, 0.0.8, 1.0.0, 1.0.1, 14.5.14, 2015.4.28, 2015.9.6, 2015.9.6.1, 2015.9.6.2, 2015.11.20, 2015.11.20.1, 2016.2.28, 2016.8.2, 2016.8.8, 2016.8.31, 2016.9.26, 2017.1.23, 2017.4.17, 2017.7.27, 2017.7.27.1, 2017.11.5, 2018.1.18, 2018.4.16, 2018.8.13, 2018.8.24, 2018.10.15, 2018.11.29, 2019.3.9, 2019.6.16, 2019.9.11, 2019.11.28, 2020.4.5, 2020.4.5.1, 2020.4.5.2, 2020.6.20, 2020.11.8, 2020.12.5, 2021.5.30, 2021.10.8)
Traceback (most recent call last):
  File "/home/andy/.local/share/venvs/42fa537b433f422253f0da1997538ecd/venv/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/home/andy/.local/share/venvs/42fa537b433f422253f0da1997538ecd/venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/andy/.local/share/venvs/42fa537b433f422253f0da1997538ecd/venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/andy/.local/share/venvs/42fa537b433f422253f0da1997538ecd/venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/andy/.local/share/venvs/42fa537b433f422253f0da1997538ecd/venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/andy/.local/share/venvs/42fa537b433f422253f0da1997538ecd/venv/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/andy/.local/share/venvs/42fa537b433f422253f0da1997538ecd/venv/lib/python3.9/site-packages/piptools/scripts/compile.py", line 466, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/home/andy/.local/share/venvs/42fa537b433f422253f0da1997538ecd/venv/lib/python3.9/site-packages/piptools/resolver.py", line 175, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/home/andy/.local/share/venvs/42fa537b433f422253f0da1997538ecd/venv/lib/python3.9/site-packages/piptools/resolver.py", line 319, in _resolve_one_round
    their_constraints.extend(self._iter_dependencies(best_match))
  File "/home/andy/.local/share/venvs/42fa537b433f422253f0da1997538ecd/venv/lib/python3.9/site-packages/piptools/resolver.py", line 428, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/home/andy/.local/share/venvs/42fa537b433f422253f0da1997538ecd/venv/lib/python3.9/site-packages/piptools/repositories/local.py", line 79, in get_dependencies
    return self.repository.get_dependencies(ireq)
  File "/home/andy/.local/share/venvs/42fa537b433f422253f0da1997538ecd/venv/lib/python3.9/site-packages/piptools/repositories/pypi.py", line 236, in get_dependencies
    self._dependencies_cache[ireq] = self.resolve_reqs(
  File "/home/andy/.local/share/venvs/42fa537b433f422253f0da1997538ecd/venv/lib/python3.9/site-packages/piptools/repositories/pypi.py", line 199, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "/home/andy/.local/share/venvs/42fa537b433f422253f0da1997538ecd/venv/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 379, in _resolve_one
    dist = self._get_dist_for(req_to_install)
  File "/home/andy/.local/share/venvs/42fa537b433f422253f0da1997538ecd/venv/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 331, in _get_dist_for
    self._populate_link(req)
  File "/home/andy/.local/share/venvs/42fa537b433f422253f0da1997538ecd/venv/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 300, in _populate_link
    req.link = self._find_requirement_link(req)
  File "/home/andy/.local/share/venvs/42fa537b433f422253f0da1997538ecd/venv/lib/python3.9/site-packages/pip/_internal/resolution/legacy/resolver.py", line 266, in _find_requirement_link
    best_candidate = self.finder.find_requirement(req, upgrade)
  File "/home/andy/.local/share/venvs/42fa537b433f422253f0da1997538ecd/venv/lib/python3.9/site-packages/pip/_internal/index/package_finder.py", line 909, in find_requirement
    raise DistributionNotFound(
pip._internal.exceptions.DistributionNotFound: No matching distribution found for certifi==2041.10.8 (from requests==2.26.0->-r reqs.in (line 1))

[AndydeCleyre]

The fix is still sitting in #1531, but I want to note here that the bug only seems to affect the legacy resolver, so using the backtracking resolver is a workaround until/unless the fix gets merged.