Explore decoupling from pip's internals, using the new --dry-run and --report flags

[pradyunsg]

What's the problem this feature will solve?

Currently, pip-tools depends on the internals of pip which is not a supported mode of use of pip.

Describe the solution you'd like

See pypa/pip#53 -- pip 22.2 (upcoming release) is going to add two new flags, that'll make it possible to get the resolution result from pip without performing an installation.

Alternative Solutions

Continue the current dependence on the fickle internals of pip, that I keep evolving and breaking pip-tools. :)

Additional context

Uhm... the issues on this repository, every time there's a new pip release? ;)

[sbidoul]

One thing that is not addressed with the current implementation of --report is the "multi platform" --generate-hashes, since pip install --dry-run --report only exposes the hashes of the artifacts it would)actually install, and not the hashes of other artifacts found on the index for the same version.

Yet, that should not stop us exploring further -- the current implementation should be sufficient for generating a single platform lock file, and I'm personally keen to better understand the needs and workflows around multi platform lock files, and if it is something that pip could help with, at least in principle.

[graingert]

I think this is a duplicate of #1526

Rather than add interfaces for pip-tools, pip should vendor this project and support both the constraints.txt format and the PEP 665 format

cc @brettcannon

[sbidoul]

I think this is a duplicate of #1526

#1526 is about implementing a new lock file format. This issue is about using a supported pip CLI instead of the unsupported pip internals to achieve the current functionality of pip-tools. So they are different.

To further clarify let me emphasize that

• the pip installation report is not meant to be a lock file format
• the pip installation report, when deemed stable, will be a supported pip feature, but it is not a PyPA interoperability standard
• pip does not and will not accept the installation report format as input for requirements to be installed

[AndydeCleyre]

Here's a primitive proof of concept using wheezy.template, which doesn't account for a lot, but demonstrates some potential of a template-oriented approach:

requirements.in:

requests
httpx

requirements.txt.wz:

@require(install)
@for pkg in sorted(install, key=lambda p: p['metadata']['name']):
@(
  required_by = sorted(
      needy_pkg['metadata']['name']
      for needy_pkg in install
      if pkg['metadata']['name'] in (
          dist.split()[0].split('[')[0]
          for dist in needy_pkg['metadata'].get('requires_dist', [])
      )
  )
  if pkg['requested']:
      required_by.insert(0, "-r requirements.in")

  annotation = f"# via {', '.join(required_by)}"

  pin = f"{pkg['metadata']['name']}=={pkg['metadata']['version']}"

  spaces = ' ' * max(26 - len(pin), 2)
)\
@{pin}@{spaces}@{annotation}
@end

$ pip install -U pip wheezy.template
$ pip install --ignore-installed --dry-run --report report.json -r requirements.in
$ wheezy.template requirements.txt.wz report.json >requirements.txt

requirements.txt:

anyio==3.6.1              # via httpcore
certifi==2022.6.15        # via httpcore, httpx, requests, urllib3
charset-normalizer==2.1.0  # via requests
h11==0.12.0               # via httpcore
httpcore==0.15.0          # via httpx
httpx==0.23.0             # via -r requirements.in
idna==3.3                 # via anyio, requests, rfc3986, urllib3
requests==2.28.1          # via -r requirements.in
rfc3986==1.5.0            # via httpx
sniffio==1.2.0            # via anyio, httpcore, httpx
urllib3==1.26.11          # via requests

I notice that for some reason, using real pip-compile doesn't include those urllib3 annotation entries 🤷🏼 .

[AndydeCleyre]

With a little more playing around with a fake pip-compile (without pip-tools, and invoking pip as a CLI), I stumble when trying to implement pip-compile's strategy of preferring to match existing output as constraints where possible, while discarding individual incompatible constraints encountered that way.

In pip-tools' code where pip's functions are called directly, we can catch the exception and inspect it to get these neat little cause items so we know which (preferred, but not mandatory) constraints to discard before trying again.

Invoking pip via CLI means we can't catch a rich exception object like that, and have the option of parsing natural language output to get the info we need.

[atugushev]

If only pip could report errors in json format too.

Without report:

$ pip install pip-tools==6.8.0 pip==20.1 --dry-run
ERROR: Cannot install pip-tools==6.8.0 and pip==20.1 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested pip==20.1
    pip-tools 6.8.0 depends on pip>=21.2

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

With report:

$ pip install pip-tools==6.8.0 pip==20.1 --dry-run --report -
{
  "error": {
    "code": "resolution_impossible",
    "description": "Cannot install pip-tools==6.8.0 and pip==20.1 because these package versions have conflicting dependencies",
    "extra": {
      "requirements": ["pip"]
    }
  }
}

[pradyunsg]

We could probably include rich information about the errors/failures in the report. Could you file an issue for this, over at pip's issue tracker?

[atugushev]

Thanks @pradyunsg. Here is the tracking issue: pypa/pip#11316.

[zundertj]

First of all: thank you for pip-tools, I find it very easy to use in developing Python applications and keeping comfort on version pinning.

I have been working on a new package, pipxl, to complement pip's offering, combining features from several existing packages without having to install all those, including compile and sync as found in pip-tools. Most of the features are powered by pip's dry-run & report json flags, and my aim is to stay away from poking with pip internals or developing my own resolver.

However, I run into the same issue as @AndydeCleyre here, on not upgrading dependencies of dependencies if there is no need. The suggested solution here, i.e. better reporting on resolver errors, does seem a bit like creating your own resolver? If I understand correctly, you would pick up the error, try a new version combination, and see if it passes? Wouldn't it be better if pip supports in addition to --ignored-installed an --assume-installed flag, overriding whatever is currently installed in the venv? Or in other words, make the resolver available as a function of <packages currently installed, packages to install> (and flags such as --no-deps, --upgrade-strategy, etc)

Hope I made myself clear here. Posting this here first, as I think we have the same problem, although I guess this is ultimately a pip request, not pip-tools.

[jeffwidman]

Over in our pip-compile update jobs are slow because pip-compile does a full-blown install. So I'm investigating ways to speed them up, and the big win appears to be when a package manager can generate requirements files without needing to do an install.

Reading this ticket it appears it's a combination of two things:

1. decoupling code from internals to using public CLI.
2. Supporting the dry-run + report flags such that pip-compile can generate the compiled requirements.txt file without doing a full install.

While obviously 1️⃣ is ideal for maintaining this library, as an end-user, I care mostly about 2️⃣ .

Is 1️⃣ a blocker for 2️⃣ ? Or is there a way I can pass a flag / option to the underlying pip to get it to use the dry-run functionality?

Now that backtracking is the default resolver, does that make this work easier?

[pradyunsg]

pip-compile doesn't do an install into the environment.

If you're seeing packages being installed, that's likely happening to build a source distribution into a wheel (https://pip.pypa.io/en/stable/reference/build-system/, specifically, the pyproject.toml case for details).

If you're seeing packages being downloaded whole, that's likely due to needing to read from the end of the .whl file to get the metadata.