stop using requirementslib models (#5793)

* Move away from requirementslib models

* Revise test since PEP-440 does not support wildcard versions but does support equivalent compatible release specifiers.

* simplify and remove dead code

* Ensure the os_name marker is AND with the other markers.

* Move what we still need from requirementslib into the pipenv utils and stop vendoring it.

* Remove requirementslib.

* force upgrade of virtualenv for python 3.12

* remove virtualenv-clone

* Update vcs specifiers documentation; infer name from specific pip line formats where possible.

* Provide helpful text and error for recently removed commands

* Set the right log levels and verbosity to show users the errors generated by pip resolver when supplying -v flag

* Fix the collection of all matching package hashes for non-pypi indexes.  Plus lesson from testing torch which contains local identifiers.

Author: matteius

.github/workflows/ci.yaml

@@ -75,7 +75,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: [3.7, 3.8, 3.9, "3.10", "3.11"]
+        python-version: [3.7, 3.8, 3.9, "3.10", "3.11"]  # "3.12-dev" Windows CI hangs indefinitely
         os: [MacOS, Ubuntu, Windows]
 
     steps:

Pipfile

@@ -25,6 +25,7 @@ myst-parser = {extras = ["linkify"], version = "*"}
 invoke = "==2.0.0"
 exceptiongroup = "==1.1.0"
 tomli = "*"
+pyyaml = "==6.0.1"
 
 [packages]
 pytz = "*"

Pipfile.lock

@@ -103,30 +103,54 @@ All sub-dependencies will get added to the `Pipfile.lock` as well. Sub-dependenc
 
 ## VCS Dependencies
 
-VCS dependencies from git and other version control systems using URLs formatted according to the following rule:
+VCS dependencies from git and other version control systems using URLs formatted using preferred pip line formats:
 
-    <vcs_type>+<scheme>://<location>/<user_or_organization>/<repository>@<branch_or_tag>#egg=<package_name>
+    <vcs_type>+<scheme>://<location>/<user_or_organization>/<repository>@<branch_or_tag>
 
-The only optional section is the `@<branch_or_tag>` section.  When using git over SSH, you may use the shorthand vcs and scheme alias `git+git@<location>:<user_or_organization>/<repository>@<branch_or_tag>#egg=<package_name>`. Note that this is translated to `git+ssh://git@<location>` when parsed.
+Extras may be specified using the following format when issuing install command:
+
+    <package_name><possible_extras>@ <vcs_type>+<scheme>://<location>/<user_or_organization>/<repository>@<branch_or_tag>
+
+Note: that the #egg fragments should only be used for legacy pip lines which are still required in editable requirements.
+
+    $ pipenv install -e git+https://github.com/requests/requests.git@v2.31.0#egg=requests
 
-Note that it is **strongly recommended** that you install any version-controlled dependencies in editable mode, using `pipenv install -e`, in order to ensure that dependency resolution can be performed with an up-to-date copy of the repository each time it is performed, and that it includes all known dependencies.
 
 Below is an example usage which installs the git repository located at `https://github.com/requests/requests.git` from tag `v2.20.1` as package name `requests`:
 
     $ pipenv install -e git+https://github.com/requests/requests.git@v2.20.1#egg=requests
-    Creating a Pipfile for this project...
     Installing -e git+https://github.com/requests/requests.git@v2.20.1#egg=requests...
-    [...snipped...]
-    Adding -e git+https://github.com/requests/requests.git@v2.20.1#egg=requests to Pipfile's [packages]...
-    [...]
+    Resolving -e git+https://github.com/requests/requests.git@v2.20.1#egg=requests...
+    Added requests to Pipfile's [packages] ...
+    Installation Succeeded
+    Pipfile.lock not found, creating...
+    Locking [packages] dependencies...
+    Building requirements...
+    Resolving dependencies...
+    Success!
+    Locking [dev-packages] dependencies...
+    Updated Pipfile.lock (389441cc656bb774aaa28c7e53a35137aace7499ca01668765d528fa79f8acc8)!
+    Installing dependencies from Pipfile.lock (f8acc8)...
+    To activate this project's virtualenv, run pipenv shell.
+    Alternatively, run a command inside the virtualenv with pipenv run.
 
     $ cat Pipfile
     [packages]
-    requests = {git = "https://github.com/requests/requests.git", editable = true, ref = "v2.20.1"}
+    requests = {editable = true, ref = "v2.20.1", git = "git+https://github.com/requests/requests.git"}
+
+    $ cat Pipfile.lock
+    ...
+    "requests": {
+        "editable": true,
+        "git": "git+https://github.com/requests/requests.git",
+        "markers": "python_version >= '3.7'",
+        "ref": "6cfbe1aedd56f8c2f9ff8b968efe65b22669795b"
+    },
+    ...
 
 Valid values for `<vcs_type>` include `git`, `bzr`, `svn`, and `hg`.  Valid values for `<scheme>` include `http`, `https`, `ssh`, and `file`.  In specific cases you also have access to other schemes: `svn` may be combined with `svn` as a scheme, and `bzr` can be combined with `sftp` and `lp`.
 
-You can read more about pip's implementation of VCS support `here <https://pip.pypa.io/en/stable/reference/pip_install/#vcs-support>`__. For more information about other options available when specifying VCS dependencies, please check the `Pipfile spec <https://github.com/pypa/pipfile>`_.
+You can read more about pip's implementation of VCS support `here <https://pip.pypa.io/en/stable/reference/pip_install/#vcs-support>`__.
 
 
 ## Specifying Package Categories

docs/specifiers.md

@@ -0,0 +1,2 @@
+Drop requirementslib for managing pip lines and InstallRequirements, bring remaining requirementslib functionality into pipenv.
+Fixes numerous reports about extras installs with vcs and file installs; format pip lines correctly to not generate deprecation warnings.

news/5793.bugfix.rst

@@ -3,8 +3,18 @@
 import sys
 import warnings
 
+# This has to come before imports of pipenv
+PIPENV_ROOT = os.path.abspath(os.path.dirname(os.path.realpath(__file__)))
+PIP_ROOT = os.sep.join([PIPENV_ROOT, "patched", "pip"])
+sys.path.insert(0, PIPENV_ROOT)
+sys.path.insert(0, PIP_ROOT)
+
+# Load patched pip instead of system pip
+os.environ["PIP_DISABLE_PIP_VERSION_CHECK"] = "1"
+
 
 def _ensure_modules():
+    # Can be removed when we drop pydantic
     spec = importlib.util.spec_from_file_location(
         "typing_extensions",
         location=os.path.join(
@@ -14,6 +24,14 @@ def _ensure_modules():
     typing_extensions = importlib.util.module_from_spec(spec)
     sys.modules["typing_extensions"] = typing_extensions
     spec.loader.exec_module(typing_extensions)
+    # Ensure when pip gets invoked it uses our patched version
+    spec = importlib.util.spec_from_file_location(
+        "pip",
+        location=os.path.join(os.path.dirname(__file__), "patched", "pip", "__init__.py"),
+    )
+    pip = importlib.util.module_from_spec(spec)
+    sys.modules["pip"] = pip
+    spec.loader.exec_module(pip)
 
 
 _ensure_modules()
@@ -26,8 +44,6 @@ def _ensure_modules():
 warnings.filterwarnings("ignore", category=ResourceWarning)
 warnings.filterwarnings("ignore", category=UserWarning)
 
-# Load patched pip instead of system pip
-os.environ["PIP_DISABLE_PIP_VERSION_CHECK"] = "1"
 
 if os.name == "nt":
     from pipenv.vendor import colorama

pipenv/__init__.py

@@ -216,7 +216,7 @@ def install(state, **kwargs):
         requirementstxt=state.installstate.requirementstxt,
         pre=state.installstate.pre,
         deploy=state.installstate.deploy,
-        index_url=state.index,
+        index=state.index,
         packages=state.installstate.packages,
         editable_packages=state.installstate.editables,
         site_packages=state.site_packages,
@@ -610,7 +610,7 @@ def run_open(state, module, *args, **kwargs):
         [
             state.project._which("python"),
             "-c",
-            "import {0}; print({0}.__file__)".format(module),
+            f"import {module}; print({module}.__file__)",
         ]
     )
     if c.returncode:
@@ -693,11 +693,10 @@ def scripts(state):
     scripts = state.project.parsed_pipfile.get("scripts", {})
     first_column_width = max(len(word) for word in ["Command"] + list(scripts))
     second_column_width = max(len(word) for word in ["Script"] + list(scripts.values()))
-    lines = ["{0:<{width}}  Script".format("Command", width=first_column_width)]
-    lines.append("{}  {}".format("-" * first_column_width, "-" * second_column_width))
+    lines = [f"{command:<{first_column_width}}  Script" for command in ["Command"]]
+    lines.append(f"{'-' * first_column_width}  {'-' * second_column_width}")
     lines.extend(
-        "{0:<{width}}  {1}".format(name, script, width=first_column_width)
-        for name, script in scripts.items()
+        f"{name:<{first_column_width}}  {script}" for name, script in scripts.items()
     )
     console.print("\n".join(line for line in lines))
 

pipenv/cli/command.py

@@ -2,6 +2,7 @@
 import re
 
 from pipenv.project import Project
+from pipenv.utils import err
 from pipenv.utils.internet import is_valid_url
 from pipenv.vendor.click import (
     BadArgumentUsage,
@@ -483,6 +484,83 @@ def validate_pypi_mirror(ctx, param, value):
     return value
 
 
+# OLD REMOVED COMMANDS THAT WE STILL DISPLAY HELP TEXT FOR #
+def skip_lock_option(f):
+    def callback(ctx, param, value):
+        if value:
+            err.print(
+                "The flag --skip-lock has been functionally removed.  "
+                "Without running the lock resolver it is not possible to manage multiple package indexes.  "
+                "Additionally it bypassed the build consistency guarantees provided by maintaining a lock file.",
+                style="yellow bold",
+            )
+            raise ValueError("The flag --skip-lock flag has been removed.")
+        return value
+
+    return option(
+        "--skip-lock",
+        is_flag=True,
+        default=False,
+        expose_value=False,
+        envvar="PIPENV_SKIP_LOCK",
+        callback=callback,
+        type=click_types.BOOL,
+        show_envvar=True,
+        hidden=True,  # This hides the option from the help text.
+    )(f)
+
+
+def keep_outdated_option(f):
+    def callback(ctx, param, value):
+        state = ctx.ensure_object(State)
+        state.installstate.keep_outdated = value
+        if value:
+            err.print(
+                "The flag --keep-outdated has been removed.  "
+                "The flag did not respect package resolver results and lead to inconsistent lock files.  "
+                "Consider using the `pipenv upgrade` command to selectively upgrade packages.",
+                style="yellow bold",
+            )
+            raise ValueError("The flag --keep-outdated flag has been removed.")
+        return value
+
+    return option(
+        "--keep-outdated",
+        is_flag=True,
+        default=False,
+        expose_value=False,
+        callback=callback,
+        type=click_types.BOOL,
+        show_envvar=True,
+        hidden=True,  # This hides the option from the help text.
+    )(f)
+
+
+def selective_upgrade_option(f):
+    def callback(ctx, param, value):
+        state = ctx.ensure_object(State)
+        state.installstate.selective_upgrade = value
+        if value:
+            err.print(
+                "The flag --selective-upgrade has been removed.  "
+                "The flag was buggy and lead to inconsistent lock files.  "
+                "Consider using the `pipenv upgrade` command to selectively upgrade packages.",
+                style="yellow bold",
+            )
+            raise ValueError("The flag --selective-upgrade flag has been removed.")
+        return value
+
+    return option(
+        "--selective-upgrade",
+        is_flag=True,
+        default=False,
+        type=click_types.BOOL,
+        help="Update specified packages.",
+        callback=callback,
+        expose_value=False,
+    )(f)
+
+
 def common_options(f):
     f = pypi_mirror_option(f)
     f = verbose_option(f)
@@ -496,6 +574,7 @@ def install_base_options(f):
     f = common_options(f)
     f = pre_option(f)
     f = extra_pip_args(f)
+    f = keep_outdated_option(f)  # Removed, but still displayed in help text.
     return f
 
 
@@ -505,6 +584,7 @@ def uninstall_options(f):
     f = uninstall_dev_option(f)
     f = editable_option(f)
     f = package_arg(f)
+    f = skip_lock_option(f)  # Removed, but still displayed in help text.
     return f
 
 
@@ -530,6 +610,8 @@ def install_options(f):
     f = ignore_pipfile_option(f)
     f = editable_option(f)
     f = package_arg(f)
+    f = skip_lock_option(f)  # Removed, but still display help text.
+    f = selective_upgrade_option(f)  # Removed, but still display help text.
     return f
 
 

pipenv/cli/options.py

@@ -12,21 +12,23 @@
 from pathlib import Path
 from sysconfig import get_paths, get_python_version, get_scheme_names
 from urllib.parse import urlparse
-from urllib.request import url2pathname
 
 import pipenv
 from pipenv.patched.pip._internal.commands.install import InstallCommand
 from pipenv.patched.pip._internal.index.package_finder import PackageFinder
+from pipenv.patched.pip._internal.req.req_install import InstallRequirement
 from pipenv.patched.pip._vendor import pkg_resources
+from pipenv.patched.pip._vendor.packaging.specifiers import SpecifierSet
 from pipenv.patched.pip._vendor.packaging.utils import canonicalize_name
 from pipenv.utils import console
+from pipenv.utils.constants import VCS_LIST
+from pipenv.utils.dependencies import as_pipfile
+from pipenv.utils.fileutils import normalize_path, temp_path
 from pipenv.utils.funktools import chunked, unnest
 from pipenv.utils.indexes import prepare_pip_source_args
 from pipenv.utils.processes import subprocess_run
-from pipenv.utils.shell import make_posix
+from pipenv.utils.shell import make_posix, temp_environ
 from pipenv.vendor.pythonfinder.utils import is_in_path
-from pipenv.vendor.requirementslib.fileutils import normalize_path, temp_path
-from pipenv.vendor.requirementslib.utils import temp_environ
 
 try:
     # this is only in Python3.8 and later
@@ -200,11 +202,6 @@ def base_paths(self) -> dict[str, str]:
         .. note:: The implementation of this is borrowed from a combination of pip and
            virtualenv and is likely to change at some point in the future.
 
-        >>> from pipenv.core import project
-        >>> from pipenv.environment import Environment
-        >>> env = Environment(prefix=project.virtualenv_location, is_venv=True, sources=project.sources)
-        >>> import pprint
-        >>> pprint.pprint(env.base_paths)
         {'PATH': '/home/hawk/.virtualenvs/pipenv-MfOPs1lW/bin::/bin:/usr/bin',
         'PYTHONPATH': '/home/hawk/.virtualenvs/pipenv-MfOPs1lW/lib/python3.7/site-packages',
         'data': '/home/hawk/.virtualenvs/pipenv-MfOPs1lW',
@@ -760,38 +757,46 @@ def is_installed(self, pkgname):
 
         return any(d for d in self.get_distributions() if d.project_name == pkgname)
 
-    def is_satisfied(self, req):
+    def is_satisfied(self, req: InstallRequirement):
         match = next(
             iter(
                 d
                 for d in self.get_distributions()
-                if canonicalize_name(d.project_name) == req.normalized_name
+                if req.name
+                and canonicalize_name(d.project_name) == canonicalize_name(req.name)
             ),
             None,
         )
         if match is not None:
-            if req.editable and req.line_instance.is_local and self.find_egg(match):
-                requested_path = req.line_instance.path
+            if req.editable and req.link and req.link.is_file:
+                requested_path = req.link.file_path
                 if os.path.exists(requested_path):
                     local_path = requested_path
                 else:
                     parsed_url = urlparse(requested_path)
-                    local_path = url2pathname(parsed_url.path)
+                    local_path = parsed_url.path
                 return requested_path and os.path.samefile(local_path, match.location)
             elif match.has_metadata("direct_url.json"):
                 direct_url_metadata = json.loads(match.get_metadata("direct_url.json"))
-                commit_id = direct_url_metadata.get("vcs_info", {}).get("commit_id", "")
+                requested_revision = direct_url_metadata.get("vcs_info", {}).get(
+                    "requested_revision", ""
+                )
                 vcs_type = direct_url_metadata.get("vcs_info", {}).get("vcs", "")
-                _, pipfile_part = req.as_pipfile().popitem()
+                _, pipfile_part = as_pipfile(req).popitem()
+                vcs_ref = ""
+                for vcs in VCS_LIST:
+                    if pipfile_part.get(vcs):
+                        vcs_ref = pipfile_part[vcs].rsplit("@", 1)[-1]
+                        break
                 return (
-                    vcs_type == req.vcs
-                    and commit_id == req.commit_hash
-                    and direct_url_metadata["url"] == pipfile_part[req.vcs]
+                    vcs_type == req.link.scheme
+                    and vcs_ref == requested_revision
+                    and direct_url_metadata["url"] == pipfile_part[req.link.scheme]
                 )
-            elif req.is_vcs or req.is_file_or_url:
+            elif req.link and req.link.is_vcs:
                 return False
-            elif req.line_instance.specifiers is not None:
-                return req.line_instance.specifiers.contains(
+            elif req.specifier is not None:
+                return SpecifierSet(str(req.specifier)).contains(
                     match.version, prereleases=True
                 )
             return True

pipenv/environment.py

@@ -5,8 +5,8 @@
 import sys
 
 from pipenv.patched.pip._vendor.platformdirs import user_cache_dir
+from pipenv.utils.fileutils import normalize_drive
 from pipenv.utils.shell import env_to_bool, is_env_truthy, isatty
-from pipenv.vendor.requirementslib.fileutils import normalize_drive
 
 # HACK: avoid resolver.py uses the wrong byte code files.
 # I hope I can remove this one day.