Issue with VCS installs in `2024.1.0` -- resolver trying to resolve vcs as regular dependency

[matteius]

Traceback (most recent call last):
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/packaging/requirements.py", line 36, in __init__
    parsed = _parse_requirement(requirement_string)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/packaging/_parser.py", line 62, in parse_requirement
    return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/packaging/_parser.py", line 80, in _parse_requirement
    url, specifier, marker = _parse_requirement_details(tokenizer)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/packaging/_parser.py", line 124, in _parse_requirement_details
    marker = _parse_requirement_marker(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/packaging/_parser.py", line 145, in _parse_requirement_marker
    tokenizer.raise_syntax_error(
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/packaging/_tokenizer.py", line 167, in raise_syntax_error
    raise ParserSyntaxError(
pipenv.patched.pip._vendor.packaging._tokenizer.ParserSyntaxError: Expected end or semicolon (after name and no valid version specifier)
    pyminizip==
             ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/patched/pip/_internal/req/constructors.py", line 362, in _parse_req_string
    return get_requirement(req_as_string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/patched/pip/_internal/utils/packaging.py", line 45, in get_requirement
    return Requirement(req_string)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/packaging/requirements.py", line 38, in __init__
    raise InvalidRequirement(str(e)) from e
pipenv.patched.pip._vendor.packaging.requirements.InvalidRequirement: Expected end or semicolon (after name and no valid version specifier)
    pyminizip==
             ^

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mdavis/.pyenv/versions/3.11.6/bin/pipenv", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/cli/options.py", line 52, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/vendor/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/cli/command.py", line 207, in install
    do_install(
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/routines/install.py", line 234, in do_install
    raise e
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/routines/install.py", line 209, in do_install
    do_init(
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/routines/install.py", line 624, in do_init
    do_update(
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/routines/update.py", line 61, in do_update
    do_lock(
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/routines/lock.py", line 67, in do_lock
    venv_resolve_deps(
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 809, in venv_resolve_deps
    results = resolver.resolve_packages(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/resolver.py", line 635, in resolve_packages
    results = clean_results(results, resolver, project, category)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/resolver.py", line 574, in clean_results
    entry_dict = translate_markers(entry.get_cleaned_dict)
                                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/resolver.py", line 546, in __getattribute__
    return super().__getattribute__(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/functools.py", line 1001, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/resolver.py", line 253, in get_cleaned_dict
    self.validate_constraints()
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/resolver.py", line 490, in validate_constraints
    pinned_version = self.updated_version
                     ^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/resolver.py", line 546, in __getattribute__
    return super().__getattribute__(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/resolver.py", line 396, in updated_version
    version = str(self.entry.specifier)
                  ^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/resolver.py", line 546, in __getattribute__
    return super().__getattribute__(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/resolver.py", line 294, in entry
    self._entry = self.make_requirement(self.name, self.entry_dict)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/resolver.py", line 136, in make_requirement
    return from_pipfile(name, entry)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/utils/dependencies.py", line 1149, in from_pipfile
    install_req, markers, req_str = install_req_from_pipfile(name, pipfile)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/utils/dependencies.py", line 1135, in install_req_from_pipfile
    install_req, _ = expansive_install_req_from_line(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/utils/dependencies.py", line 1034, in expansive_install_req_from_line
    parts = parse_req_from_line(pip_line, line_source)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/patched/pip/_internal/req/constructors.py", line 379, in parse_req_from_line
    req: Optional[Requirement] = _parse_req_string(req_as_string)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdavis/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pipenv/patched/pip/_internal/req/constructors.py", line 376, in _parse_req_string
    raise InstallationError(msg)
pipenv.patched.pip._internal.exceptions.InstallationError: Invalid requirement: 'pyminizip==': Expected end or semicolon (after name and no valid version specifier)
    pyminizip==

[fredcohen-aa]

Seeing what I assume is the the same issue with this dependency in latest pipenv:
tensorflow-cpu = {"version"="~=2.13.0", markers="platform_machine != 'arm64'"}

Installing dependencies from Pipfile.lock (611b50)...
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/packaging/requirements.py", line 36, in __init__
    parsed = _parse_requirement(requirement_string)
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/packaging/_parser.py", line 62, in parse_requirement
    return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES))
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/packaging/_parser.py", line 80, in _parse_requirement
    url, specifier, marker = _parse_requirement_details(tokenizer)
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/packaging/_parser.py", line 124, in _parse_requirement_details
    marker = _parse_requirement_marker(
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/packaging/_parser.py", line 145, in _parse_requirement_marker
    tokenizer.raise_syntax_error(
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/packaging/_tokenizer.py", line 167, in raise_syntax_error
    raise ParserSyntaxError(
pipenv.patched.pip._vendor.packaging._tokenizer.ParserSyntaxError: Expected end or semicolon (after name and no valid version specifier)
    tensorflow==~=2.13.0
              ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/pipenv/patched/pip/_internal/req/constructors.py", line 362, in _parse_req_string
    return get_requirement(req_as_string)
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/pipenv/patched/pip/_internal/utils/packaging.py", line 45, in get_requirement
    return Requirement(req_string)
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/packaging/requirements.py", line 38, in __init__
    raise InvalidRequirement(str(e)) from e
pipenv.patched.pip._vendor.packaging.requirements.InvalidRequirement: Expected end or semicolon (after name and no valid version specifier)
    tensorflow==~=2.13.0

[matteius]

@fredcohen-aa 2024.0.3 is working better for vcs (assuming you aren't using env vars), but also what you are reporting is slightly different than a vcs issue though it seems to manifest similarly. I plan to look at this more this weekend, but if anyone has any ideas ahead of that, be sure to comment. We did upgrade a minor version of pip too, which is something else at play. I am fairly surprised the tests + the ones added didn't catch this.

[matteius]

I think this is going to be challenging issue as we move forward with pip >= 24.1.2 it appears to be more strict about some things such as version specifiers. For example -- I am unable to lock the parent package django-trench successfully from git/vcs, sometimes it did work but always had a null ref, and adding additional vcs it failed it. But I forked the package and tried changing the version like so: matteius/django-trench@821b421

Now it just works again 🤔

[matteius]

@fredcohen-aa please open a different issue report for what you are seeing with even more details than you provide above.

[fredcohen-aa]

@fredcohen-aa please open a different issue report for what you are seeing with even more details than you provide above.

Issue #6273 opened

[matteius]

Analysis for Issue #6266:

Analysis of Pipenv Issue #6266

1. Problem Summary:

The core problem is that Pipenv's resolver in version 2024.1.0 mistakenly attempts to resolve VCS dependencies as regular dependencies, leading to a ParserSyntaxError. The issue seems to occur when processing requirement strings, specifically when parsing VCS URLs. The provided traceback indicates an issue with a requirement line containing "pyminizip==" without a valid version specifier, suggesting Pipenv is treating the VCS URL as a standard package name and version.

2. Comment Discussion Analysis:

• Similar but Different Issue: The maintainer (@matteius) acknowledges the issue but points out that the reported case differs from a pure VCS issue. They mention that Pipenv 2024.0.3 improved VCS handling, implying there may be separate bugs related to VCS and marker parsing.
• Stricter Parsing in Pip: The maintainer highlights the stricter parsing in newer Pip versions (>= 24.1.2) as a potential contributing factor. This suggests that Pipenv's internal logic for constructing requirement strings may need to adapt to these changes.
• Potential Conflict with Markers: The comment from @fredcohen-aa suggests that the issue might be related to the presence of environment markers in the requirement line. The traceback he provides shows a similar error ("Expected end or semicolon") when using a marker alongside a version specifier.

3. Proposed Resolution:

The issue seems to be rooted in how Pipenv constructs requirement strings for VCS dependencies. Pipenv needs to ensure that VCS URLs are treated differently from standard package names and versions. This includes:

• Preserving VCS URL structure: The requirement string passed to the resolver should retain the VCS prefix and the entire URL structure, including the "@" symbol, to ensure correct parsing.
• Handling environment markers: If markers are present, they should be appended correctly to the VCS URL requirement string, potentially using a semicolon (;) as a separator.

Potential Code Changes:

• File: pipenv/utils/dependencies.py
• Function: dependency_as_pip_install_line

4. Code Snippet:

def dependency_as_pip_install_line(
    # ... existing parameters
):
    # ... existing code ...
    elif vcs and vcs in dep:  # VCS Requirements
        # ... (existing logic for constructing git_req) ...

        # Ensure VCS URL is preserved
        if include_markers and dep.get("markers"):
            line.append(f"{git_req}; {dep['markers']}")  # append markers with semicolon
        else:
            line.append(git_req)  # append the unmodified VCS URL string

    # ... remaining code ...

5. Additional Steps/Investigations:

• Thorough Review of Requirement String Construction: Carefully review the dependency_as_pip_install_line function and related code to identify any other instances where VCS URLs might be incorrectly parsed or modified.
• Extensive Testing: Develop test cases covering various VCS URLs with and without environment markers, ensuring the generated requirement strings are valid and the resolver handles them correctly.
• Collaboration with Pip Maintainers: Given the interaction with Pip's stricter parsing, collaborating with Pip maintainers could help identify the root cause and ensure a robust solution.
• Documentation: Update the Pipenv documentation to clarify the supported syntax for VCS dependencies in Pipfiles.

By addressing these points, Pipenv can ensure that VCS dependencies are correctly handled and prevent the ParserSyntaxError while accommodating newer Pip versions' stricter parsing requirements.

[matteius]

Fixed by: #6276