pipenv (patched pip) corrupts path for editable requirement on Windows

[asrelo]

Issue description

I try to install a package into pipenv's virtual environment as editable (either using relative path pipenv install -e . or absolute path pipenv install -e D:\long\windows\path\project). I get the error with a traceback with this exception: pipenv.patched.pip._internal.exceptions.InstallationError: D:longwindowspathproject is not a valid editable requirement. It should either be a path to a local project or a VCS URL.... (Full traceback provided below.)

With some adhoc debugging I found that the path is first resolved if it's relative, and then the absolute path gets corrupted (maybe not only) here (link):

options = shlex.split(options_str)

While running install -e ... commands, this line executes twice:

1. options_str == '-i https://pypi.org/simple C:\\Users\\user\\AppData\\Local\\pipenv\\pipenv\\Cache',
   options == ['-i', 'https://pypi.org/simple', 'C:UsersuserAppDataLocalpipenvpipenvCache']

2. options_str == '-e D:\\long\\windows\\path\\project'
options == ['-e', 'D:longwindowspathproject']

shlex.split for some reason removes every backslash from a Windows path, I have tested it in IDLE. I have also tried to run pipenv install -e D:/long/windows/path/project, no changes happened.

When I tried to replicate it with just virtualenv and pip, this line in pip was never executed. (theories ahead) I suppose this is because pipenv creates a temporary requirements.txt to supply it to pip, and somehow it goes wrong. Maybe for local files it should use URLs with file scheme instead of plain absolute paths? ( hinted link)

Steps to replicate

1. be on Windows (10)

2. have a Python project which already has associated virtual environment managed by pipenv (I created it with python -m pipenv --python=<python-path> at project root)

3.1. from the root directory of the project, run python -m pipenv install -e . (as in docs)
OR
3.2. run python -m pipenv install -e <project-absolute-path>

Expected result

The package is installed into associated virtual environment as editable.

Actual result

The package is not installed, pipenv exits with an error, producing this:

long traceback

d:\long\windows\path\project>python -m pipenv install -e . --dev --verbose
Loading .env environment variables...
Installing -e ....
Resolving -e ....
Installation Succeeded
Pipfile.lock (c606b7) out of date, updating to (916e2e)...
Locking [packages] dependencies...
Locking [dev-packages] dependencies...
Building requirements...
Resolving dependencies...
Traceback (most recent call last):
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 675, in <module>
    main()
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 661, in main
    _main(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 645, in _main
    resolve_packages(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 612, in resolve_packages
    results, resolver = resolve(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 592, in resolve
    return resolve_deps(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 918, in resolve_deps
    results, hashes, internal_resolver = actually_resolve_deps(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 691, in actually_resolve_deps
    resolver.resolve()
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 443, in resolve
    constraints = self.constraints
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 401, in constraints
    possible_constraints_list = self.possible_constraints
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 388, in possible_constraints
    possible_constraints_list = [
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 389, in <listcomp>
    install_req_from_parsed_requirement(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_internal\req\constructors.py", line 485, in install_req_from_parsed_requirement
    req = install_req_from_editable(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_internal\req\constructors.py", line 240, in install_req_from_editable
    parts = parse_req_from_editable(editable_req)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_internal\req\constructors.py", line 209, in parse_req_from_editable
    name, url, extras_override = parse_editable(editable_req)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_internal\req\constructors.py", line 126, in parse_editable
    raise InstallationError(
pipenv.patched.pip._internal.exceptions.InstallationError: D:longwindowspathproject is not a valid editable requirement. It should either be a path to a local project or a VCS URL (beginning with bzr+http, bzr+https, bzr+ssh, bzr+sftp, bzr+ftp, bzr+lp, bzr+file, git+http, git+https, git+ssh, git+git, git+file, hg+file, hg+http, hg+https, hg+ssh, hg+static-http, svn+ssh, svn+http, svn+https, svn+svn, svn+file).
Locking Failed!
[    ] Locking...
Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\__main__.py", line 4, in <module>
    cli()
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\cli\options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\cli\command.py", line 209, in install
    do_install(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\install.py", line 297, in do_install
    raise e
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\install.py", line 281, in do_install
    do_init(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\install.py", line 648, in do_init
    do_lock(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\lock.py", line 65, in do_lock
    venv_resolve_deps(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 859, in venv_resolve_deps
    c = resolve(cmd, st, project=project)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 728, in resolve
    raise RuntimeError("Failed to lock Pipfile.lock!")
RuntimeError: Failed to lock Pipfile.lock!

---

$ pipenv --support

Pipenv version: '2023.11.17'

Pipenv location: 'C:\\Users\\user\\AppData\\Roaming\\Python\\Python310\\site-packages\\pipenv'

Python location: 'C:\\Program Files\\Python310\\python.exe'

OS Name: 'nt'

User pip version: '23.3.2'

user Python installations found:

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.10.11',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.19045',
 'python_full_version': '3.10.11',
 'python_version': '3.10',
 'sys_platform': 'win32'}

System environment variables:

...

• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIPENV_VENV_IN_PROJECT
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv-specific environment variables:

• PIPENV_VENV_IN_PROJECT: 1

Debug-specific environment variables:

• PATH: ...;C:\Program Files\Python310\Scripts\;C:\Program Files\Python310\;...;C:\Windows\System32;C:\Windows;C:\Windows\System32\wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;...;C:\Users\user\AppData\Roaming\Python\Python310\Scripts;...;

---

Contents of Pipfile ('D:\long\windows\path\project\Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.10"
python_full_version = "3.10.11"

Contents of Pipfile.lock ('D:\long\windows\path\project\Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "..."
        },
        "pipfile-spec": 6,
        "requires": {
            "python_full_version": "3.10.11",
            "python_version": "3.10"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {},
    "develop": {}
}

[matteius]

Please try specifying the name@ prefix syntax before specifying your editable requirement, I think its struggling on identifying the correct name from the project config for your library.

[asrelo]

I also tried to specify path on command as file-scheme URL, it got resolved to the same D:longwindowspathproject.

[asrelo]

Is it like this...

d:\long\windows\path\project>python -m pipenv install --dev -e package@"." --verbose

output

Loading .env environment variables...
Installing -e package@....
Resolving -e package@....
Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\__main__.py", line 4, in <module>
    cli()
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\cli\options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\cli\command.py", line 209, in install
    do_install(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\install.py", line 212, in do_install
    pkg_requirement, _ = expansive_install_req_from_line(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\dependencies.py", line 962, in expansive_install_req_from_line
    return install_req_from_editable(pip_line, line_source), name
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_internal\req\constructors.py", line 240, in install_req_from_editable
    parts = parse_req_from_editable(editable_req)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_internal\req\constructors.py", line 209, in parse_req_from_editable
    name, url, extras_override = parse_editable(editable_req)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_internal\req\constructors.py", line 126, in parse_editable
    raise InstallationError(
pipenv.patched.pip._internal.exceptions.InstallationError: package@. is not a valid editable requirement. It should either be a path to a local project or a VCS URL (beginning with bzr+http, bzr+https, bzr+ssh, bzr+sftp, bzr+ftp, bzr+lp, bzr+file, git+http, git+https, git+ssh, git+git, git+file, hg+file, hg+http, hg+https, hg+ssh, hg+static-http, svn+ssh, svn+http, svn+https, svn+svn, svn+file).

Same with an absolute path, but the exception text is pipenv.patched.pip._internal.exceptions.InstallationError: package@D:\long\windows\path\project is not a valid editable requirement...

[matteius]

Hmmm, I'll have to think on this more, just wondering though which shell, is it windows power shell? I want to be able to reproduce the issue.

[asrelo]

I used cmd.exe on Windows 10 initially.

I just tried python -m pipenv install -e . in powershell, it got installed on first try. I could not replicate the problem in powershell, though I tried it several times both from cmd.exe (with the error) and powershell (successfully, cleaning Pipfile and lockfile afterwards).

I don't know powershell stuff, printf debugging doesn't work well... That corrupting line is executed twice:

1. options_str == '-i https://pypi.org/simple C:\\Users\\user\\AppData\\Local\\pipenv\\pipenv\\Cache'
options == ['-i', 'https://pypi.org/simple', 'C:UsersuserAppDataLocalpipenvpipenvCache']

2. options_str == '-e .'
options == ['-e', '.']

Also, even when I specify the absolute path on the same drive, it comes to this line as relative. If it's the same path and the relative is ., installation proceeds without errors. If it's a different path, the same exception pipenv.patched.pip._internal.exceptions.InstallationError occurs with corrupted relative path.

So the problem with shlex persists, but since the path . is not resolved when running from powershell (for some reason), it does not cause a visible error.

Also, when I specify the absolute path on a different drive, I get this:

command, output

PS D:\long\windows\path\project> python -m pipenv install -e "C:\Users\user\project\"
Loading .env environment variables...
Installing -e C:\Users\user\package\...
Resolving -e C:\Users\user\package\...
Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\__main__.py", line 4, in <module>
    cli()
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\cli\options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\cli\command.py", line 209, in install
    do_install(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\install.py", line 255, in do_install
    added, cat, normalized_name = project.add_package_to_pipfile(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\project.py", line 1211, in add_package_to_pipfile
    name, normalized_name, entry = self.generate_package_pipfile_entry(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\project.py", line 1155, in generate_package_pipfile_entry
    req_name = determine_package_name(package)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\dependencies.py", line 792, in determine_package_name
    req_name = find_package_name_from_directory(package.link.file_path)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\dependencies.py", line 644, in find_package_name_from_directory
    os.listdir(directory),
FileNotFoundError: [WinError 3] The system cannot find the path specified: '\\Users\\user\\project'

It happens because here in pipenv the original absolute path 'C:\\Users\\user\\project' is supplied to urllib.parse.urlparse, which treats drive letter as URL scheme c, and the following code removes this scheme. I have no idea what's going on there.

I checked the cache dir which path is also corrupted: any recent files are related to my other work with pipenv where I don't use editable dependencies.

[asrelo]

example to reproduce:

1. be on Windows (Windows 10)
2. clone this repo into pipenv-issue-6068-demo (commit 2131b01d7ed7c723261f0e4ba298a46524dad41e )
3. open cmd.exe
4. cd pipenv-issue-6068-demo
5. python -m pipenv install -e . (I run Python 3.10, pipenv 2023.11.17)

this produces the error from the initial comment, with exception: pipenv.patched.pip._internal.exceptions.InstallationError: D:longpathpipenv-issue-6068-demo is not a valid editable requirement. It should either be a path to a local project or a VCS URL ...

[matteius]

@asrelo thanks for providing an example -- I am checking it out, but I do see that the local file install doesn't have a pyproject.toml, or setup.cfg or setup.py so technically I don't think its installable -- are you actually able to pip install that as a local package?

[asrelo]

There is pyproject.toml at root of pipenv-issue-6068-demo, I'm not aware what is wrong with that.

Yes, virtualenv + pip install -e . successfully installs the package. This was the session from within this same repo in cmd.exe:

cmd.exe session

d:\long\windows\path\pipenv-issue-6068-demo>python -m virtualenv testenv
created virtual environment CPython3.10.11.final.0-64 in 7748ms
  creator CPython3Windows(dest=D:\long\windows\path\pipenv-issue-6068-demo\testenv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\user\AppData\Local\pypa\virtualenv)
    added seed packages: pip==23.3.2, setuptools==69.0.3, wheel==0.42.0
  activators BashActivator,BatchActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

d:\long\windows\path\pipenv-issue-6068-demo>.\testenv\Scripts\activate

(testenv) d:\long\windows\path\pipenv-issue-6068-demo>pip install -e .
Obtaining file:///D:/long/windows/path/pipenv-issue-6068-demo
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: problem-project
  Building editable for problem-project (pyproject.toml) ... done
  Created wheel for problem-project: filename=problem_project-0.1.0-0.editable-py3-none-any.whl size=1594 sha256=f76979778a63df8472b46c266f35d72fefe6ea130c3bf8329b955e1a92c2a68f
  Stored in directory: C:\Users\user\AppData\Local\Temp\pip-ephem-wheel-cache-q_w0s3yo\wheels\ab\55\bc\7e8acf252f651d596add8fe1a7bd9aa460d7f1424001943dd5
Successfully built problem-project
Installing collected packages: problem-project
Successfully installed problem-project-0.1.0

(testenv) d:\long\windows\path\pipenv-issue-6068-demo>problem_package
Hello World!

upd: and it is actually editable

[matteius]

git bash, cmd.exe and powershell are all working on mine:

C:\Users\matte\Projects\pipenv-triage\pipenv-issue-6068-demo>pipenv install .
Loading .env environment variables...
Installing ....
Resolving ....
Installation Succeeded
Installing dependencies from Pipfile.lock (95c512)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

C:\Users\matte\Projects\pipenv-triage\pipenv-issue-6068-demo>pipenv run problem_package
Loading .env environment variables...
Hello World!

🤔 Only change I made to your example was to allow python 3.11 so I wouldn't have to edit my PATH for the python3.10 version.

[asrelo]

I just tried with freshly installed Python 3.11.7 and pipenv 2023.11.17. The error was the same, only the traceback was fancier...

I'll try to reproduce the example on a different machine soon (though shlex itself worked there exactly the same).

[asrelo]

Tried to reproduce with another person. The initial error with pipenv install -e . does not happen; shlex.split there receives relative path . and proceeds without errors. I have no clue why that path . gets resolved to absolute path only when pipenv is running from my cmd.exe.

However, pipenv install -e <relative-path> does not work, as in (comment):

If it's a different path, the same exception pipenv.patched.pip._internal.exceptions.InstallationError occurs with corrupted relative path.

Cache path gets corrupted too.

We didn't try to specify a path on a different drive as in the comment, now I humbly assume there would be the same error as before.

[matteius]

@asrelo I believe that is because the relative path is now the src directory which is missing the pyproject.toml (its in the parent).