Errors under Windows with installation prefix starting with \\?\

[grv87]

Description

Error when pip install is run with --prefix starting with \\?\ (for example, a directory on letter-less volume in #10597, but not limited to).

pipenv install -e . calls pip with this option, so this issue hinders the use of pipenv.

Expected behavior

Successful installation

pip version

22.1.2

Python version

3.10

OS

Windows 10

How to Reproduce

pip install --ignore-installed --prefix \\?\%TEMP% setuptools

Output

Collecting setuptools
  Using cached setuptools-62.4.0-py3-none-any.whl (1.2 MB)
Installing collected packages: setuptools
ERROR: Could not install packages due to an OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: '\\\\?\\C:\\Users\\<user>\\AppData\\Local\\Temp/Lib'

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[grv87]

I did a preliminary investigation:

• The error is returned from CreateDirectoryW. This function normalize separators in regular paths (starting with drive letter), but doesn't normalize paths starting with \\?\. So does os.mkdir in Python. This explains why this issue was not discovered before
• sysconfig defines default paths for Windows with / separator: https://github.com/python/cpython/blob/274769c61c5f3f3d2d749824cbed26e15813ce14/Lib/sysconfig.py#L49-L58
• These paths escape to file paths without change in
  

  pip/src/pip/_internal/operations/install/wheel.py

  Lines 458 to 461 in bd1563d

  	if wheel_root_is_purelib(metadata):
  	lib_dir = scheme.purelib
  	else:
  	lib_dir = scheme.platlib

  and later

I don't know why sysconfig uses / separator. If I change that to \\, this issue is solved. But if that's intentional, then this change would lead to other issues.

[uranusjr]

Looks like we could fix this by normalising the path? i.e. pathlib.Path(lib_dir). I don’t think sysconfig chose / intentionally; it is likely just for convenience since the slashes are expected to be normalised at some point anyway.

[grv87]

And one other error

 pip install --verbose pefile
Using pip 22.1.2 from C:\Users\<user>\AppData\Roaming\Python\Python310\site-packages\pip (python 3.10)
Collecting pefile
  Using cached pefile-2022.5.30.tar.gz (72 kB)
ERROR: Could not install packages due to an OSError.
Traceback (most recent call last):
  File "C:\Users\<user>\AppData\Roaming\Python\Python310\site-packages\pip\_internal\commands\install.py", line 341, in run
    requirement_set = resolver.resolve(
  File "C:\Users\<user>\AppData\Roaming\Python\Python310\site-packages\pip\_internal\resolution\resolvelib\resolver.py", line 94, in resolve
    result = self._result = resolver.resolve(
  File "C:\Users\<user>\AppData\Roaming\Python\Python310\site-packages\pip\_vendor\resolvelib\resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "C:\Users\<user>\AppData\Roaming\Python\Python310\site-packages\pip\_vendor\resolvelib\resolvers.py", line 348, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "C:\Users\<user>\AppData\Roaming\Python\Python310\site-packages\pip\_vendor\resolvelib\resolvers.py", line 172, in _add_to_criteria
    if not criterion.candidates:
  File "C:\Users\<user>\AppData\Roaming\Python\Python310\site-packages\pip\_vendor\resolvelib\structs.py", line 151, in __bool__
    return bool(self._sequence)
  File "C:\Users\<user>\AppData\Roaming\Python\Python310\site-packages\pip\_internal\resolution\resolvelib\found_candidates.py", line 155, in __bool__
    return any(self)
  File "C:\Users\<user>\AppData\Roaming\Python\Python310\site-packages\pip\_internal\resolution\resolvelib\found_candidates.py", line 143, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
  File "C:\Users\<user>\AppData\Roaming\Python\Python310\site-packages\pip\_internal\resolution\resolvelib\found_candidates.py", line 47, in _iter_built
    candidate = func()
  File "C:\Users\<user>\AppData\Roaming\Python\Python310\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 215, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "C:\Users\<user>\AppData\Roaming\Python\Python310\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 291, in __init__
    super().__init__(
  File "C:\Users\<user>\AppData\Roaming\Python\Python310\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 161, in __init__
    self.dist = self._prepare()
  File "C:\Users\<user>\AppData\Roaming\Python\Python310\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 230, in _prepare
    dist = self._prepare_distribution()
  File "C:\Users\<user>\AppData\Roaming\Python\Python310\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 302, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "C:\Users\<user>\AppData\Roaming\Python\Python310\site-packages\pip\_internal\operations\prepare.py", line 428, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "C:\Users\<user>\AppData\Roaming\Python\Python310\site-packages\pip\_internal\operations\prepare.py", line 473, in _prepare_linked_requirement
    local_file = unpack_url(
  File "C:\Users\<user>\AppData\Roaming\Python\Python310\site-packages\pip\_internal\operations\prepare.py", line 165, in unpack_url
    unpack_file(file.path, location, file.content_type)
  File "C:\Users\<user>\AppData\Roaming\Python\Python310\site-packages\pip\_internal\utils\unpacking.py", line 246, in unpack_file
    untar_file(filename, location)
  File "C:\Users\<user>\AppData\Roaming\Python\Python310\site-packages\pip\_internal\utils\unpacking.py", line 217, in untar_file
    with open(path, "wb") as destfp:
OSError: [Errno 22] Invalid argument: '\\\\?\\Volume{<guid>}\\<folder>\\pip-install-f2mz4q6x\\pefile_1e6cb812a1fa4017962551a08e0f55c0\\ordlookup/__init__.py'

The problem is here:

pip/src/pip/_internal/utils/unpacking.py

Line 180 in 11b0654

path = os.path.join(location, fn)

Neither pip not Python normalize the second argument (path coming from tar).

[grv87]

Looks like we could fix this by normalising the path?

@uranusjr, are you asking me?
I think that would work.
But I'm personally very unhappy with path handling in pip and Python. So, if it was me, I'd rewrite the whole pip to pathlib. To make sure that there are no corner cases left.

[grv87]

since the slashes are expected to be normalised at some point anyway

I don't see this expectation expressed anywhere in Python's os.path documentation. Also, I don't see any separate built-in function to normalize slashes. So, I would not consider this as decided, but explicitly ask Python's team.

[pfmoore]

To be honest, the main reason pip doesn't use pathlib throughout is purely because much of pip's code predates pathlib (and almost all of it was written before we stopped supporting versions of Python that didn't have pathlib). If anyone wanted to submit a series of PRs moving to pathlib, I'm fairly sure they would be accepted. But in the meantime, we have to deal with what we have...

And actually, it appears that pathlib is no better here:

>>> p = Path("\\\\C:\\Work")
>>> (p / "a").mkdir()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Gustav\AppData\Local\Programs\Python\Python310\lib\pathlib.py", line 1173, in mkdir
    self._accessor.mkdir(self, mode)
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: '\\\\C:\\Work\\a'
>>> p / "a"
WindowsPath('//C:/Work/a')

Given the behaviour here, I'd start by raising one or more issues with CPython, asking the following questions:

1. Should the quoted example I gave above work? Assuming it should, what is causing the error?
2. What is the correct way to join a \\?\-prefixed path to another path segment using the functions in os.path (those functions are still supported, so I don't consider "use pathlib" to be a valid answer here). In particular, in the case where the added segment uses / for the separator.
3. Given the complexities involved in normalising slashes, is there a good reason why sysconfig paths on Windows use / as the separator?

Depending on the answers to these questions, we can work out a way forward here.

Looking at using os.path.normpath as a short-term possibility, one thing that we'd need to be careful of is that os.path.normpath states that "This string manipulation may change the meaning of a path that contains symbolic links". So we probably shouldn't use that function to normalise the resulting path.

But we could probably get away with using os.path.normpath on the return value from sysconfig, before joining. That would be safe, as normalising a/./b or a/../b would likely not cause issues in that context. But I still wouldn't be comfortable asserting that there are no other issues with using a \\?\-prefixed path for --prefix - core Python's behaviour is simply not consistent enough to make such an assertion, it seems to me.

[grv87]

If anyone wanted to submit a series of PRs moving to pathlib, I'm fairly sure they would be accepted

🆗 Got it

it appears that pathlib is no better here

Looks that you missed a question mark.
With \\? prefix works for me. Or, it's already fixed in my local installation.

[pfmoore]

Looks that you missed a question mark

Doh. Thanks for catching that 🤦 It does indeed work for me then. So scrub point 1, pathlib works fine.

[6hundred9]

what the hell did you do to get that prefix, it (probably from where i remember) means windows marked it as a suspicious file

[uranusjr]

It is called a UNC path, nothing special.