Short paths in Windows seem to fail to be collected

[Erotemic]

This is an initial report of an issue I observed when updating xdoctest to support pytest 8.0.

What I found is that when I ran a test effectivly using the invocation:

C:\hostedtoolcache\windows\Python\3.12.1\x64\python.exe -m pytest C:\Users\RUNNER~1\AppData\Local\Temp\tmpcttcs8zz

Pytest would collect 0 tests even though there was a python file with a test in that directory.

By adding this code to convert the short path to a long path, the xdoctest failure went away, and things seem to be working now.

dpath = tempfile.mkdtemp()
# https://stackoverflow.com/questions/11420689/how-to-get-long-file-system-path-from-python-on-windows
from ctypes import create_unicode_buffer, windll
BUFFER_SIZE = 500
buffer = create_unicode_buffer(BUFFER_SIZE)
get_long_path_name = windll.kernel32.GetLongPathNameW
get_long_path_name(dpath, buffer, BUFFER_SIZE)
dpath = buffer.value

I've documented more of the problem here Erotemic/xdoctest#151

I don't have a MWE as I don't have a windows machine, but I'm reasonably confident that something in pytest 8 broke short path recognition on windows. I'll leave it to other devs to test further.

[bluetech]

Thanks for the report.

I don't have a MWE as I don't have a windows machine, but I'm reasonably confident that something in pytest 8 broke short path recognition on windows. I'll leave it to other devs to test further.

In order to narrow down on the issue, we'll probably need to bisect. I don't have windows either, but I can try with Wine. Can you provide instructions for reproducing the CI failure locally? (Doesn't have to be minimal).

[Erotemic]

I think this should work to reproduce it. On a window machine:

• clone the latest version of xdoctest: https://github.com/Erotemic/xdoctest/tree/v1.1.3

• pip install -e . to install it in development mode

• Apply the following patch to remove the workaround I added to convert short paths into long paths:

diff --git a/src/xdoctest/utils/util_path.py b/src/xdoctest/utils/util_path.py
index d49b371..83a0230 100644
--- a/src/xdoctest/utils/util_path.py
+++ b/src/xdoctest/utils/util_path.py
@@ -34,19 +34,8 @@ class TempDir(object):
 
     def ensure(self):
         import tempfile
-        import sys
         if not self.dpath:
             dpath = tempfile.mkdtemp()
-            if sys.platform.startswith('win32'):
-                # Force a long path
-                # References:
-                # https://stackoverflow.com/questions/11420689/how-to-get-long-file-system-path-from-python-on-windows
-                from ctypes import create_unicode_buffer, windll
-                BUFFER_SIZE = 500
-                buffer = create_unicode_buffer(BUFFER_SIZE)
-                get_long_path_name = windll.kernel32.GetLongPathNameW
-                get_long_path_name(dpath, buffer, BUFFER_SIZE)
-                dpath = buffer.value
             self.dpath = dpath
         return self.dpath

• install the version of pytest you want to test

• Run the tests that cause the error (relative to the xdoctest repo root):

pytest tests/test_pytest_cli.py -k test_simple_pytest_import_error_cli -s
pytest tests/test_pytest_cli.py -k test_simple_pytest_cli -s

You can likely make this much more minimal by using tempfile.mkdtemp() to make a directory, record the path that it returns (on the actions CLI it was a short path, so I expect it will be the same on a different machine). Then write a small test file that pytest should pickup to that directory and then run pytest <path> to that directory. That is effectively what the above xdoctest instructions are doing.

[nicoddemus]

You can likely make this much more minimal by using tempfile.mkdtemp() to make a directory, record the path that it returns (on the actions CLI it was a short path, so I expect it will be the same on a different machine). Then write a small test file that pytest should pickup to that directory and then run pytest to that directory. That is effectively what the above xdoctest instructions are doing.

I did not investigate into details, but I cannot reproduce the behavior using a very short path:

λ pwd
W:\

λ cat a\test_foo.py
def test(): pass

λ pytest w:\a
======================== test session starts ========================
platform win32 -- Python 3.10.9, pytest-8.0.0, pluggy-1.4.0
rootdir: w:\a
collected 1 item

a\test_foo.py .                                                [100%]

========================= 1 passed in 0.00s =========================

λ pytest a
======================== test session starts ========================
platform win32 -- Python 3.10.9, pytest-8.0.0, pluggy-1.4.0
rootdir: W:\a
collected 1 item

a\test_foo.py .                                                [100%]

========================= 1 passed in 0.00s =========================

I'm afraid we will need more details to reproduce the issue.

[Erotemic]

There's a misunderstanding on what I mean by "short path". It's not a path with a short length, it's a windows shorthand to refer to a real path that's longer. It's like an autocomlete. E.g.

c:\PROGRA~2\Android\ANDROI~1 resolves to c:\Program Files (x86)\Android\android-sdk

Reference: https://superuser.com/questions/348079/how-can-i-find-the-short-path-of-a-windows-directory-file

It's a silly feature, but it seems tempfile.mkdtemp is able to return one, so it probably makes sense to support it.

[nicoddemus]

Ahh of course, brain fart on my part.

I can reproduce the issue:

(.env310) λ dir c:\PORTAB~1\test\ /b
test_foo.py
__pycache__

>>> from pathlib import Path
>>> p=Path(r'c:\PORTAB~1\test')
>>> p.is_dir()
True
>>> (p / "test_foo.py").is_file()
True
>>>

λ pytest c:\PORTAB~1\test\
======================== test session starts ========================
platform win32 -- Python 3.10.10, pytest-8.1.0.dev108+gca3790102, pluggy-1.4.0
rootdir: e:\projects\pytest
plugins: hypothesis-6.84.0, replay-1.4.0
collected 0 items

======================= no tests ran in 0.02s =======================
ERROR: not found: c:\PORTAB~1\test
(no match in any of [<Dir >])

I will investigate this when I find some time, thanks for the report!

[bluetech]

I'm surprised this ever worked. Windows paths - the gift that keeps on giving :)