shutil.which() can return non-executable path starting with 3.12 on Windows

[lazka]

Bug report

Bug description:

Starting with Python 3.12 if a file is in PATH that does not end with PATHEXT, but happens to be named the same as the searched command it is returned instead of the real command later in PATH which includes PATHEXT.

# Assume:
# PATH=C:\foo;C:\WINDOWS\system32
# "C:\foo\cmd" exists
# "C:\WINDOWS\system32\cmd.exe" exists

import shutil
print(shutil.which('cmd'))
# Actual: C:\foo\cmd
# Expected: C:\WINDOWS\system32\cmd.exe (3.11 result)

I have verified that this change was introduced with #103179 and reverting it fixes the issue.

@csm10495 ^

Downstream context: mesonbuild/meson#13886

CPython versions tested on:

3.12

Operating systems tested on:

Windows

Linked PRs

• gh-127001: Fix PATHEXT issues in shutil.which() on Windows #127035
• [3.13] gh-127001: Fix PATHEXT issues in shutil.which() on Windows (GH-127035) #127156
• [3.12] gh-127001: Fix PATHEXT issues in shutil.which() on Windows (GH-127035) #127158

[zooba]

Reverting the entire change is clearly not going to happen - the logic was incorrect before, and apparently is still incorrect, so let's figure out what correct logic for handling PATHEXT would be.

[serhiy-storchaka]

Should not it be fixed by #109590?

[serhiy-storchaka]

Ah, it still checks for file without extension.

[lazka]

Should not it be fixed by #109590?

That only handles the "in the same directory" case, and not in different directories, from what I see.

But, that issue mentioned Get-Command cmd in powershell, which I now tried, and that also returns the wrong path as "CommandType=Application" here, even though the docs say that it only returns files ending with PATHEXT (https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/get-command?view=powershell-7.4#-commandtype) so I don't know what's "right" anymore :)

PS C:\Users\user> echo $env:PATH
C:\msys64\usr\bin;C:\Windows\System32;C:\Windows
PS C:\Users\user> echo $env:PATHEXT
.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.CPL
PS C:\Users\user> Get-Command cmd

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     cmd                                                0.0.0.0    C:\msys64\usr\bin\cmd

PS C:\Users\user> Get-Command -All cmd

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     cmd                                                0.0.0.0    C:\msys64\usr\bin\cmd
Application     cmd.exe                                            10.0.2610… C:\Windows\System32\cmd.exe

[zooba]

The right behaviour ought to be "whatever CreateProcess("cmd") would run". Terminals all layer their own logic on top, but Python tries to generally stick to the closest public API that implements the functionality.

(One explicit exception we make to CreateProcess behaviour is that we don't search the application directory first with which, because if we did, it wouldn't be useful. By skipping that somewhat strange edge case, it is possible to use which to override the application directory. This mainly only matters for python.exe - CreateProcess("python") will always launch sys._base_executable due to how the API works, but CreateProcess(shutil.which("python")) follows PATH, which is likely what was intended if a user provided the command name.)