`py ./x.py` no longer works on Windows

[CAD97]

Workaround for Windows developers

Put the following as py.ini in %LOCALAPPDATA%:

[commands]
bash=python

This will make py interpret the #!/usr/bin/env bash shebang as running python.

You can also put py.ini alongside py.exe. This is likely C:\Windows; you can find it with where py_cmd or Get-Command py_pwsh.

Alternatively, run x.py with py -3 to explicitly select to use the default Python 3 version rather than looking at the shebang. You should also be able to set ftype Python.File="C:\Windows\py.exe" "-3" "%L" %* so ./x.py will use py -3 to run.

To debug, set $env:PYLAUNCH_DEBUG=1_pwsh/set PYLAUNCH_DEBUG=1_cmd.

---

Issue description

#98474 added a #!/usr/bin/env bash shebang line to x.py to allow ./x.py to work on unixy platforms with python3 but not python (the previous shebang line was #!/usr/bin/env python).

This means that x.py no longer works on Windows machines which run python via the py Python Launcher for Windows, as py tries to interpret the shebang line (in order to use the correct python version) and ends up instead trying to run /usr/bin/env, which obviously doesn't exist on a Windows machine. Explicitly asking for a specific version from py (e.g. by running py -3 for the latest 3.x version) will successfully run the script, as then py ignores the shebang line.

In theory this is an upstream bug; py should not be trying to run something that isn't python. In practice, though, this is how most Windows users will be launching python, as this is the launcher set up to handle .py files when installing python from the official installer from the website or Python.Python.3 in winget. (The msstore python which the default-installed python and python3 shims point to reportedly do not install the py launcher.) Additionally, this worked fine previously, and only broke with #98474.

#71818 (comment) reports that #!/usr/bin/env python didn't work on Windows; I believe this is due to the order of events used in the described testing. @Walther first installed the msstore python. They then uninstalled the msstore python and installed the official python distribution, with "add python 3.10 to PATH" unchecked, reporting it to be unchecked by default. However, it was my experience that the official distribution did add python to the path by default; I suspect that the reason it didn't work for Walther was because (parts of) the msstore python was still in PATH, and the installer did not want to overwrite an existing installation. With my installation, a python and python3 shebang both work, and a python2 shebang results in a "Requested Python version (2) is not installed".

Changing the shebang to python3 has previously been delayed due to concerns about Windows compatibility. The specific situation seems to be this:

• When python is handled through the py.exe launcher (i.e. the shell uses the ftype registered handler), all python virtual commands will seamlessly work on Windows.
• However, when using a MSYS-like shell (e.g. Git Bash) which directly interprets shebangs, it will try to run the python3 program, which most likely does not exist on a Windows machine. (E.g. in my winget install of Python.Python.3, I have python.exe and python3.dll in path, but no python3.exe.)

[Rageking8]

@rustbot label C-bug

[CAD97]

Since this is also relevant, from the Using Python on Windows docs:

There are a number of different installers available for Windows, each with certain benefits and downsides.

The full installer contains all components and is the best option for developers using Python for any kind of project.

The Microsoft Store package is a simple installation of Python that is suitable for running scripts and packages, and using IDLE or other development environments. It requires Windows 10, but can be safely installed without corrupting other programs. It also provides many convenient commands for launching Python and its tools.

[...]

The Microsoft Store package is an easily installable Python interpreter that is intended mainly for interactive use, for example, by students.

As much as MS would like to push the msstore version of python, the full installer is still generally preferred for developers.

(Also, thanks, I used the blank template because I wasn't sure what to use for build tooling.)

[Rageking8]

@rustbot label A-rustbuild

[dtolnay]

While this is being shaken out in python/cpython#94399 (I agree with python/cpython#94399 (comment) — ideally this would be at most a warning, and py should interpret the file as if there were no shebang, which I believe already works):

Is there a different possible entrypoint with better availability across systems? python/cpython#94399 (comment) explains that py would look for it inside of "%SystemRoot%\System32" or "%SystemRoot%\SysWOW64". Perhaps perl? @CAD97 do you notice anything promising-looking in those directories that might exist on most platforms in a default installation?

[dtolnay]

From the discussion on the cpython issue it sounds like this is already fixed in the upcoming Python 3.11's py launcher. The release is scheduled for October 2022.

We can decide whether to revert our x.py change until October. I am hesitant because it would involve re-breaking the Linux and macOS users, while Linux distributions (since 2020) and macOS upgrades (since October 2021) are deleting python.

[CAD97]

§ 4.8.4.1 Customization via INI files provides a workaround: py.ini can be used to specify that a bash shebang should instead run {insert executable}. Unfortunately for us at least, py.ini is loaded from %LOCALAPPDATA% and the same directory py.exe is in (e.g. C:\Windows with a default install), so we can't just drop a py.ini next to x.py to fix the behavior of py.

[commands]
bash=python

From the discussion on the cpython issue it sounds like this is already fixed in the upcoming Python 3.11's py launcher.

I'm not 100% sure whether this is considered a feature or a bug in the new launcher at the moment, though, given the comment about not changing the old launcher's behavior, even though they said the no-env-lookup behavior might make sense in the new launcher...

We can decide whether to revert our x.py change until October.

A different alternative would be to change the shebang to python3 in the interim; this was previously rejected as it was thought to break Windows, but in theory it shouldn't¹. This would instead break old unixy users who were partially broken by only having python2 available, as well as MSYS users who do not have a python3 available (but imho if you're running a unixy environment, you should probably have python3, or other things will probably cause issues as well.)

do you notice anything promising-looking in those directories that might exist on most platforms in a default installation?

Unless cmd exists on unixy systems... probably not. (My system is fairly clean; I did a clean install of Win11 around the start of summer.)

A list of maybe-potentially-useful-but-probably-not executables I spotted scrolling through:

Yes this list includes pretty obviously non-useful commands. I picked basically anything with even a vaguely not-windows-specific name in System32.

• cmd
• curl
• help
• ipconfig
• mode
• more
• notepad
• nslookup
• perfmon
• ping
• print
• rdpshell
• rdpsign
• replace
• reset
• route
• runas
• runonce
• shutdown 🙃
• sort
• timeout
• where
• whoami
• wsl (note: installer. I don't have WSL installed on this machine yet, otherwise I'd probably have a bash available)
• xcopy

Footnotes

1. As I uninstalled the python3 shim to msstore, I can't test to see if py would detect and try to launch that instead of an actual python interpreter. py will attempt to use msstore-installed python if the full installer python is not available, so it's possible the shim would be selected. ↩

[wesleywiser]

I'm probably missing something obvious (I have basically 0 Python experience), but why can't we just document that Windows users do python x.py ...? That works both before and after this change.

[ChrisDenton]

If you use the python.org installer with the default options then it won't add python to the path. That said, it is just a tick box to select it.

[ChrisDenton]

Btw, I feel like I mentioned the failure mode in the OP on the original PR but perhaps I wasn't clear enough.