Document shebang recipe execution in readme

[casey]

No description provided.

[pfmoore]

Thanks for documenting this. There is still a problem on Windows, which probably needs at least documenting, and might need an actual fix.

Specifically, if the command in the shebang line, when run on the temporary file, also interprets the shebang line, things get complicated. This is a real issue for Python, as the python command is not on PATH by default, and the recommended command to use is py (the "Python launcher for Windows"), which parses and interprets shebang lines.

The simplest fix for this would probably be to omit the shebang line when writing the temporary file on Windows (as it's already been interpreted by just). I can't think of any example where this would break anything, but it is a behaviour change, so I can understand if you had reservations. Is this something you'd consider?

The alternative is probably to add a note specifically on how to work around all this for Python. But at the moment, while I have a solution that works for me, I'm not sure it works reliably for older versions of Python, and there's a risk it might fail rather drastically (creating an infinite loop running more and more processes, ultimately crashing the user's PC!) An added complication here is that the reliable solutions for the launcher use /usr/bin/env in the shebang - but just doesn't support than unless Cygwin is available. So it would take some work to get a good solution here.

[casey]

Ah, I didn't realize that there was another issue beyond just documenting what just was doing.

So the issue is that if you use py, windows will parse the shebang line, and go into an infinite loop?

[pfmoore]

Ah, I didn't realize that there was another issue beyond just documenting what just was doing.

Nor did I until I had a discussion with the Python developers 🙂

So the issue is that if you use py, windows will parse the shebang line, and go into an infinite loop?

Windows doesn't parse the shebang line, the py executable supplied with Python does. This is why it's only an issue for Python. The rules just uses are different from the rules py uses¹, so the interactions are, to be polite, weird. I believe #!py -3 successfully navigates between the two, and runs the script using Python 3, but I've not tried all the variations.

The new documentation is fine, in that it documents what's going on and the user can reason from that. So if you wanted to leave it at that, I wouldn't argue. As a practical point, though, in the same way that you have a section "Safer Bash Shebang Recipes" it might be nice to have a section "Python Shebang Recipes on Windows". I'd be willing to help write that section, but to be honest it could be skipped, and the behaviour would be much easier to understand, if you simply didn't write the shebang line to the temporary file, as I suggested. From a quick look at the source, it might be as simple as changing include_shebang_line to always return true on Windows, rather than just for cmd.

Footnotes

1. And to confuse things further, Python 3.11 ships with a rewritten version of py, which has slightly different rules. ↩

[casey]

Okay, got it, thanks for clarifying. That makes sense, I just opened #1417, which doesn't include shebang lines on windows at all. I think this is such a minor detail that it's safe to change. In the long run, I'd like to add a recipe annotation [script] (or something, name TBD) which doesn't use shebangs to execute. It uses sh by default, can be changed with a setting, or can be set per recipe with something like [script("python3")]. This issue has some details #1413. I'm now leaning towards not doing the other things in that issue, and just sticking to changing how they're executed, so that issue is out of date.

[runeimp]

I don't know why py.exe exists. Whenever I use Python on windows I access python.exe directly and that works as expected every time. When I use py.exe it almost never works as expected and seems to only exist to shorten the call name and be some sort of bridge to use from the GUI part of Windows. Does anyone actually benefit from the existence of py.exe?

[pfmoore]

This is off topic, but people with multiple versions of python installed. Plus, python is not added to PATH by default (and before you say “just add it”, that has issues in a per-user install of python).

[runeimp]

While it might have been a bit antagonistic it was not off topic. We are discussing a fix for which there is only a single scripting language (that I know of) that requires such special consideration. And under many circumstances is not even required, if you just target python.exe instead of py.exe for instance. I avoid Windows as much as possible except for gaming and verification that any of my cross-platform apps work on it. So I've not come across a situation myself that is ever fixed by py.exe instead of just targeting python.exe. I also never install multiple versions of Python on Windows. Which is why I asked the question...

Does anyone actually benefit from the existence of py.exe?

I did a quick search on it a few years ago and could find no compelling use case that effected me in the least. But did find lots of issues caused by it. Since it's been part of Python for years (a decade or more, no?) I guessed there must be some need. I've yet to hear one that is truly compelling though. But I will look into the multiple versions of Python issue again. I fully accept I may have overlooked something in the past that should have been obvious. Maybe I'll find it this time.

[pfmoore]

Fair point. The crucial issue here is that #!python won't work because by default the python executable is not on PATH. Certainly users can add it to PATH, but that has other implications and issues (which have been debated endlessly in the past on various python lists, but ultimately they are the reason the Python installer doesn't make adding python to PATH the default).

[melMass]

I know it's an old PR sorry, but I think this page should also provide an example, I just now after reading a tail of issue seem to understand that it expects the full binary path? (talking about windows)

EDIT: Oh wow even cooler, just setting the name of the binary if it's in PATH like:

list-examples:
    #!nu
    let examples = ("{{libs}}" | lines | each {|l| ls $"libs/($l)" | get name | find examples } | flatten | each {|e| ls $e | get name} | flatten | input list)
    print $examples

[runeimp]

@melMass shebang isn't supported on Windows natively. It is a Unix standard that all Unix variants (BSDi, Linux, macOS, etc.) supports. Casey added an emulation to make it work on Windows that doesn't actually follow the standard. But it does so in a way that I wish the standard allowed as well. In any case you don't have to specify the path. But that is only true on Windows in Just. Anywhere the path is supported natively your stuck having to use the full path or something like #!/usr/bin/env nu to get the same flexibility. 😉

[melMass]

@runeimp thanks, yes and I like the idea, but to me it's not clear how to use from just the doc, I got it from reading all related issues. I would simply add an example and maybe a small line like, "either a binary in PATH or its full path"

[pfmoore]

It seems to me that "invokes the split command and arguments" (in the windows-specific behaviour part of this chapter is fairly clear - in #! py -3.10 the "split command and arguments" is py -3.10 and invoking it means running it like any other command (which would do a path search). But maybe that's only obvious to Windows users?

An extra example/comment in the docs seems like it would be fine, although it would be (IMO) quite hard to strike a balance between including something simple, and not over-complicating things by discussing all the edge cases. (To give an explicit example, "either a binary in PATH or its full path" is still incorrect, as .bat files are acceptable, as are relative paths).

[melMass]

Not sure what you mean I don't see a code example and it's the reason of my first comment. I did not want to start a debate, just give an insight from my perspective as a just user.

[pfmoore]

Something like this?

For example, on Windows, if a recipe starts with #! py, the final command the OS runs will be something like py C:\Temp\PATH_TO_SAVED_RECIPE_BODY"

That seems like a reasonable thing to add (to the windows-specific section). I've created #1709 which the maintainers can merge if they agree.