python pathname of the shebang in the console scripts in '/Scripts'

[return42]

• Pip version: any (tested with 9.0.1)
• Python version: any (tested with 3.6.1)
• Operating system: any (tested on MS-Win)

Question / Improvement?

When using pip under MS-Win, the shebang of the console scripts is always a absolute pathname. So far I can see, for shebang pip uses the value from:

import sys
sys.executable

Since this is always an absolute pathname, those console '.exe' scripts will not work in different (virtual) environments or in portable python setups. In POSIX like environments, it is very common to use a shebang with an env call:

#!/usr/bin/env python

Since there is no 'env' tool on windows, I expect a simple shebang wich does nearly the same, using first python interpreter from PATH:

#!python

My question is, why do we use absolute pathnames in shebang instead of using the first python from the PATH env?

What I've run (my workaround):

I wrote a small hackish pip.bat wrapper around my pip

python -c "import pip, sys;sys.executable='python.exe';pip.main()" %*

which replaces the absolute pathname with sys.executable='python.exe'. So I get portable console scripts with the #!python shebang.

But the question remains, why is it better to use absolute pathnames? I guess this is the same on POSIX systems, where pip also uses absolute pathnames instead of using /usr/bin/env.

Thanks!

Update 10/2018

See #4616 (comment) from @amr66, newer versions (18.1) of pip have moved the main function to pip._internal and you need:

 python -c "from pip._internal import main, sys;sys.executable='python.exe';main()" %*

[pfmoore]

Simple answer, because the scripts are only intended for use in the specific environment the package was installed in. If you install the package in another environment, a separate script for that environment is created. Making the scripts pick the currently active Python means that they will fail if that environment doesn't have the right dependencies installed.

Does that answer your question? If not, can you explain precisely what you are trying to do that is not possible because of the current script behaviour?

[pfmoore]

Simple answer, because the scripts are only intended for use in the specific environment the package was installed in

More precisely, you can use the scripts from anywhere, but they will always use the environment they were created in for their Python interpreter (because that's the environment known to have the correct packages installed).

[return42]

Thanks for your quick reply ..

Does that answer your question?

Yes, I see there are some drawbacks with python taken from PATH.

More precisely, you can use the scripts from anywhere, but they will always use the environment ..

I guess you know more use-cases than I. I thought only about virtualenv, where you have a 'activate/deactivate' script which selects the right environment.

... can you explain precisely what you are trying to do that is not possible because of the current script behaviour?

I setup a portable python with I can go to my customers and run some of my self-written maintenance and analyses tools. On most linux system I do not have problems running my py-scripts (py2/py3 compatible) .. but often on MS-Win a python is not installed and the server is placed in a very restricted array with no access to the internet. In short: no chance to install python. These are my use-case, where I need a portable python (in a ZIP) which I can copy to and run from in my users HOME folder.

To be more general; does it make sense to add an option to pip where the caller can chose if he wants absolute pathname in the shebang (default) or a shebang with a python from the env (PATH)?

[pfmoore]

OK, I see what you're trying to do. Portable python installations on Windows are tricky to get right (I don't think any version of Python before 3.6 gets things completely portable - although the corner cases where there are issues are pretty rare in practice, and likely not things you need to worry about). There are going to be some issues with wrapper scripts having "find Python on PATH" shebang lines, and I doubt it would be workable in general.

For pip, and many other packages that provide a CLI wrapper, you can simply use python -m pip - that's exactly equivalent to using the pip.exe wrapper, and avoids the need for any hacking of the wrapper. For a more general solution, your hack is probably OK. It's not supported, obviously, but I did a quick check of the pip sources and I can't see anything obvious that would cause it to fail.

[return42]

Portable python installations on Windows are tricky to get right

Yes, today it is tricky but for every task there is a starting point ;) For pure python without external C libraries it seems to work fine (I need much more practical experience).

For pip, and many other packages that provide a CLI wrapper ..

Right, that was my first attempt .. but after while more and more packages came into my portable python and I was looking for a more general solution.

For a more general solution, your hack is probably OK. It's not supported, obviously, but I did a quick check of the pip sources and I can't see anything obvious that would cause it to fail.

Thanks for your effort. I also looked at _vendor.distlib.utils.get_executable() and its history gives my the hope that this will not change in the near future ;)

@pfmoore thanks for clarifying .. if you know about other portable pure CPython efforts, please let me know .. I have seen e.g. WinPython, but this is only MS-Win and not pure CPython / not what I was looking for.

[pradyunsg]

@return42 - nice name - is your question resolved?

[return42]

Yes, my question is answered .. but resolved?

I understand why pip use absolute path names and I have a workaround. The cons is: my workaround is hakish and might break in the (near) future ..

The absolute pathname decision of pip does not fit for all scenarios. It would be nice to have a option to set the shebang line. E.g. --shebang=python which inserts the simple shebang #!python.

[DancingQuanta]

I have an issue with this when using Cygwin.

My current setup with Cygwin and Windows python relies on a bash script ~/bin/python, which is included in PATH, that converts the POSIX paths of python scripts to Windows paths and passes that to Windows python to execute the script.

if [[ -z $PYTHON ]]; then
  echo "Cygwin"
  /usr/bin/python $@
else
  ARGS=()
  for var in "$@"; do
    if [[ $var == -* ]]; then
      ARGS+=("$var")
    elif [[ -f $var ]]; then
      file="$(cygpath -w -a $var)"
      echo $file
      ARGS+=("$file")
    else
      ARGS+=("$var")
    fi
    echo "Var $var, Args${ARGS[@]}"
  done
  $PYTHON "${ARGS[@]}"
  echo "Done!"
fi

where $PYTHON is set as path to my Windows python interpreter.

When I install a python script via pip, the script might have a custom shebang that points to the Windows python interpreter.
This bypasses my ~/bin/python and so I am unable to execute this script through Cygwin.

Oh and I am using Anaconda which itself is installed in Windows and have no Cygwin equivalence.

So I like the idea of controlling the shebang during installations via pip.

[return42]

@DancingQuanta did you test my hackish pip install replacement:

python -c "import pip, sys;sys.executable='~/bin/python';pip.main()" install ...

does this workaround work for you?

[DancingQuanta]

No dice, the hack does not work for me

13:23:11 andre@LAPTOP-V9S562AR:~
$ ./Anaconda3/python.exe -c "import pip, sys, os; sys.executable='~/bin/python'; pip.main()" install yaml2json            Collecting yaml2json
  Using cached yaml2json-1.0.0.tar.gz
    Error [WinError 2] The system cannot find the file specified while executing command python setup.py egg_info
Exception:
Traceback (most recent call last):
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\commands\install.py", line 335, in run
    wb.build(autobuilding=True)
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\req\req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\req\req_set.py", line 634, in _prepare_file
    abstract_dist.prep_for_dist()
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\req\req_set.py", line 129, in prep_for_dist
    self.req_to_install.run_egg_info()
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\req\req_install.py", line 439, in run_egg_info
    command_desc='python setup.py egg_info')
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\utils\__init__.py", line 667, in call_subprocess
    cwd=cwd, env=env)
  File "C:\Users\andre\Anaconda3\lib\subprocess.py", line 676, in __init__
    restore_signals, start_new_session)
  File "C:\Users\andre\Anaconda3\lib\subprocess.py", line 955, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

What file it is trying to find? ~/bin/python? I tried expanding out '~' but this yields different errors

13:24:52 andre@LAPTOP-V9S562AR:~
$ ./Anaconda3/python.exe -c "import pip, sys, os; path = os.path.join(os.path.expanduser('~'), 'bin', 'python'); sys.executable=path; pip.main()" install yaml2json
Collecting yaml2json
  Using cached yaml2json-1.0.0.tar.gz
    Error [WinError 193] %1 is not a valid Win32 application while executing command python setup.py egg_info
Exception:
Traceback (most recent call last):
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\commands\install.py", line 335, in run
    wb.build(autobuilding=True)
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\req\req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\req\req_set.py", line 634, in _prepare_file
    abstract_dist.prep_for_dist()
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\req\req_set.py", line 129, in prep_for_dist
    self.req_to_install.run_egg_info()
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\req\req_install.py", line 439, in run_egg_info
    command_desc='python setup.py egg_info')
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\utils\__init__.py", line 667, in call_subprocess
    cwd=cwd, env=env)
  File "C:\Users\andre\Anaconda3\lib\subprocess.py", line 676, in __init__
    restore_signals, start_new_session)
  File "C:\Users\andre\Anaconda3\lib\subprocess.py", line 955, in _execute_child
    startupinfo)
OSError: [WinError 193] %1 is not a valid Win32 application

[return42]

@DancingQuanta :

OSError: [WinError 193] %1 is not a valid Win32 application

Your ~/bin/python is a shell script. I do not know if it will work at all.

Ok, now I understand what you try to do: You are on Win and you start a shell script, which starts a suitable Python interpreter as a child process of the shell script. On Win shell scripts are not executable, you need to call the bash .. puh ..

What do you think is the perfect shebang for this (I have no idea)?