Scripts installed with wrong shebang under subprocess

[jaraco]

On Python 3.5.1 but not on 2.7.10, when I invoke pip through a subprocess in a virtualenv, the scripts are created with the wrong shebang and thus fail to run:

$ cat > invoke.py
import sys
import subprocess

subprocess.Popen(sys.argv[1:]).wait()
$ rm -Rf env
$ python -m virtualenv --version
14.0.0
$ python --version
Python 3.5.1
$ python -m virtualenv env
Using base prefix '/Library/Frameworks/Python.framework/Versions/3.5'
New python executable in /Users/jaraco/Dropbox/code/public/aspen/env/bin/python3
Also creating executable in /Users/jaraco/Dropbox/code/public/aspen/env/bin/python
Installing setuptools, pip, wheel...done.
$ python invoke.py env/bin/pip install pytest
Collecting pytest
  Using cached pytest-2.8.5-py2.py3-none-any.whl
Collecting py>=1.4.29 (from pytest)
  Using cached py-1.4.31-py2.py3-none-any.whl
Installing collected packages: py, pytest
Successfully installed py-1.4.31 pytest-2.8.5
$ head -n 1 env/bin/py.test
#!/Library/Frameworks/Python.framework/Versions/3.5/bin/python3

One would expect the py.test script to have a shebang with 'env' in it.

Using pyvenv instead is not subject to the issue. Removing the __PYVENV_LAUNCHER__ environment variable before launching the subprocess works around the issue, as reported here. The issue was observed with 13.1.2 and 14.0.0.

Is this behavior expected? If so, is removing the environment variable the appropriate thing for a parent process to do to work around the issue?

[Ivoz]

does

$ env/bin/pip uninstall pytest
$ env/bin/pip install pytest
$ head -n 1 env/bin/py.test

Give same, or different results?

And same question with

$ env/bin/python -m pip install pytest

?

[Ivoz]

Possibly a regression somehow on #322 / #541

In which case, could you test whether

subprocess.Popen(sys.argv[1:], env={}).wait()

helps or not?

[Ivoz]

Hmm, there was code in pip / distlib to perhaps deal with this, but it got commented out a year ago-

https://github.com/pypa/pip/blob/7cea748d0fb2e8980c2676304607426585550b41/pip/_vendor/distlib/util.py#L157-L165

[jaraco]

Give same, or different results?

In both cases, invoking pip directly results in a proper shebang:

$ env/bin/pip install pytest
Collecting pytest
  Using cached pytest-2.8.5-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): py>=1.4.29 in ./env/lib/python3.5/site-packages (from pytest)
Installing collected packages: pytest
Successfully installed pytest-2.8.5
$ head -n 1 env/bin/py.test
#!/Users/jaraco/Dropbox/code/public/aspen/env/bin/python3
$ env/bin/pip uninstall -y pytest
Uninstalling pytest-2.8.5:
  Successfully uninstalled pytest-2.8.5
$ env/bin/python -m pip install pytest
Collecting pytest
  Using cached pytest-2.8.5-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): py>=1.4.29 in ./env/lib/python3.5/site-packages (from pytest)
Installing collected packages: pytest
Successfully installed pytest-2.8.5
$ head -n 1 env/bin/py.test
#!/Users/jaraco/Dropbox/code/public/aspen/env/bin/python

Also, invoking with an empty env works:

$ cat invoke.py 
import sys
import os
import subprocess

env = dict(os.environ)
env.pop('__PYVENV_LAUNCHER__')
env = {}

subprocess.Popen(sys.argv[1:], env=env).wait()
$ python -m virtualenv env
Using base prefix '/Library/Frameworks/Python.framework/Versions/3.5'
New python executable in /Users/jaraco/Dropbox/code/public/aspen/env/bin/python3
Also creating executable in /Users/jaraco/Dropbox/code/public/aspen/env/bin/python
Installing setuptools, pip, wheel...done.
$ python invoke.py env/bin/pip install pytest
Collecting pytest
  Using cached pytest-2.8.5-py2.py3-none-any.whl
Collecting py>=1.4.29 (from pytest)
  Using cached py-1.4.31-py2.py3-none-any.whl
Installing collected packages: py, pytest
Successfully installed py-1.4.31 pytest-2.8.5
$ head -n 1 env/bin/py.test
#!/Users/jaraco/Dropbox/code/public/aspen/env/bin/python3

[jaraco]

According to the commit, that workaround was commented with inclusion of distlib 0.2.0 released in pip 6.0.

[jaraco]

That commit references pip 2031.

[jaraco]

Also, I've installed Python from the python.org Mac installer, not through homebrew or other methods.

[jaraco]

The code was removed from distlib in this commit, which provides little rationale for the change.

[Ivoz]

I guess it's to be "expected" given that your initial $ python is setting __VENV_LAUNCHER__ and thereafter fooling other scripts / executables. Unfortunately I don't have OS X to debug how exactly that process of "fooling" goes on.

Also could be similar / same issue as was reported in #620

@jaraco it could be worth modifying a pip in a virtualenv to uncomment the mentioned distlib lines and seeing if that seems to fix pip's behaviour of picking up the wrong python to write

[jaraco]

I traced that venv detection code back to this commit, which unfortunately doesn't elucidate the origins of the idea.

[jaraco]

Seems nobody knows the meaning of this env var.

[jaraco]

I've found this explanation.

[jaraco]

My inclination was similar to what Ronald suggested here.

[Ivoz]

@vsajip could you give any advice?

[vsajip]

Some thoughts:

• __PYVENV_LAUNCHER__ was added because for earlier versions of Python (< 3.3, according to Ronald) but not for later versions, Python's site.py on OS X needed to distinguish the location of stub launchers from framework executables. According to this comment by Ned Deily, sys.executable now points to the stub launcher, and this is why distlib stopped using __PYVENV_LAUNCHER__. The only usage of it now is in site.py - if available, on OS X, it's used to locate the pyvenv.cfg file.
• It may be worth checking to see what happens in e.g. Python 3.4, to see if's some sort of regression, or whether all Python >= 3.3 has the same behaviour.
• My guess is that something, somewhere, is doing an os.path.realpath() call when it shouldn't be. It doesn't look like it's distlib doing that. If (as per Ned's comment) sys.executable and the env var are always the same, then it shouldn't matter; but if something does a realpath on a sys.executable and executes Python through the result, then that would result in an unexpected shebang (because sys.executable would then be a dereferenced path).