bootstrap -> No module named pip

[jedie]

I created a minimal bootstrap with virtualenv v1.11.6
A run results in this:

$ python2 test-bootstrap.py ~/test_py2
extend_parser called.
adjust_options args: ['/home/jens/test_py2']
New python executable in /home/jens/test_py2/bin/python2
Not overwriting existing python script /home/jens/test_py2/bin/python (you must use /home/jens/test_py2/bin/python2)
Cannot find a wheel for setuptools
Cannot find a wheel for pip
Installing setuptools, pip...
  Complete output from command /home/jens/test_py2/bin/python2 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip:
  Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named pip
----------------------------------------
...Installing setuptools, pip...done.
Traceback (most recent call last):
  File "test-bootstrap.py", line 2355, in <module>
    main()
  File "test-bootstrap.py", line 826, in main
    symlink=options.symlink)
  File "test-bootstrap.py", line 994, in create_environment
    install_wheel(to_install, py_executable, search_dirs)
  File "test-bootstrap.py", line 962, in install_wheel
    'PIP_NO_INDEX': '1'
  File "test-bootstrap.py", line 904, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /home/jens/test_py2/bin/python2 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip failed with error code 1

All files are here: https://gist.github.com/jedie/4bc671239b8959376d4b

more info:

$ python2 --version
Python 2.7.6
$ python2 -c "import pip;print pip.__version__"
1.5.4

$ /home/jens/test_py2/bin/python --version
Python 2.7.6
$ /home/jens/test_py2/bin/python -c "import pip;print pip.__version__"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named pip

Maybe it's related to pypa/pip#1918 ?

[jedie]

btw. it works with python3:

$ python3 test-bootstrap.py ~/test_py3
extend_parser called.
adjust_options args: ['/home/jens/test_py3']
Using base prefix '/usr'
New python executable in /home/jens/test_py3/bin/python3
Also creating executable in /home/jens/test_py3/bin/python
Installing setuptools, pip...done.
after_install from '/home/jens/test_py3'

Some more info:

$ python3 -V
Python 3.4.0
$ python3 -m pip -V
pip 1.5.4 from /usr/lib/python3/dist-packages (python 3.4)

$ ~/test_py3/bin/python -V
Python 3.4.0
$ ~/test_py3/bin/python -m pip -V
pip 1.5.6 from /home/jens/test_py3/lib/python3.4/site-packages (python 3.4)

[jedie]

btw. i have python-pip and python-pip3 installed. Both are: 1.5.4-1 and the package python-virtualenv 1.11.4-1 was installed.

I have removed all pip/virtualenv packages and re-try again my test-bootstrap.

Now it will fail with python2 and python3 with the same error.

[jedie]

I created a fresh new venv with python3 and installed pip and virtualenv from github:

(test_venv) ~/test_venv $ bin/python -m virtualenv --version
12.0.dev1
(test_venv)  ~/test_venv $ bin/python -m pip -V
pip 6.0.dev1 from /home/jens/test_venv/lib/python3.4/site-packages (python 3.4)

...and recreate the bootstrap script and try it: Same "No module named pip" error.

[pfmoore]

Looks like your bootstrap script doesn't have the necessary virtualenv_support directory alongside it.

[jedie]

Can you explain a little bit more, what you mean?

My script is not really more, than from the docu here: https://virtualenv.pypa.io/en/latest/virtualenv.html#bootstrap-example

[pfmoore]

When you create a bootstrap script, it is essentially just a customised version of virtualenv.py. But as noted in https://virtualenv.pypa.io/en/latest/virtualenv.html (under "Installation", at the end of that section, the blue note), any virtualenv.py script (vanilla or customised via the bootstrap process) must " include a virtualenv_support directory alongside virtualenv.py which contains a complete set of pip and setuptools distributions".

So, wherever your bootstrap script is located, you need to create a directory called virtualenv_support next to it, and put wheels for pip and setuptools in there. These can either be from PyPI, or you can get the ones from your virtualenv installation.

[jedie]

Ah! Thanks for the Information!

That's really boring. So a single bootstrap file is a little bit useless, if other files must be downloaded before :(

Was that always been so?

EDIT: Seems that #632 is a similar request for a self contained virtualenv.py...

btw. i think the error message is not really good. Maybe a simple solution like the following is more helpfull, e.g.: :

try:
    import pip
except ImportError as err:
    sys.stderr.write("Please install pip fist, before bootstrap! (Origin error: %s)" % err)
    sys.exit(-1)

EDIT2: The "node: pip is needed" should be added in the https://virtualenv.pypa.io/en/latest/virtualenv.html#creating-your-own-bootstrap-scripts section, too.

[pfmoore]

It's been that way for a long while. virtualenv used to download pip and setuptools from the internet, but that was potentially insecure, often very slow, and unreliable (your internet goes down and you can't create virtualenvs). The change was deliberate, and while it does remove the "single file application" option, we viewed it as worth the cost.

[jedie]

What's about to include pip into virtualenv.py?

Or maybe better a separate "bootstrap" project, that's put everything together in one file?

[pfmoore]

I've been through this thought process a long time ago :-)

We use the pip wheel to install itself and setuptools. To do that we add the pip wheel to sys.path, using the fact that wheel files are actually zip files that can be added to sys.path (with some limitations, that don't apply here). Embedding the pip wheel into virtualenv wouldn't allow that.

Sorry...

[jedie]

One Question: It also doesn't work if pip is systemwide installed, isn't it?

On creating i see this:

Cannot find sdist setuptools-*.tar.gz
Cannot find sdist pip-*.tar.gz

$ /home/jens/testenv/bin/python -m pip -V
/home/jens/testenv/bin/python: No module named pip

So my added info ( jedie/PyLucid@dd3b5db ) is simply wrong :(

I really must have setuptools and pip as sdist ?

The a installation will result in this:

1. create a virtualenv_support directory
2. download setuptools sdist file and store it in virtualenv_support
3. download pip sdist file and store it in virtualenv_support
4. download bootstrap file
5. run bootstrap

That much to do, isn't it?

One reason to remove the functionality was that it's potentially insecure...
But now, the user must download setuptools and pip and has to check the integrity by hand, isn't it?
So, he must do this:

1. create a virtualenv_support directory
2. download setuptools sdist file and store it in virtualenv_support
3. verify download
4. download pip sdist file and store it in virtualenv_support
5. verify download
6. download bootstrap file
7. verify download
8. run bootstrap

I'm wrong?

[pfmoore]

setuptools and pip as wheels not sdists, but yes.

You manually download over https (which is secure). The previous implementation used http (which is not).

And why not just use the virtualenv_support that is installed with your virtualenv distribution? Surely that's easier than downloading, if that's what matters to you?