Upgrading pip fails in Python Virtual Environment

[pcgeek86]

Problem

I am running Visual Studio 2015 RTM with Python Tools for Visual Studio (PTVS) on a fresh install (July 29th) of Windows 10 Enterprise RTM (Build 10240). I create a new Python Application project, add a new Virtual Environment to it, and then try to install a Python library (eg. sqlite3) into the Virtual Environment. I get a warning message, saying that I should upgrade pip from 6.0.8 to the latest available version. When I attempt to do so, I get an error message (see below).

NOTE: My non-virtual environment is successfully upgraded to 7.1.2.

Virtual environment is being created at 'C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env'
Virtual environment was successfully created at 'C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env'
----- Installing 'pip' -----
You are using pip version 6.0.8, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-7.1.2-py2.py3-none-any.whl#md5=5ff9fec0be479e4e36df467556deed4d
  Using cached pip-7.1.2-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 6.0.8
    Uninstalling pip-6.0.8:
      Exception:
      Traceback (most recent call last):
        File "C:\Python34\lib\shutil.py", line 523, in move
          os.rename(src, real_dst)
      FileNotFoundError: [WinError 3] The system cannot find the path specified: 'c:\\users\\trevorsullivan\\source\\repos\\pythonapplication11\\pythonapplication11\\env\\lib\\site-packages\\pip\\_vendor\\requests\\packages\\urllib3\\packages\\ssl_match_hostname\\__pycache__\\_implementation.cpython-34.pyc' -> 'C:\\Users\\TREVOR~1\\AppData\\Local\\Temp\\pip-h64zdfhc-uninstall\\users\\trevorsullivan\\source\\repos\\pythonapplication11\\pythonapplication11\\env\\lib\\site-packages\\pip\\_vendor\\requests\\packages\\urllib3\\packages\\ssl_match_hostname\\__pycache__\\_implementation.cpython-34.pyc'

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env\lib\site-packages\pip\basecommand.py", line 232, in main
          status = self.run(options, args)
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env\lib\site-packages\pip\commands\install.py", line 347, in run
          root=options.root_path,
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env\lib\site-packages\pip\req\req_set.py", line 543, in install
          requirement.uninstall(auto_confirm=True)
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env\lib\site-packages\pip\req\req_install.py", line 667, in uninstall
          paths_to_remove.remove(auto_confirm)
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env\lib\site-packages\pip\req\req_uninstall.py", line 126, in remove
          renames(path, new_path)
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env\lib\site-packages\pip\utils\__init__.py", line 316, in renames
          shutil.move(old, new)
        File "C:\Python34\lib\shutil.py", line 535, in move
          copy2(src, real_dst)
        File "C:\Python34\lib\shutil.py", line 245, in copy2
          copyfile(src, dst, follow_symlinks=follow_symlinks)
        File "C:\Python34\lib\shutil.py", line 109, in copyfile
          with open(dst, 'wb') as fdst:
      FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\TREVOR~1\\AppData\\Local\\Temp\\pip-h64zdfhc-uninstall\\users\\trevorsullivan\\source\\repos\\pythonapplication11\\pythonapplication11\\env\\lib\\site-packages\\pip\\_vendor\\requests\\packages\\urllib3\\packages\\ssl_match_hostname\\__pycache__\\_implementation.cpython-34.pyc'
----- Failed to install 'pip' -----

After this array of errors occurs, future attempts to upgrade pip in the Virtual Environment result in the following output.

----- Installing 'pip' -----
C:\Users\TrevorSullivan\Source\Repos\PythonApplication10\PythonApplication10\env\Scripts\python.exe: No module named pip.__main__; 'pip' is a package and cannot be directly executed
----- Failed to install 'pip' -----

Repro Steps

1. Open VS 2015 RTM.
2. Create a new Python Application project.
3. Add a Virtual Environment.
4. Attempt to upgrade pip inside the Virtual Environment.

Expected Result

My current expectation would be that the pip Python module would be successfully upgraded from version 6.0.8 to the latest version available inside the Virtual Environment.

Cheers,
Trevor Sullivan
Microsoft MVP: PowerShell
http://trevorsullivan.net
http://twitter.com/pcgeek86

[zooba]

Thanks for the detailed report. It looks like pip is hitting Window's MAX_PATH limit with some of its temporary build files. I'll file an issue on their tracker, since they really shouldn't need to include the full path as a subdirectory of the temp directory.

You should be able to fix this by deleting the pip directories from your virtual environment's Lib\site-packages folder, and then installing it through PTVS (you should get a link in the Environments window when pip is not there).

[pcgeek86]

@zooba Steve, thanks for your detailed response and root cause analysis. I'm not sure I clearly understand your suggested workaround though. I'm already trying to install pip through PTVS.

When you delete pip from the Virtual Environment, pip is gone, so it can't install / upgrade itself. Should I be using easy_install somehow, instead?

----- Installing 'pip' -----
C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env\Scripts\python.exe: No module named pip
----- Failed to install 'pip' -----

[zooba]

We include a separate script that will install pip if you don't have it. There's a link in the Environments window pip section to trigger it when you don't have pip installed at all.

[pcgeek86]

@zooba That link disappears when I delete pip from the virtual environment directory, per your suggestion.

[zooba]

@pcgeek86 I don't remember exactly what the caching behavior here is, but you may need to close/reopen the project or VS after doing the delete. (Because we use Python to find pip, it's fairly expensive to keep querying repeatedly, so we don't do it.)

[pcgeek86]

@zooba I closed and re-opened the Visual Studio process and the link re-appeared. I now have pip 7.1.2 running in the virtual environment. However, because I ripped out pip 6.0.8 from the filesystem directly, rather than through the tooling, it complained about pip missing.

----- Installing pip -----
Installing with ensurepip
Ignoring indexes: https://pypi.python.org/simple
Requirement already up-to-date: setuptools in c:\users\trevorsullivan\source\repos\pythonapplication12\pythonapplication12\env\lib\site-packages
Collecting pip
Installing collected packages: pip
Successfully installed pip-6.0.8
You are using pip version 6.0.8, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-7.1.2-py2.py3-none-any.whl#md5=5ff9fec0be479e4e36df467556deed4d
  Using cached pip-7.1.2-py2.py3-none-any.whl
Collecting setuptools from https://pypi.python.org/packages/3.4/s/setuptools/setuptools-18.2-py2.py3-none-any.whl#md5=2b79c46f12ca11fca3720dcf0f327086
  Downloading setuptools-18.2-py2.py3-none-any.whl (461kB)
Collecting wheel
  Downloading wheel-0.24.0-py2.py3-none-any.whl (63kB)
Installing collected packages: wheel, setuptools, pip
  Found existing installation: setuptools 12.0.5
    Uninstalling setuptools-12.0.5:
      Successfully uninstalled setuptools-12.0.5
  Found existing installation: pip 6.0.8
    Uninstalling pip-6.0.8:
      Exception:
      Traceback (most recent call last):
        File "C:\Python34\lib\shutil.py", line 523, in move
          os.rename(src, real_dst)
      FileNotFoundError: [WinError 3] The system cannot find the path specified: 'c:\\users\\trevorsullivan\\source\\repos\\pythonapplication12\\pythonapplication12\\env\\lib\\site-packages\\pip\\_vendor\\requests\\packages\\urllib3\\packages\\ssl_match_hostname\\__pycache__\\_implementation.cpython-34.pyc' -> 'C:\\Users\\TREVOR~1\\AppData\\Local\\Temp\\pip-u24ad31b-uninstall\\users\\trevorsullivan\\source\\repos\\pythonapplication12\\pythonapplication12\\env\\lib\\site-packages\\pip\\_vendor\\requests\\packages\\urllib3\\packages\\ssl_match_hostname\\__pycache__\\_implementation.cpython-34.pyc'

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication12\PythonApplication12\env\lib\site-packages\pip\basecommand.py", line 232, in main
          status = self.run(options, args)
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication12\PythonApplication12\env\lib\site-packages\pip\commands\install.py", line 347, in run
          root=options.root_path,
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication12\PythonApplication12\env\lib\site-packages\pip\req\req_set.py", line 543, in install
          requirement.uninstall(auto_confirm=True)
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication12\PythonApplication12\env\lib\site-packages\pip\req\req_install.py", line 667, in uninstall
          paths_to_remove.remove(auto_confirm)
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication12\PythonApplication12\env\lib\site-packages\pip\req\req_uninstall.py", line 126, in remove
          renames(path, new_path)
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication12\PythonApplication12\env\lib\site-packages\pip\utils\__init__.py", line 316, in renames
          shutil.move(old, new)
        File "C:\Python34\lib\shutil.py", line 535, in move
          copy2(src, real_dst)
        File "C:\Python34\lib\shutil.py", line 245, in copy2
          copyfile(src, dst, follow_symlinks=follow_symlinks)
        File "C:\Python34\lib\shutil.py", line 109, in copyfile
          with open(dst, 'wb') as fdst:
      FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\TREVOR~1\\AppData\\Local\\Temp\\pip-u24ad31b-uninstall\\users\\trevorsullivan\\source\\repos\\pythonapplication12\\pythonapplication12\\env\\lib\\site-packages\\pip\\_vendor\\requests\\packages\\urllib3\\packages\\ssl_match_hostname\\__pycache__\\_implementation.cpython-34.pyc'
Failed to upgrade pip, which probably indicates that it isn't installed properly.
Downloading pip from https://go.microsoft.com/fwlink/?LinkId=616663
Installing from C:\Users\TREVOR~1\AppData\Local\Temp\ptvs-g20gn_k5-pip\get-pip.py
Collecting pip
  Using cached pip-7.1.2-py2.py3-none-any.whl
Installing collected packages: pip
Successfully installed pip-7.1.2
Installation Complete
----- Successfully installed pip -----

Looking forward to a long term fix for this bug. Kinda annoying. Should probably provide some guidance in the short term, via docs for PTVS.

Cheers,
Trevor Sullivan

[zooba]

There was probably another folder named pip.dist-info or similar alongside where the pip folder was, but it seems to be installed okay now.

This kind of surgery on a Python environment is not something we'll want to do automatically, so the long term fix will come from pip's developers (or from Python or Windows gaining support for long path names). I've already reported the issue on pip's tracker, so we'll see what they come up with. Ultimately, there's nothing PTVS can do here - the copy during uninstall step is entirely out of our hands.

[zooba]

Thanks for the detailed report. It looks like pip is hitting Window's MAX_PATH limit with some of its temporary build files. I'll file an issue on their tracker, since they really shouldn't need to include the full path as a subdirectory of the temp directory.

You should be able to fix this by deleting the pip directories from your virtual environment's Lib\site-packages folder, and then installing it through PTVS (you should get a link in the Environments window when pip is not there).

[pcgeek86]

@zooba Steve, thanks for your detailed response and root cause analysis. I'm not sure I clearly understand your suggested workaround though. I'm already trying to install pip through PTVS.

When you delete pip from the Virtual Environment, pip is gone, so it can't install / upgrade itself. Should I be using easy_install somehow, instead?

----- Installing 'pip' -----
C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env\Scripts\python.exe: No module named pip
----- Failed to install 'pip' -----

[zooba]

We include a separate script that will install pip if you don't have it. There's a link in the Environments window pip section to trigger it when you don't have pip installed at all.

[pcgeek86]

@zooba That link disappears when I delete pip from the virtual environment directory, per your suggestion.

[zooba]

@pcgeek86 I don't remember exactly what the caching behavior here is, but you may need to close/reopen the project or VS after doing the delete. (Because we use Python to find pip, it's fairly expensive to keep querying repeatedly, so we don't do it.)

[pcgeek86]

@zooba I closed and re-opened the Visual Studio process and the link re-appeared. I now have pip 7.1.2 running in the virtual environment. However, because I ripped out pip 6.0.8 from the filesystem directly, rather than through the tooling, it complained about pip missing.

----- Installing pip -----
Installing with ensurepip
Ignoring indexes: https://pypi.python.org/simple
Requirement already up-to-date: setuptools in c:\users\trevorsullivan\source\repos\pythonapplication12\pythonapplication12\env\lib\site-packages
Collecting pip
Installing collected packages: pip
Successfully installed pip-6.0.8
You are using pip version 6.0.8, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-7.1.2-py2.py3-none-any.whl#md5=5ff9fec0be479e4e36df467556deed4d
  Using cached pip-7.1.2-py2.py3-none-any.whl
Collecting setuptools from https://pypi.python.org/packages/3.4/s/setuptools/setuptools-18.2-py2.py3-none-any.whl#md5=2b79c46f12ca11fca3720dcf0f327086
  Downloading setuptools-18.2-py2.py3-none-any.whl (461kB)
Collecting wheel
  Downloading wheel-0.24.0-py2.py3-none-any.whl (63kB)
Installing collected packages: wheel, setuptools, pip
  Found existing installation: setuptools 12.0.5
    Uninstalling setuptools-12.0.5:
      Successfully uninstalled setuptools-12.0.5
  Found existing installation: pip 6.0.8
    Uninstalling pip-6.0.8:
      Exception:
      Traceback (most recent call last):
        File "C:\Python34\lib\shutil.py", line 523, in move
          os.rename(src, real_dst)
      FileNotFoundError: [WinError 3] The system cannot find the path specified: 'c:\\users\\trevorsullivan\\source\\repos\\pythonapplication12\\pythonapplication12\\env\\lib\\site-packages\\pip\\_vendor\\requests\\packages\\urllib3\\packages\\ssl_match_hostname\\__pycache__\\_implementation.cpython-34.pyc' -> 'C:\\Users\\TREVOR~1\\AppData\\Local\\Temp\\pip-u24ad31b-uninstall\\users\\trevorsullivan\\source\\repos\\pythonapplication12\\pythonapplication12\\env\\lib\\site-packages\\pip\\_vendor\\requests\\packages\\urllib3\\packages\\ssl_match_hostname\\__pycache__\\_implementation.cpython-34.pyc'

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication12\PythonApplication12\env\lib\site-packages\pip\basecommand.py", line 232, in main
          status = self.run(options, args)
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication12\PythonApplication12\env\lib\site-packages\pip\commands\install.py", line 347, in run
          root=options.root_path,
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication12\PythonApplication12\env\lib\site-packages\pip\req\req_set.py", line 543, in install
          requirement.uninstall(auto_confirm=True)
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication12\PythonApplication12\env\lib\site-packages\pip\req\req_install.py", line 667, in uninstall
          paths_to_remove.remove(auto_confirm)
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication12\PythonApplication12\env\lib\site-packages\pip\req\req_uninstall.py", line 126, in remove
          renames(path, new_path)
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication12\PythonApplication12\env\lib\site-packages\pip\utils\__init__.py", line 316, in renames
          shutil.move(old, new)
        File "C:\Python34\lib\shutil.py", line 535, in move
          copy2(src, real_dst)
        File "C:\Python34\lib\shutil.py", line 245, in copy2
          copyfile(src, dst, follow_symlinks=follow_symlinks)
        File "C:\Python34\lib\shutil.py", line 109, in copyfile
          with open(dst, 'wb') as fdst:
      FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\TREVOR~1\\AppData\\Local\\Temp\\pip-u24ad31b-uninstall\\users\\trevorsullivan\\source\\repos\\pythonapplication12\\pythonapplication12\\env\\lib\\site-packages\\pip\\_vendor\\requests\\packages\\urllib3\\packages\\ssl_match_hostname\\__pycache__\\_implementation.cpython-34.pyc'
Failed to upgrade pip, which probably indicates that it isn't installed properly.
Downloading pip from https://go.microsoft.com/fwlink/?LinkId=616663
Installing from C:\Users\TREVOR~1\AppData\Local\Temp\ptvs-g20gn_k5-pip\get-pip.py
Collecting pip
  Using cached pip-7.1.2-py2.py3-none-any.whl
Installing collected packages: pip
Successfully installed pip-7.1.2
Installation Complete
----- Successfully installed pip -----

Looking forward to a long term fix for this bug. Kinda annoying. Should probably provide some guidance in the short term, via docs for PTVS.

Cheers,
Trevor Sullivan

[zooba]

There was probably another folder named pip.dist-info or similar alongside where the pip folder was, but it seems to be installed okay now.

This kind of surgery on a Python environment is not something we'll want to do automatically, so the long term fix will come from pip's developers (or from Python or Windows gaining support for long path names). I've already reported the issue on pip's tracker, so we'll see what they come up with. Ultimately, there's nothing PTVS can do here - the copy during uninstall step is entirely out of our hands.