Inconsistent behaviour of pip-install when `pipenv lock` is run from a setup.py on windows

[theodox]

Issue description

I'm seeing different results when the same repo is used as an upstream dependency for two different pipenvs. One of them installs as expected with pipenv, the other fails with when the repo's setup.py tries to generate the requirements field using a call to pipenv lock -r. Same code, same OS, same machine -- different result. I added a debug printout to the setup.py and on the failure case it appears the calling pipenv lock is trying and failing to create a virtualenv (error txt below).

I've worked around the issue by converting the setup.py to have a conventional requirements field in setup() instead of invoking pipenv to do so.

Expected result

I'd expect that a repo which can be installed as a dependency works the same way for any pipfile in which it is included. AFAICT the only difference between the pipfile which install correctly and the one which does not is that the failing one installs the repo with editable=True -- however I tried installing it with out the -e flag and lock still failed (see below).

Actual result

The setup.py fails with exit code 1. I added a catch around the source of the error and captured the output, which indicates that pipenv lock -r is trying and failing to create a virtualenv. Here's the debug spew:

Creating a virtualenv for this project…
Pipfile: C:\Users\Steve\AppData\Local\Temp\tmpp__azx9bsource\studio-pylib\Pipfile
Using C:/ul/tools/python/python_installs/371/python.exe (3.7.1) to create virtualenv…

[    ] Creating virtual environment...�
<snip/>
[   =] Creating virtual environment...Running virtualenv with interpreter C:/ul/tools/python/python_installs/371/python.exe
Using base prefix 'C:\\ul\\tools\\python\\python_installs\\371'
New python executable in C:\Users\Steve\.virtualenvs\studio-pylib-_e8yWT1L\Scripts\python.exe
Installing setuptools, pip, wheel...

  Complete output from command C:\Users\Steve\.virt...L\Scripts\python.exe - setuptools pip wheel:
  ERROR: Can not use any platform or abi specific options unless installing via '--target'
----------------------------------------
...Installing setuptools, pip, wheel...done.

�Failed creating virtual environment 
[pipenv.exceptions.VirtualenvCreationException]:   File "c:\ul\tools\python\python_installs\371\lib\site-packages\pipenv\vendor\click\decorators.py", line 17, in new_func
[pipenv.exceptions.VirtualenvCreationException]:       return f(get_current_context(), *args, **kwargs)
[pipenv.exceptions.VirtualenvCreationException]:   File "c:\ul\tools\python\python_installs\371\lib\site-packages\pipenv\cli\command.py", line 319, in lock
[pipenv.exceptions.VirtualenvCreationException]:       ensure_project(three=state.three, python=state.python, pypi_mirror=state.pypi_mirror)
[pipenv.exceptions.VirtualenvCreationException]:   File "c:\ul\tools\python\python_installs\371\lib\site-packages\pipenv\core.py", line 574, in ensure_project
[pipenv.exceptions.VirtualenvCreationException]:       pypi_mirror=pypi_mirror,
[pipenv.exceptions.VirtualenvCreationException]:   File "c:\ul\tools\python\python_installs\371\lib\site-packages\pipenv\core.py", line 506, in ensure_virtualenv
[pipenv.exceptions.VirtualenvCreationException]:       python=python, site_packages=site_packages, pypi_mirror=pypi_mirror
[pipenv.exceptions.VirtualenvCreationException]:   File "c:\ul\tools\python\python_installs\371\lib\site-packages\pipenv\core.py", line 935, in do_create_virtualenv
[pipenv.exceptions.VirtualenvCreationException]:       extra=[crayons.blue("{0}".format(c.err)),]
[pipenv.exceptions.VirtualenvCreationException]: Traceback (most recent call last):
  File "c:\ul\tools\python\python_installs\371\lib\site-packages\virtualenv.py", line 2462, in <module>
    main()
  File "c:\ul\tools\python\python_installs\371\lib\site-packages\virtualenv.py", line 762, in main
    symlink=options.symlink,
  File "c:\ul\tools\python\python_installs\371\lib\site-packages\virtualenv.py", line 1015, in create_environment
    install_wheel(to_install, py_executable, search_dirs, download=download)
  File "c:\ul\tools\python\python_installs\371\lib\site-packages\virtualenv.py", line 968, in install_wheel
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
  File "c:\ul\tools\python\python_installs\371\lib\site-packages\virtualenv.py", line 854, in call_subprocess
    raise OSError("Command {} failed with error code {}".format(cmd_desc, proc.returncode))
OSError: Command C:\Users\Steve\.virt...L\Scripts\python.exe - setuptools pip wheel failed with error code 1

Failed to create virtual environment.

Installing the same repo without the editable flag still failed, but with a different error:

An error occurred while installing git+ssh://git@github.com/undeadlabs/studio_pylib.git@9b1bc5b6dcd0144e781754c1acd260230d5dde96#egg=studio-pylib! Will try again.
  ================================ 7/7 - 00:00:09
[pipenv.exceptions.InstallError]:   File "c:\ul\tools\python\python_installs\371\lib\site-packages\pipenv\core.py", line 1874, in do_install
[pipenv.exceptions.InstallError]:       keep_outdated=keep_outdated
[pipenv.exceptions.InstallError]:   File "c:\ul\tools\python\python_installs\371\lib\site-packages\pipenv\core.py", line 1253, in do_init
[pipenv.exceptions.InstallError]:       pypi_mirror=pypi_mirror,
[pipenv.exceptions.InstallError]:   File "c:\ul\tools\python\python_installs\371\lib\site-packages\pipenv\core.py", line 859, in do_install_dependencies
[pipenv.exceptions.InstallError]:       retry_list, procs, failed_deps_queue, requirements_dir, **install_kwargs
[pipenv.exceptions.InstallError]:   File "c:\ul\tools\python\python_installs\371\lib\site-packages\pipenv\core.py", line 763, in batch_install
[pipenv.exceptions.InstallError]:       _cleanup_procs(procs, not blocking, failed_deps_queue, retry=retry)
[pipenv.exceptions.InstallError]:   File "c:\ul\tools\python\python_installs\371\lib\site-packages\pipenv\core.py", line 681, in _cleanup_procs
[pipenv.exceptions.InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: ['Collecting studio-pylib from git+ssh://git@github.com/undeadlabs/studio_pylib.git@9b1bc5b6dcd0144e781754c1acd260230d5dde96#egg=studio-pylib', '  Cloning ssh://git@github.com/undeadlabs/studio_pylib.git (to revision 9b1bc5b6dcd0144e781754c1acd260230d5dde96) to c:\\users\\steve\\appdata\\local\\temp\\pip-install-65x28ke7\\studio-pylib', '    Complete output from command python setup.py egg_info:', '    Traceback (most recent call last):', '      File "C:\\ul\\tools\\python\\python_installs\\371\\Lib\\subprocess.py", line 468, in run', '        stdout, stderr = process.communicate(input, timeout=timeout)', '      File "C:\\ul\\tools\\python\\python_installs\\371\\Lib\\subprocess.py", line 933, in communicate', '        stdout, stderr = self._communicate(input, endtime, timeout)', '      File "C:\\ul\\tools\\python\\python_installs\\371\\Lib\\subprocess.py", line 1263, in _communicate', '        raise TimeoutExpired(self.args, orig_timeout)', "    subprocess.TimeoutExpired: Command 'pipenv lock -r' timed out after 3 seconds", '    ', '    During handling of the above exception, another exception occurred:', '    ', '    Traceback (most recent call last):', '      File "<string>", line 1, in <module>', '      File "C:\\Users\\Steve\\AppData\\Local\\Temp\\pip-install-65x28ke7\\studio-pylib\\setup.py", line 12, in <module>', "        universal_newlines=True).decode('utf-8').splitlines()", '      File "C:\\ul\\tools\\python\\python_installs\\371\\Lib\\subprocess.py", line 389, in check_output', '        **kwargs).stdout', '      File "C:\\ul\\tools\\python\\python_installs\\371\\Lib\\subprocess.py", line 473, in run', '        stderr=stderr)', "    subprocess.TimeoutExpired: Command 'pipenv lock -r' timed out after 3 seconds", '    ', '    ----------------------------------------']
[pipenv.exceptions.InstallError]: ['Command "python setup.py egg_info" failed with error code 1 in C:\\Users\\Steve\\AppData\\Local\\Temp\\pip-install-65x28ke7\\studio-pylib\\']
ERROR: ERROR: Package installation failed...
     ================================ 0/1 - 00:00:08

Steps to replicate

I can't provide the repo, which is private and company specific. However here's the contents of its pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
pywin32 = "*"

[requires]
python_version = "3.7"

and here's the setup.py:

# studio pylib setup py
from setuptools import setup, find_packages
import subprocess
import glob

try:

    requires = subprocess.check_output('pipenv lock -r',
                                       stderr=subprocess.STDOUT,
                                       shell=True,
                                       timeout=3,
                                       universal_newlines=True).decode('utf-8').splitlines()


except subprocess.CalledProcessError as e:
    with open("wtf.txt", "w")as output:
        output.writelines(e.output)
        output.writelines(e.returncode)

    requires = [i for i in requires[1:] if not i.startswith("-")]

    setup(name='studio-pylib',
          version="0.1.6",
          install_requires=requires,
          package_dir={"": "packages"},
          packages=find_packages('packages'),
          package_data={"": ["*.*"]},
          scripts=glob.glob('scripts/*.py'),
          long_description='xxxxx',
          author='xxxx')

---

$ pipenv --support

Pipenv version: '2018.11.26'

Pipenv location: 'c:\\ul\\tools\\python\\python_installs\\371\\lib\\site-packages\\pipenv'

Python location: 'c:\\ul\\tools\\python\\python_installs\\371\\python.exe'

Python installations found:

• 3.7.1: C:\ul\tools\python\python_installs\371\python.exe
• 2.7: C:\ul\tools\python\python_installs\2715\python.exe

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.1',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.17134',
 'python_full_version': '3.7.1',
 'python_version': '3.7',
 'sys_platform': 'win32'}

System environment variables:

• ACLOCAL_PATH
• ADSK_CLM_WPAD_PROXY_CHECK
• ALLUSERSPROFILE
• ANSICON
• ANSICON_DEF
• APPDATA
• COMMONPROGRAMFILES
• COMPUTERNAME
• COMSPEC
• CONFIG_SITE
• CHOCOLATEYINSTALL
• CHOCOLATEYLASTPATHUPDATE
• COMMONPROGRAMFILES(X86)
• COMMONPROGRAMW6432
• CONEMUANSI
• CONEMUANSILOG
• CONEMUARGS2
• CONEMUARGS
• CONEMUBACKHWND
• CONEMUBASEDIR
• CONEMUBUILD
• CONEMUCONFIG
• CONEMUDIR
• CONEMUDRAWHWND
• CONEMUDRIVE
• CONEMUHWND
• CONEMUHOOKS
• CONEMUPID
• CONEMUPALETTE
• CONEMUSERVERPID
• CONEMUTASK
• CONEMUWORKDIR
• CONEMUWORKDRIVE
• DISPLAY
• DRIVERDATA
• DURANGOXDK
• EXEPATH
• FPS_BROWSER_APP_PROFILE_STRING
• FPS_BROWSER_USER_PROFILE_STRING
• GPA_GLOBAL_INJECTION_MODE
• GPA_UWP_INJECTION_MODE
• HOME
• HOMEDRIVE
• HOMEPATH
• HOSTNAME
• INFOPATH
• INTEL_SEA_FILTER
• LANG
• LOCALAPPDATA
• LOGONSERVER
• MANPATH
• MAYA_DISABLE_CER
• MAYA_DISABLE_CIP
• MAYA_DISABLE_CLIC_IPM
• MINGW_CHOST
• MINGW_PACKAGE_PREFIX
• MINGW_PREFIX
• MSYSTEM
• MSYSTEM_CARCH
• MSYSTEM_CHOST
• MSYSTEM_PREFIX
• NUMBER_OF_PROCESSORS
• OLDPWD
• ORIGINAL_PATH
• ORIGINAL_TEMP
• ORIGINAL_TMP
• OS
• PATH
• PATHEXT
• PKG_CONFIG_PATH
• PLINK_PROTOCOL
• PROCESSOR_ARCHITECTURE
• PROCESSOR_IDENTIFIER
• PROCESSOR_LEVEL
• PROCESSOR_REVISION
• PROGRAMFILES
• PROJECT_HOME
• PS1
• PSMODULEPATH
• PUBLIC
• PWD
• PROGRAMDATA
• PROGRAMFILES(X86)
• PROGRAMW6432
• SESSIONNAME
• SHELL
• SHLVL
• SSH_ASKPASS
• SYSTEMDRIVE
• SYSTEMROOT
• TEMP
• TERM
• TMP
• TMPDIR
• UL_ENGINE
• UL_PROJECT
• USERDNSDOMAIN
• USERDOMAIN
• USERDOMAIN_ROAMINGPROFILE
• USERNAME
• USERPROFILE
• VS140COMNTOOLS
• WINDIR
• WORKON_HOME
• XBOXONEEXTENSIONSDKLATEST
• XBOXONEXDKLATEST
• _
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenvûspecific environment variables:

Debugûspecific environment variables:

• PATH: C:\Users\Steve\bin;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\local\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\cmd;C:\Program Files\ConEmu\ConEmu\Scripts;C:\Program Files\ConEmu;C:\Program Files\ConEmu\ConEmu;C:\ProgramData\DockerDesktop\version-bin;C:\Program Files\Docker\Docker\Resources\bin;C:\Program Files (x86)\Intel\iCLS Client;C:\Program Files\Intel\iCLS Client;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\OpenVPN\bin;C:\Program Files\Perforce;C:\Program Files (x86)\Xoreax\IncrediBuild;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\ul\New folder\products\bin;C:\ProgramData\chocolatey\bin;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit;C:\Program Files (x86)\Skype\Phone;C:\Program Files\Autodesk\Maya2016\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0;C:\WINDOWS\System32\OpenSSH;C:\Program Files\Perforce;C:\Users\Steve\AppData\Local\Microsoft\WindowsApps;C:\ul\labware\products\bin;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\WINDOWS\system32\config\systemprofile\AppData\Local\Microsoft\WindowsApps;C:\ul\labware\products\bin;C:\ul\tools\python\python_installs\371\Scripts;C:\ul\tools\python\python_installs\371;C:\Users\Steve\AppData\Local\Programs\Python\Launcher;C:\ul\tools\python\python37\Scripts;C:\ul\tools\python\python37;C:\Users\Steve\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Git\usr\bin\vendor_perl;C:\Program Files\Git\usr\bin\core_perl
• SHELL: C:\Program Files\Git\usr\bin\bash.exe
• LANG: en_US.UTF-8
• PWD: C:/ul/tools/python/launchpad

---

Contents of Pipfile ('C:\ul\tools\python\launchpad\Pipfile'):

name = "pypi" 
url = "https://pypi.org/simple" 
verify_ssl = true  

[dev-packages]  

[packages] 
studio-pylib = {editable = true,git = "git@github.com/undeadlabs/studio_pylib.git"}
pyside2 = "*"
appdirs = "*"
web-pdb = "*"

[requires] 
python_version = "3.7"

[techalchemy]

This was a known issue with timeouts being set incorrectly in the last release, we adjusted the timeouts (which won't fix the underlying issue) and you can also adjust this on the fly for now by setting PIPENV_TIMEOUT=900

[techalchemy]

This was actually fixed in #3298 but I haven't pushed a news entry up, I'll make sure to capture that in a pr shortly

[techalchemy]

Hm any chance you can confirm if this is working? Just noticed the --target related messaging in here so I want to be sure we did actually fix this

[techalchemy]

Also @theodox would you mind explaining the goal of your call to pipenv lock -r here? It's possible you can accomplish it differently

[techalchemy]

try:

    requires = subprocess.check_output('pipenv lock -r',
                                       stderr=subprocess.STDOUT,
                                       shell=True,
                                       timeout=3,
                                       universal_newlines=True).decode('utf-8').splitlines()

Under the new version(s) of pip, nothing causes setuptools to get invoked directly, which means you are never actually chdir-ed into the directory of your setup.py anymore. So, this code will execute, but it will execute somewhere besides the directory where your lockfile is located, which is both why there is a a failure and why your command is timing out. Note that this will also always dynamically generate a virtualenv using that location as the virtualenv location which is probably not desirable.

If you just want to add things from your pipfile, you can do this (since pipenv is already installed I can assume):

import os
from pipenv.vendor.requirementslib.models.pipfile import Pipfile

pipfile = Pipfile.load(os.path.dirname(os.path.abspath(__file__)))
pipfile_requirements = [r for r in pipfile.dev_requirements] + [r for r in pipfile.requirements]
requirement_lines = [r.as_line(include_hashes=False) for r in pipfile_requirements]

If you want to do this with the lockfile instead (you really should include the lockfile and not rely on locking on the fly with a 3 second timeout):

import os
from pipenv.vendor.requirementslib.models.lockfile import Lockfile

lockfile = Lockfile.load(os.path.dirname(os.path.abspath(__file__)))
lockfile_requirements = [r for r in lockfile.dev_requirements] + [r for r in lockfile.requirements]
requirement_lines = [r.as_line(include_hashes=False) for r in lockfile_requirements]

Let me know if any of this is helpful -- the subprocess call in your setup.py is definitely not something we support so I'm not too sure if it will work properly in any case