pipenv fails to install all my packages, and then installs all of them

[jacek-jablonski]

Hi, it might be something that I do not understand or so. I have 2 sources specified and none of them is official pypi, but mirrors, so here is my Pipfile:

[[source]]

url = "https://company/api/pypi/pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[[source]]

url = "https://company/api/pypi/pypi-local/simple"
verify_ssl = true
name = "pypi-local"


[packages]

cachetools = "==2.0.1"
click = "==6.2"
data-object = "==1.2.1"
flask-cache = "==0.13.1"
flask-classy = "==0.6.10"
flask-humanize = "==0.3.0"
flask-migrate = "==2.1.1"
flask-oauthlib = "==0.9.4"
flask-sqlalchemy = "==2.3.2"
flask-wtf = "==0.14.2"
flask = "==0.12.2"
huey = "==1.6.0"
pendulum = "==1.4.0"
pyyaml = "==3.12"
redis = "==2.10.6"
requests = "==2.18.4"
requests-cache = "==0.4.13"
sqlalchemy = "==1.2.1"
whitenoise = "==3.3.1"
gunicorn = "==19.7.1"


[dev-packages]

ipdb = "*"
ipython = "*"
flask-shell-ipython = "*"
coverage = "*"
flake8 = "*"
flask-debugtoolbar = "*"

Describe your environment

1. OS Type Ubuntu
2. Python version: $ python -V Python 3.6.1
3. Pipenv version: $ pipenv --version pipenv, version 9.0.3

Expected result

All packages installed immediatly.

Actual result

Installing dependencies from Pipfile.lock (7e9249)…
An error occurred while installing blinker==1.4! Will try again.
An error occurred while installing click==6.2! Will try again.
An error occurred while installing coverage==4.4.2! Will try again.
An error occurred while installing decorator==4.2.1! Will try again.
An error occurred while installing flask-debugtoolbar==0.10.1! Will try again.
An error occurred while installing flask-shell-ipython==0.3.0! Will try again.
An error occurred while installing ipdb==0.10.3! Will try again.
An error occurred while installing ipython==6.2.1! Will try again.
An error occurred while installing ipython-genutils==0.2.0! Will try again.
An error occurred while installing jedi==0.11.1! Will try again.
An error occurred while installing parso==0.1.1! Will try again.
An error occurred while installing pexpect==4.3.1; sys_platform != 'win32'! Will try again.
An error occurred while installing pickleshare==0.7.4! Will try again.
An error occurred while installing prompt-toolkit==1.0.15! Will try again.
An error occurred while installing ptyprocess==0.5.2! Will try again.
An error occurred while installing pygments==2.2.0! Will try again.
An error occurred while installing simplegeneric==0.8.1! Will try again.
An error occurred while installing traitlets==4.3.2! Will try again.
An error occurred while installing wcwidth==0.1.7! Will try again.
Installing initially–failed dependencies…
Success installing blinker==1.4!
Success installing click==6.7!
Success installing coverage==4.4.2!
Success installing decorator==4.2.1!
Success installing flask-debugtoolbar==0.10.1!
Success installing flask-shell-ipython==0.3.0!
Success installing ipdb==0.10.3!
Success installing ipython==6.2.1!
Success installing ipython-genutils==0.2.0!
Success installing jedi==0.11.1!
Success installing parso==0.1.1!
Success installing pexpect==4.3.1; sys_platform != 'win32'!
Success installing pickleshare==0.7.4!
Success installing prompt-toolkit==1.0.15!
Success installing ptyprocess==0.5.2!
Success installing pygments==2.2.0!
Success installing simplegeneric==0.8.1!
Success installing traitlets==4.3.2!
Success installing wcwidth==0.1.7!

Steps to replicate

pipenv install --system
Please also note, that pipenv is installing click==6.7 despite specifying 6.2 version.

[alexpilk]

I'm also struggling with this issue, are there any updates or solutions?

[techalchemy]

Hi @JacekJab sorry you are running into this -- the initial failure isn't actually a bug, per se, I'm guessing it is related to concurrent installs from a non-pypi repository. They initially failed and then fell back to sequential installation. As far as the click installation, you will need to provide the full context of the command you ran, your $PATH, the output of env, that stuff would help

[kennethreitz]

closing since they are successful in the end — we'll need much more information in order to replicate

[asyd]

I have the same issue with latest version of pipenv. I'm curious, @JacekJab are you using Nexus for your repositories?

[pawelswiecki]

I'm having the same issue. Pretty annoying and should be looked into.

[techalchemy]

@pawelswiecki please provide details including the output and Pipfile you used per the issue template. Telling us to look into something with no additional detail is not productive and as volunteers would rank pretty high on the list of things we also find annoying

[wjlow]

I'm having the same issue. I'm running tox.

My tox.ini:

[tox]
envlist = flake8-py3
skipsdist = True

[testenv]
passenv=HOME
deps = pipenv
setenv=
    PYTHONPATH = {toxinidir}/dag
commands=
    pipenv install --skip-lock
    pipenv install --dev --skip-lock
    pipenv run flake8 --version
    pipenv run flake8 dag
    pipenv run pytest

My pipfile:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
"flake8" = "*"
docker-py = "*"
google-cloud-dataflow = "==2.2.0"
google-cloud-bigquery = "*"
apache-airflow = {extras = ["crypto", "postgres", "hive", "celery", "jdbc", "gcp_api"], version = "==1.9.0"}
paramiko = "*"

[dev-packages]
pytest = "*"
coverage = "*"

[requires]
python_version = "3.6"

The output (truncated because the rest of it is just test output, which is okay):

flake8-py3 installed: You are using pip version 9.0.3, however version 10.0.1 is available.,You should consider upgrading via the 'pip install --upgrade pip' command.,alembic==0.8.10,amqp==2.2.2,apache-airflow==1.9.0,apache-beam==2.2.0,asn1crypto==0.24.0,attrs==17.4.0,avro==1.8.2,Babel==2.5.3,bcrypt==3.1.4,billiard==3.5.0.3,bitarray==0.8.1,bleach==2.1.2,cachetools==2.0.1,celery==4.1.0,certifi==2018.1.18,cffi==1.11.5,chardet==3.0.4,click==6.7,configparser==3.5.0,coverage==4.5.1,crcmod==1.7,croniter==0.3.20,cryptography==2.2.2,dill==0.2.6,docker-py==1.10.6,docker-pycreds==0.2.2,docutils==0.14,flake8==3.5.0,Flask==0.11.1,Flask-Admin==1.4.1,Flask-Cache==0.13.1,Flask-Login==0.2.11,flask-swagger==0.2.13,Flask-WTF==0.14,flower==0.9.2,funcsigs==1.0.0,future==0.16.0,gapic-google-cloud-pubsub-v1==0.15.4,gitdb2==2.0.3,GitPython==2.1.9,google-api-core==1.1.0,google-api-python-client==1.5.5,google-apitools==0.5.11,google-auth==1.4.1,google-auth-httplib2==0.0.3,google-auth-oauthlib==0.2.0,google-cloud-bigquery==0.32.0,google-cloud-core==0.28.1,google-cloud-dataflow==2.2.0,google-cloud-pubsub==0.26.0,google-gax==0.15.16,google-resumable-media==0.3.1,googleapis-common-protos==1.5.3,googledatastore==7.0.1,grpc-google-iam-v1==0.11.4,grpcio==1.10.0,gunicorn==19.7.1,hive-thrift-py==0.0.1,html5lib==1.0.1,httplib2==0.9.2,idna==2.6,impyla==0.14.1,itsdangerous==0.24,JayDeBeApi==1.1.1,Jinja2==2.8.1,JPype1==0.6.3,kombu==4.1.0,lockfile==0.12.2,lxml==3.8.0,Mako==1.0.7,Markdown==2.6.11,MarkupSafe==1.0,mccabe==0.6.1,mock==2.0.0,more-itertools==4.1.0,numpy==1.14.2,oauth2client==2.0.2,oauthlib==2.0.7,ordereddict==1.1,pandas==0.22.0,pandas-gbq==0.4.0,paramiko==2.4.1,pbr==4.0.1,pipenv==11.9.0,pluggy==0.6.0,ply==3.8,proto-google-cloud-datastore-v1==0.90.4,proto-google-cloud-pubsub-v1==0.15.4,protobuf==3.3.0,psutil==4.4.2,psycopg2==2.7.4,py==1.5.3,pyasn1==0.4.2,pyasn1-modules==0.2.1,pycodestyle==2.3.1,pycparser==2.18,pyflakes==1.6.0,Pygments==2.2.0,PyHive==0.5.1,PyNaCl==1.2.1,pyOpenSSL==17.5.0,pytest==3.5.0,python-daemon==2.1.2,python-dateutil==2.7.2,python-editor==1.0.3,python-nvd3==0.14.2,python-slugify==1.1.4,pytz==2018.3,PyYAML==3.12,requests==2.18.4,requests-oauthlib==0.8.0,rsa==3.4.2,setproctitle==1.1.10,six==1.10.0,smmap2==2.0.3,SQLAlchemy==1.2.6,tabulate==0.7.7,thrift==0.11.0,thriftpy==0.3.9,tornado==5.0.1,typing==3.6.4,unicodecsv==0.14.1,Unidecode==1.0.22,uritemplate==3.0.0,urllib3==1.22,vine==1.1.4,virtualenv==15.2.0,virtualenv-clone==0.3.0,webencodings==0.5.1,websocket-client==0.47.0,Werkzeug==0.14.1,WTForms==2.1,zope.deprecation==4.3.0
flake8-py3 runtests: PYTHONHASHSEED='4112944039'
flake8-py3 runtests: commands[0] | pipenv install --skip-lock
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project.
Installing dependencies from Pipfile…
An error occurred while installing apache-airflow[crypto,postgres,hive,celery,jdbc,gcp_api]==1.9.0! Will try again.
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 6/6 — 00:00:02
Installing initially–failed dependencies…
Success installing apache-airflow[crypto,postgres,hive,celery,jdbc,gcp_api]==1.9.0!
  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:01
flake8-py3 runtests: commands[1] | pipenv install --dev --skip-lock
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project.
Installing dependencies from Pipfile…
An error occurred while installing apache-airflow[crypto,postgres,hive,celery,jdbc,gcp_api]==1.9.0! Will try again.
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 8/8 — 00:00:02
Installing initially–failed dependencies…
Success installing apache-airflow[crypto,postgres,hive,celery,jdbc,gcp_api]==1.9.0!
  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:01
flake8-py3 runtests: commands[2] | pipenv run flake8 --version
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project.
3.5.0 (mccabe: 0.6.1, pycodestyle: 2.3.1, pyflakes: 1.6.0) CPython 3.6.4 on Darwin
flake8-py3 runtests: commands[3] | pipenv run flake8 dag
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project.
flake8-py3 runtests: commands[4] | pipenv run pytest
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project.

[techalchemy]

does this work if you upgrade pip first in your CI build with python -m pip install --upgrade pip?

[wjlow]

Nope, same problem. Here are the steps I took and the output:

Okay I removed .tox/ directory and ran tox. I did this twice and both times I got the following output:

flake8-py3 create: /Users/jack.low/work/dpd_seller_triggers/.tox/flake8-py3
flake8-py3 installdeps: pipenv
flake8-py3 installed: certifi==2018.4.16,pipenv==2018.5.18,virtualenv==15.2.0,virtualenv-clone==0.3.0
flake8-py3 runtests: PYTHONHASHSEED='4227619038'
flake8-py3 runtests: commands[0] | pipenv install --skip-lock
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead.
Installing dependencies from Pipfile…
An error occurred while installing apache-airflow[crypto,postgres,hive,celery,jdbc,gcp_api]==1.9.0! Will try again.
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 6/6 — 00:00:35
Installing initially–failed dependencies…
Success installing apache-airflow[crypto,postgres,hive,celery,jdbc,gcp_api]==1.9.0!
  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:01:12
flake8-py3 runtests: commands[1] | pipenv install --dev --skip-lock
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead.
Installing dependencies from Pipfile…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 8/8 — 00:00:08
flake8-py3 runtests: commands[2] | pipenv run flake8 --version
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead.
3.5.0 (mccabe: 0.6.1, pycodestyle: 2.3.1, pyflakes: 1.6.0) CPython 3.6.4 on Darwin
flake8-py3 runtests: commands[3] | pipenv run flake8 dag
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead.
flake8-py3 runtests: commands[4] | pipenv run pytest
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead.

Now I deleted .tox/ again, and added the following line python -m pip install --upgrade pip to the first command in my tox.ini:

[tox]
envlist = flake8-py3
skipsdist = True

[testenv]
passenv=HOME
deps = pipenv
setenv=
    PYTHONPATH = {toxinidir}/dag
commands=
    python -m pip install --upgrade pip
    pipenv install --skip-lock
    pipenv install --dev --skip-lock
    pipenv run flake8 --version
    pipenv run flake8 dag
    pipenv run pytest

Running tox again gives me this:

flake8-py3 create: /Users/jack.low/work/dpd_seller_triggers/.tox/flake8-py3
flake8-py3 installdeps: pipenv
flake8-py3 installed: certifi==2018.4.16,pipenv==2018.5.18,virtualenv==15.2.0,virtualenv-clone==0.3.0
flake8-py3 runtests: PYTHONHASHSEED='2202490466'
flake8-py3 runtests: commands[0] | python -m pip install --upgrade pip
Requirement already up-to-date: pip in ./.tox/flake8-py3/lib/python3.6/site-packages (10.0.1)
flake8-py3 runtests: commands[1] | pipenv install --skip-lock
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead.
Installing dependencies from Pipfile…
An error occurred while installing apache-airflow[crypto,postgres,hive,celery,jdbc,gcp_api]==1.9.0! Will try again.
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 6/6 — 00:00:12
Installing initially–failed dependencies…
Success installing apache-airflow[crypto,postgres,hive,celery,jdbc,gcp_api]==1.9.0!
  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:30
flake8-py3 runtests: commands[2] | pipenv install --dev --skip-lock
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead.
Installing dependencies from Pipfile…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 8/8 — 00:00:06
flake8-py3 runtests: commands[3] | pipenv run flake8 --version
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead.
3.5.0 (mccabe: 0.6.1, pycodestyle: 2.3.1, pyflakes: 1.6.0) CPython 3.6.4 on Darwin
flake8-py3 runtests: commands[4] | pipenv run flake8 dag
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead.
flake8-py3 runtests: commands[5] | pipenv run pytest
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead.

[wjlow]

FYI: Just running python3 -m pip install --upgrade pip manually before running tox gives me the same issue.

I'm not a Python dev so I can't help too much in providing useful info so I apologise ahead of time.

[uranusjr]

@wjlow I think what @techalchemy meant is you need to upgrade the pip inside the tox-managed environment, not the one used by your project, i.e. add it to the commands config, like this:

commands =
    pipenv run python -m pip install --upgrade pip
    pipenv install --skip-lock
    pipenv install --dev --skip-lock
    pipenv run flake8 --version
    pipenv run flake8 dag
    pipenv run pytest

[wjlow]

Same issue, unfortunately!