Pipenv lock still takes way too long to complete

[mjpieters]

pipenv lock is still taking up to 30 minutes to complete for my medium-sized project.

While #4403 was closed as fixed and indeed no full tree copies are made any more, pipenv lock is still unworkably slow. What used to take 50 seconds under 2018.11.26, now takes 30 minutes to figure out that nothing changed:

$ export PIPENV_INSTALL_TIMEOUT=10000
$ cp Pipfile.lock Pipfile.lock.before
$ time pipenv lock
Locking [dev-packages] dependencies…
Building requirements...
Resolving dependencies...
✔ Success!
Locking [packages] dependencies…
Building requirements...
Resolving dependencies...
✔ Success!
Updated Pipfile.lock (0e1198)!

real	29m19.701s
user	28m39.300s
sys	4m8.222s
$ diff -su Pipfile.lock.old Pipfile.lock
Files Pipfile.lock.old and Pipfile.lock are identical

This is making it untenable to keep using Pipenv.

Full pipenv lock -v output in a gist.

Pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
flake8 = "*"
flake8-bugbear = "*"
black = "==19.10b0"
pre-commit = "*"
pytest = "*"
pytest-cov = "*"

[packages]
project-name = {editable = true,path = "."}
"zc.buildout" = "*"

[requires]
python_version = "3.6"

where project-name (redacted name) lists the following install-requires lines:

install_requires =
    apache-airflow[aws,celery,postgres,redis] >= 1.10.11
    airflow_multi_dagrun
    airflow-prometheus-exporter

---

$ pipenv --support

Pipenv version: '2020.8.13'

Pipenv location: '/usr/local/Cellar/pipenv/2020.8.13/libexec/lib/python3.8/site-packages/pipenv'

Python location: '/usr/local/Cellar/pipenv/2020.8.13/libexec/bin/python3.8'

Python installations found:

• 3.9.0: /Users/mj/bin/python3.9
• 3.8.5: /usr/local/bin/python3
• 3.8.5: /usr/local/bin/python3.8
• 3.8.3: /Users/mj/bin/python3.8
• 3.7.7: /Users/mj/bin/python3.7
• 3.7.6: /opt/miniconda3/bin/python3
• 3.7.6: /opt/miniconda3/bin/python3.7m
• 3.7.6: /opt/miniconda3/bin/python3.7
• 3.7.3: /Users/mj/.pyenv/versions/3.7.3/bin/python3
• 3.6.10: /Users/mj/bin/python3.6
• 3.6.1: /Users/mj/bin/pypy3
• 3.5.9: /Users/mj/bin/python3.5
• 3.4.10: /Users/mj/bin/python3.4
• 3.3.7: /Users/mj/bin/python3.3
• 3.2.6: /Users/mj/bin/python3.2
• 2.7.18: /Users/mj/bin/python2.7
• 2.7.16: /usr/bin/python2.7
• 2.6.9: /Users/mj/bin/python2.6
• 2.5.6: /Users/mj/bin/python2.5
• 2.4.6: /Users/mj/bin/python2.4

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.7.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.7.0: Wed May 27 17:00:02 PDT '
                     '2020; root:xnu-4570.71.80.1~1/RELEASE_X86_64',
 'python_full_version': '3.8.5',
 'python_version': '3.8',
 'sys_platform': 'darwin'}

System environment variables:

• PATH
• AUTOJUMP_ERROR_PATH
• NVM_INC
• TERM_PROGRAM
• GIT_PROMPT_END
• rvm_bin_path
• AUTOJUMP_SOURCED
• GEM_HOME
• NVM_CD_FLAGS
• SHELL
• TERM
• HISTSIZE
• TMPDIR
• COPYFILE_DISABLE
• IRBRC
• Apple_PubSub_Socket_Render
• CONDA_SHLVL
• PERL5LIB
• TERM_PROGRAM_VERSION
• CONDA_PROMPT_MODIFIER
• TERM_SESSION_ID
• MY_RUBY_HOME
• PERL_MB_OPT
• LC_ALL
• USER
• HISTFILESIZE
• NVM_DIR
• COMMAND_MODE
• CONDA_EXE
• rvm_path
• SSH_AUTH_SOCK
• __CF_USER_TEXT_ENCODING
• PTPIMG_API_KEY
• _CE_CONDA
• rvm_prefix
• _
• CONDA_PREFIX
• PWD
• HOMEBREW_GITHUB_API_TOKEN
• EDITOR
• LANG
• ITERM_PROFILE
• PYTHONSTARTUP
• XPC_FLAGS
• ITERM_ORIG_PS1
• XPC_SERVICE_NAME
• HISTCONTROL
• _CE_M
• FIGNORE
• rvm_version
• COLORFGBG
• HOME
• SHLVL
• LC_TERMINAL_VERSION
• ITERM_PREV_PS1
• ITERM_SESSION_ID
• PERL_LOCAL_LIB_ROOT
• LOGNAME
• CONDA_PYTHON_EXE
• LC_CTYPE
• GEM_PATH
• GIT_PROMPT_ONLY_IN_REPO
• CONDA_DEFAULT_ENV
• NVM_BIN
• DISPLAY
• LC_TERMINAL
• HGSRC
• SECURITYSESSIONID
• RUBY_VERSION
• PERL_MM_OPT
• COLORTERM
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /usr/local/Cellar/pipenv/2020.8.13/libexec/tools:/usr/local/google-sdks/google-cloud-sdk/bin:/Users/mj/perl5/bin:/Users/mj/.nvm/versions/node/v12.14.1/bin:/Users/mj/.rvm/gems/ruby-2.3.0/bin:/Users/mj/.rvm/gems/ruby-2.3.0@global/bin:/Users/mj/.rvm/rubies/ruby-2.3.0/bin:/opt/miniconda3/bin:/opt/miniconda3/condabin:/Users/mj/bin:/Users/mj/Development/Library/zopatista_setup/bin:/Users/mj/Development/Library/zopatista_setup/audio:/usr/local/Cellar/ruby/2.7.1_2/bin:/Users/mj/.cargo/bin:/Users/mj/.poetry/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/MacGPG2/bin:/usr/local/sbin:/opt/X11/bin:/Applications/Postgres.app/Contents/Versions/latest/bin:/Users/mj/.local/bin:/Users/mj/.rvm/bin:/Applications/Visual Studio Code.app/Contents/Resources/app/bin
• SHELL: /bin/bash
• EDITOR: vim
• LANG: en_GB.UTF-8
• PWD: /Users/mj

---

[conradg]

Thanks for raising this @mjpieters, thought I was going crazy. Downgrading to 2018.11.26 speeds things up massively for me too

[witsch]

Thanks for raising this @mjpieters, thought I was going crazy. Downgrading to 2018.11.26 speeds things up massively for me too

Yes, +1! Thanks for looking into this, Martijn! We had exactly the same issue (lots of big tmp directories), but sorta gave up for the time being… I couldn't really argue much when pipenv lock took 10+ minutes for simply adding another package.

(and hi, btw 👋)

[mgmarino]

A "workaround" that may work (and what we have used from time to time) is to remove the lock file and rerun the lock. This may also hint at the underlying problem, which may have something to do with how pipenv is using the existing lock file during the lock process.

[Can-Sahin]

Deleting the Pipfile.lock speeds things up for sure. I just install all the packages from scratch and run pipenv lock without changing anything and it gets stuck eventhough there is no change at all

[frostming]

Can you have a try on the master branch? I recently disabled the copying of the local source tree.

[mjpieters]

Can you have a try on the master branch? I recently disabled the copying of the local source tree.

Do you mean #4421? That's been part of the 2020.8.13 release that I tested with for this issue. I did note that the tree copying no longer takes place.

I also don't see any changes on master since 2020.8.13 was released that look relevant. Did I miss something?

[frostming]

If I remember correctly, 2018.11.26 didn't resolve transient dependencies of an editable package. You can compare the lock files generated by both versions.

[mjpieters]

If I remember correctly, 2018.11.26 didn't resolve transient dependencies of an editable package. You can compare the lock files generated by both versions.

I see no differences that look problematic. With 2020.8.13.dev0 (master branch) I see markers entries added. The dependencies I see added are ones that were due to other dependencies having been upgraded (I deleted the lock file to see if there was much speed difference).

With the lock file deleted and using 2020.8.13.dev0, a pipenv lock took 10 minutes.

[frostming]

I removed all other dependencies except the editable one. It turns out no significant difference is seen in the time costs of 2018.11.26 and 2020.8.13. It will be great if you can narrow down which package brings the performance degrading.

I created a demo repo to showcase the issue, which you can clone and adjust to reproduce the performance issue:
https://github.com/frostming/pipenv-lock-performance

$ docker build . --build-arg PIPENV_VER=2020.8.13 -t pipenv-bug
$ docker build . --build-arg PIPENV_VER=2018.11.26 -t pipenv-bug

[mjpieters]

Sorry, I meant to get back to this but current project work is keeping me over-utilised. I do still find Pipenv to be excruciatingly slow, and regularly have to delete the lock file just to get a solution, but that's not really very actionable feedback.