pipenv update --outdated crashes with IndexError in Git repo

[sersorrel]

When I run pipenv update --outdated, it crashes with an IndexError, but only if a package is installed in editable mode where the package is also a Git repository.

This looks kind of like pypa/pip#4759? I don't know if you would be willing to merge the PR that apparently fixes it into Pipenv's vendored Pip; alternatively, ignoring lines from Pip that start with # might work, since apparently requirements files are allowed to contain comments, and I guess Pipenv is probably parsing the output of pip freeze.

$ python -m pipenv.help output

Pipenv version: '11.9.0'

Pipenv location: '/home/josh/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv'

Python location: '/home/josh/.local/venvs/pipenv/bin/python'

Other Python installations in PATH:

• 2.7: /usr/bin/python2.7

• 2.7: /usr/bin/python2.7

• 3.5: /usr/bin/python3.5m

• 3.5: /usr/bin/python3.5

• 3.6: /usr/bin/python3.6m

• 3.6: /usr/bin/python3.6

• 2.7.12: /usr/bin/python

• 2.7.12: /usr/bin/python2

• 3.5.2: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.3',
 'os_name': 'posix',
 'platform_machine': 'armv7l',
 'platform_python_implementation': 'CPython',
 'platform_release': '3.10.18',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Sat Feb 24 13:21:23 PST 2018',
 'python_full_version': '3.6.3',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

• GIT_PS1_SHOWDIRTYSTATE
• SSH_AGENT_PID
• XDG_DATA_HOME
• PYENV_ROOT
• SHELL
• TERM
• PIPENV_VENV_IN_PROJECT
• RIPGREP_CONFIG_PATH
• TMUX_VERSION
• NVM_DIR
• USER
• LS_COLORS
• DEBEMAIL
• SSH_AUTH_SOCK
• TMUX
• MAIL
• PATH
• PWD
• EDITOR
• LANG
• TMUX_PANE
• GIT_PS1_SHOWUNTRACKEDFILES
• GPG_TTY
• HOME
• MANPAGER
• SHLVL
• XDG_CONFIG_HOME
• GIT_PS1_SHOWSTASHSTATE
• PIPENV_SHELL_FANCY
• XDG_CACHE_HOME
• LOGNAME
• GIT_PS1_SHOWUPSTREAM
• VISUAL
• TMUX_PLUGIN_MANAGER_PATH
• DEBFULLNAME
• GOPATH
• LESSOPEN
• LESSCLOSE
• _
• OLDPWD
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH
• PYTHONUNBUFFERED

Pipenv–specific environment variables:

• PIPENV_VENV_IN_PROJECT: 1
• PIPENV_SHELL_FANCY: 1

Debug–specific environment variables:

• PATH: /home/josh/go/bin:/home/josh/.pyenv/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/josh/.cargo/bin:/home/josh/.cabal/bin:/home/josh/bin:/home/josh/.local/bin
• SHELL: /bin/bash
• EDITOR: vim
• LANG: en_GB.UTF-8
• PWD: /home/josh/pipenv_test

---

Contents of Pipfile ('/home/josh/pipenv_test/Pipfile'):

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
"e1839a8" = {path = ".", editable = true}

[dev-packages]

[requires]
python_version = "3.6"

Contents of Pipfile.lock ('/home/josh/pipenv_test/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "0e120892e832e8b76faa321dfe89c434016f9d29056cb6f8ecd1cf2dae86895b"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "e1839a8": {
            "editable": true,
            "path": "."
        }
    },
    "develop": {}
}

---

Expected result

The outdated packages should be displayed? (I've actually never used this command before, but I guess it probably shouldn't just crash 😃)

Actual result

$ pipenv update --outdated
Traceback (most recent call last):
  File "/home/josh/.local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/home/josh/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/josh/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/josh/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/josh/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/josh/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/josh/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/josh/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/cli.py", line 731, in update
    core.do_outdated()
  File "/home/josh/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/core.py", line 1698, in do_outdated
    packages.update(convert_deps_from_pip(result))
  File "/home/josh/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/utils.py", line 520, in convert_deps_from_pip
    req = get_requirement(dep)
  File "/home/josh/.local/venvs/pipenv/lib/python3.6/site-packages/pipenv/utils.py", line 127, in get_requirement
    req = [r for r in requirements.parse(dep)][0]
IndexError: list index out of range

I inserted some debugging, and in the final stack frame, the value of dep is "## !! Could not determine repository location"; a couple of frames up in pipenv.core.do_outdated, the value of results is ['## !! Could not determine repository location', 'foo==0.1'].

Steps to replicate

Run in an empty directory:

cat > setup.py << EOF
from setuptools import setup

setup(
    name="foo",
    version="0.1",
)
EOF
pipenv --python 3.6
pipenv install -e .
pipenv update --outdated  # works fine, no output
git init
pipenv update --outdated  # now everything explodes

[techalchemy]

I think this is kind of complicated to be honest mainly due to the VCS involvement. I’ve been sitting on a significant refactor of the requirement parser and this is one of the pieces I am considering. If I put the branch up, maybe you can grab it and see how it works for you. Probably related: #1690

[sersorrel]

Sure, I'm happy to test whatever.

As a quick fix, pipenv.core.do_outdated could remove results from Pip that start with #, via something like:

results = filter(lambda r: not r.startswith("#"), results)

before passing them to convert_deps_from_pip.

Really it should remove comments from within lines as well, but I would be surprised if pip freeze ever emitted those.

[techalchemy]

@anowlcalledjosh sorry for dropping the ball on this, was working through a big datacenter migration this weekend. I looked over that PR, seems like a good idea. I I have time I can bring it in, but feel free to PR it if you like as well. I think we keep commented lines for some reason, would have to dig to find out why

[cjerdonek]

FYI, pypa/pip#4759 is now addressed in pip's master branch.

[techalchemy]

Thanks @cjerdonek :) I added a bunch of custom handling for this in requirementslib which is where I am currently doing some dependency resolution on local editable packages, and we cut over our reliance on pip freeze to a more direct reliance on pkg_resources for the moment, so I think we can actually close this one!