2018.11.14 regression: `pipenv lock -r > ...` creates corrupted requirements file

[Zebradil]

Issue description

I'm trying to re-build an image from the following Dockerfile:

FROM python:3.6-alpine
WORKDIR /opt/app
RUN pip install pipenv
COPY Pipfile .
RUN pipenv lock -r > requirements.txt; pip install -U -r requirements.txt

and getting error on the last step.

Expected result

pip install packages from the requirements.txt successfully.

Actual result

Sending build context to Docker daemon  8.192kB
Step 1/5 : FROM python:3.6-alpine
 ---> 1d981af1e3b4
Step 2/5 : WORKDIR /opt/app
 ---> Using cache
 ---> 1f6e844884d3
Step 3/5 : RUN pip install pipenv
 ---> Using cache
 ---> fe80c45921ee
Step 4/5 : COPY Pipfile .
 ---> 1410d77cb800
Step 5/5 : RUN pipenv lock -r > requirements.txt; pip install -U -r requirements.txt
 ---> Running in 837048256df0
Creating a virtualenv for this project…
Pipfile: /opt/app/Pipfile
Using /usr/local/bin/python (3.6.7) to create virtualenv…
Already using interpreter /usr/local/bin/python
Using base prefix '/usr/local'
New python executable in /root/.local/share/virtualenvs/app-ueEJiAOq/bin/python
Installing setuptools, pip, wheel...
done.
Virtualenv location: /root/.local/share/virtualenvs/app-ueEJiAOq
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (8407de)!
Invalid requirement: '⠋ Running..'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/packaging/requirements.py", line 93, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1654, in parseString
    raise exc
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1644, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 3417, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 3739, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 3400, in parseImpl
    loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1406, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 2711, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pip._vendor.pyparsing.ParseException: Expected W:(abcd...) (at char 0), (line:1, col:1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/req/constructors.py", line 253, in install_req_from_line
    req = Requirement(req)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/packaging/requirements.py", line 96, in __init__
    requirement_string[e.loc:e.loc + 8], e.msg
pip._vendor.packaging.requirements.InvalidRequirement: Parse error at "'⠋ Runnin'": Expected W:(abcd...)

The command '/bin/sh -c pipenv lock -r > requirements.txt; pip install -U -r requirements.txt' returned a non-zero code: 1

Steps to replicate

1. Create Dockerfile with the contents from the above
2. Create Pipfile with the following content:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pylint = "*"

[dev-packages]

[requires]
python_version = "3.6"

3. Run docker build -t pipenv-test .

---

docker run python:3.6-alpine ash -c 'pip install pipenv; pipenv --support' produces:

$ pipenv --support

Pipenv version: '2018.11.14'

Pipenv location: '/usr/local/lib/python3.6/site-packages/pipenv'

Python location: '/usr/local/bin/python'

Python installations found:

• 3.6.7: /usr/local/bin/python
• 3.6.7: /usr/local/bin/python3.6m

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.7',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.9.93-boot2docker',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Thu Jul 19 18:29:50 UTC 2018',
 'python_full_version': '3.6.7',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

• HOSTNAME
• PYTHON_PIP_VERSION
• SHLVL
• HOME
• GPG_KEY
• PATH
• LANG
• PYTHON_VERSION
• PWD
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
• LANG: C.UTF-8
• PWD: /

---

[duplicate-issues]

Hey @Zebradil,

We did a quick check and this issue looks very darn similar to

• #3238 - 2018.11.14 regression: support for editables within an src dir broken
• #3237 - Fix race condition when installing 2+ editable non-VCS pkgs at once
• #3236 - 2018.11.14 regression: Installing Multiple local editable packages leaves some unusable

This could be a coincidence, but if any of these issues solves your problem then I did a good job 😄

If not, the maintainers will get to this issue shortly.

Cheers,
Your Friendly Neighborhood ProBot

[Zebradil]

Tried the same with the previous version (2018.10.13) — works as expected.
I also checked the content of requirements.txt:

✔ Complete .
✔ Success! .
-i https://pypi.org/simple
astroid==2.0.4
isort==4.3.4
lazy-object-proxy==1.3.1
mccabe==0.6.1
pylint==2.1.1
six==1.11.0
typed-ast==1.1.0 ; python_version < '3.7' and implementation_name == 'cpython'
wrapt==1.10.11

Look at first two lines. However, I'm not sure why in the error message I see '⠋ Runnin'.

[karlsvec]

When you run pipenv lock -r, and no virtualenv exists yet, Pipenv will create one. On 2018.11.14, it looks like the animated spinner goes to stdout (instead of stderr with the rest of Pipenv's verbose/informative output). The spinner output ends up in your requirements.txt, which is why you see Invalid requirement: '⠋ Running..'.

Some context: I use the following pattern for bundling Python dependencies as part of an archive builder script for AWS Lambda:

pip3 install -r <(pipenv lock -r) --target .

As a workaround, I was able to restore the old behavior with PIPENV_NOSPIN=1 pipenv --bare lock -r 2>/dev/null in place of pipenv lock -r.

[techalchemy]

hrm, this is a good observation. We need to fix this

[techalchemy]

(I believe only the frozen final frame of the spinner goes to stdout)

[techalchemy]

I have been attempting to track this down but am actually struggling to find the frame which gets written to stdout

[techalchemy]

Got it! sarugaku/vistir#41

Successfully installed certifi-2018.10.15 pipenv-2018.11.15.dev0 virtualenv-16.1.0 virtualenv-clone-0.4.0
Removing intermediate container 032e11add394
 ---> c8b1be7e75a5
Step 6/6 : RUN pipenv lock -r > requirements.txt; pip install -U -r requirements.txt
 ---> Running in 42668600ea5d
Creating a virtualenv for this project…
Pipfile: /opt/app/Pipfile
Running...
Using /usr/local/bin/python (3.6.7) to create virtualenv…
Running... virtual environment...
⠴ Creating virtual environment...Already using interpreter /usr/local/bin/python
Using base prefix '/usr/local'
New python executable in /root/.local/share/virtualenvs/app-ueEJiAOq/bin/python
Installing setuptools, pip, wheel...
done.

Removing intermediate container 42668600ea5d
 ---> 312bc682aa72
Successfully built 312bc682aa72
Successfully tagged pipenv-test:latest

[Locustv2]

@Zebradil - i am having the same issue. Can you share what you did?
I usually get some other errors with pipenv and when i downgrade the pip version using pipenv run pip install pip==18.0 then it works. But i haven't tested for this issue yet.

[Zebradil]

As a workaround you can just downgrade pipenv version:

pip install pipenv==2018.10.13

To check all available versions use

pip install pipenv==