Difference in binary builds between pip 19.3 and 19.3.1

[iamareebjamal]

Environment

• pip version: 19.3.1
• Python version: 3.7.4/3.7.5
• OS: Linux (alpine)

Description

We use multistage builds to make our docker image compact.

https://github.com/fossasia/open-event-server/blob/b7ba05cfaefdc257e9ae07d264044668dadffce5/Dockerfile

We are using python:3.7-alpine as base image which had pip 19.3 pre-installed.
Installing dependencies with build tools like gcc and others in the initial stage and then copying them to the later stage. This is a common practice and recommended in the docker community.

We installed the build tools and requirements using apt and pip which installed correctly using this command:

pip install --install-option="--prefix=/install" setuptools && \
    LIBRARY_PATH=/lib:/usr/lib pip install --install-option="--prefix=/install" -r /requirements.txt

Then, we copy the installed libraries to the next stage using the command
COPY --from=builder /install /usr/local.

Now, all the libraries installed in the previous stage should be present in the next stage. Which was the case until today.

When we built the release container, it did not have importlib_metadata and threw ModuleError.

After hours of debugging, and jumping through diffs of previously working diff environment, we found that it was not installed at all along with some other libraries. More time consuming debugging later, I found out a difference between environments. The broken environment had Python 3.7.5 and pip 19.3.1 and working one had Python 3.7.4 and pip 19.3.

A quick matrix test on Python 3.7.4 with pip 19.3.1 and Python 3.7.5 with pip 19.3 showed that indeed the problem was with pip and not Python version.

After analyzing the missing dependencies, we found out that the only ones missing were the ones that were built during the previous stage. One of which was the aforementioned importlib_metadata along with several other built libraries.

Expected behavior

pip 19.3.1 which is a patch version should not break backward compatibility and retain the behavior as 19.3.

How to Reproduce

1. Disable wheel packages like in alpine and build the libraries
2. Copy from one install to another
3. Find the library to be missing

OR

1. Use the above-linked dockerfile.
2. Build it using this command - docker build . -t eventyay/open-event-server
3. Run the shell using this command - docker run -it --entrypoint /bin/sh eventyay/open-event-server -s
4. Run pip freeze | grep import
5. Change the version from python:3.7-alpine to python:3.7.4-alpine or add pip install pip==19.3 in the first RUN step to lock pip, and repeat steps 2-4 and you'll find the library

Output

import importlib_metadata
ModuleNotFoundError: No module named 'importlib_metadata'

[xavfernandez]

Hello @iamareebjamal,

If I understand correctly your Dockerfile basically installs https://github.com/fossasia/open-event-server/blob/b7ba05cfaefdc257e9ae07d264044668dadffce5/requirements/common.txt and nothing in there depends on importlib_metadata so having it installed was a bug that was fixed in the patch version ?

[iamareebjamal]

No, the problem is not related to importlib_metadata. You change importlib_metadata to any library which is built during install

Also, importlib_metadata is a transitive dependency of several of our dependencies.

I can provide a pipdeptree if it helps. Many direct dependencies weren't installed as well

[xavfernandez]

I can provide a pipdeptree if it helps.

It would 👍

[iamareebjamal]

https://www.diffchecker.com/ryCo05HG

19.3

info

APScheduler==3.6.1
  - pytz [required: Any, installed: 2019.3]
  - setuptools [required: >=0.7, installed: 41.4.0]
  - six [required: >=1.4.0, installed: 1.12.0]
  - tzlocal [required: >=1.2, installed: 2.0.0]
    - pytz [required: Any, installed: 2019.3]
arrow==0.15.2
  - python-dateutil [required: Any, installed: 2.8.0]
    - six [required: >=1.5, installed: 1.12.0]
bleach==3.1.0
  - six [required: >=1.9.0, installed: 1.12.0]
  - webencodings [required: Any, installed: 0.5.1]
blinker==1.4
diff-match-patch==20181111
elasticsearch-dsl==7.0.0
  - elasticsearch [required: >=7.0.0,<8.0.0, installed: 7.0.5]
    - urllib3 [required: >=1.21.1, installed: 1.25.6]
  - python-dateutil [required: Any, installed: 2.8.0]
    - six [required: >=1.5, installed: 1.12.0]
  - six [required: Any, installed: 1.12.0]
envparse==0.2.0
eventlet==0.25.1
  - dnspython [required: >=1.15.0, installed: 1.16.0]
  - greenlet [required: >=0.3, installed: 0.4.15]
  - monotonic [required: >=1.4, installed: 1.5]
  - six [required: >=1.10.0, installed: 1.12.0]
factory-boy==2.12.0
  - Faker [required: >=0.7.0, installed: 2.0.3]
    - python-dateutil [required: >=2.4, installed: 2.8.0]
      - six [required: >=1.5, installed: 1.12.0]
    - six [required: >=1.10, installed: 1.12.0]
    - text-unidecode [required: ==1.3, installed: 1.3]
Flask-Admin==1.5.4
  - Flask [required: >=0.7, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - wtforms [required: Any, installed: 2.2.1]
Flask-Caching==1.7.2
  - Flask [required: Any, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
Flask-CeleryExt==0.3.2
  - celery [required: >=3.1, installed: 4.3.0]
    - billiard [required: >=3.6.0,<4.0, installed: 3.6.1.0]
    - kombu [required: >=4.4.0,<5.0, installed: 4.6.5]
      - amqp [required: ==2.5.1, installed: 2.5.1]
        - vine [required: >=1.1.3,<5.0.0a1, installed: 1.3.0]
      - importlib-metadata [required: >=0.18, installed: 0.23]
        - zipp [required: >=0.5, installed: 0.6.0]
          - more-itertools [required: Any, installed: 7.2.0]
    - pytz [required: >dev, installed: 2019.3]
    - vine [required: >=1.3.0, installed: 1.3.0]
  - celery [required: >=4.3, installed: 4.3.0]
    - billiard [required: >=3.6.0,<4.0, installed: 3.6.1.0]
    - kombu [required: >=4.4.0,<5.0, installed: 4.6.5]
      - amqp [required: ==2.5.1, installed: 2.5.1]
        - vine [required: >=1.1.3,<5.0.0a1, installed: 1.3.0]
      - importlib-metadata [required: >=0.18, installed: 0.23]
        - zipp [required: >=0.5, installed: 0.6.0]
          - more-itertools [required: Any, installed: 7.2.0]
    - pytz [required: >dev, installed: 2019.3]
    - vine [required: >=1.3.0, installed: 1.3.0]
  - Flask [required: >=0.10, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
Flask-Cors==3.0.8
  - Flask [required: >=0.9, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - Six [required: Any, installed: 1.12.0]
Flask-Elasticsearch==0.2.5
  - Elasticsearch [required: Any, installed: 7.0.5]
    - urllib3 [required: >=1.21.1, installed: 1.25.6]
  - Flask [required: Any, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
Flask-JWT-Extended==3.23.0
  - Flask [required: >=1.0, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - PyJWT [required: >=1.6.4, installed: 1.7.1]
  - six [required: Any, installed: 1.12.0]
  - Werkzeug [required: >=0.14, installed: 0.16.0]
Flask-Limiter==1.1.0
  - Flask [required: >=0.8, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - limits [required: Any, installed: 1.3]
    - six [required: >=1.4.1, installed: 1.12.0]
  - six [required: >=1.4.1, installed: 1.12.0]
Flask-Login==0.4.1
  - Flask [required: Any, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
Flask-Migrate==2.5.2
  - alembic [required: >=0.7, installed: 1.2.1]
    - Mako [required: Any, installed: 1.1.0]
      - MarkupSafe [required: >=0.9.2, installed: 1.1.1]
    - python-dateutil [required: Any, installed: 2.8.0]
      - six [required: >=1.5, installed: 1.12.0]
    - python-editor [required: >=0.3, installed: 1.0.4]
    - SQLAlchemy [required: >=1.1.0, installed: 1.3.10]
  - Flask [required: >=0.9, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - Flask-SQLAlchemy [required: >=1.0, installed: 2.4.1]
    - Flask [required: >=0.10, installed: 1.1.1]
      - click [required: >=5.1, installed: 7.0]
      - itsdangerous [required: >=0.24, installed: 1.1.0]
      - Jinja2 [required: >=2.10.1, installed: 2.10.3]
        - MarkupSafe [required: >=0.23, installed: 1.1.1]
      - Werkzeug [required: >=0.15, installed: 0.16.0]
    - SQLAlchemy [required: >=0.8.0, installed: 1.3.10]
flask-redis==0.4.0
  - Flask [required: >=0.8, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - redis [required: >=2.7.6, installed: 3.3.11]
Flask-REST-JSONAPI==0.12.6
  - Flask [required: >=0.11, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - marshmallow [required: >=2.15.1,<3, installed: 2.15.6]
  - marshmallow-jsonapi [required: Any, installed: 0.22.0]
    - marshmallow [required: >=2.15.2, installed: 2.15.6]
  - six [required: Any, installed: 1.12.0]
  - sqlalchemy [required: Any, installed: 1.3.10]
Flask-Script==2.0.6
  - Flask [required: Any, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
Flask-Scrypt==0.1.3.6
  - Flask [required: Any, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - scrypt [required: Any, installed: 0.8.13]
forex-python==1.5
  - requests [required: Any, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
  - simplejson [required: Any, installed: 3.16.0]
geoip2==2.9.0
  - maxminddb [required: >=1.4.0, installed: 1.5.1]
  - requests [required: >=2.9, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
google-compute-engine==2.8.13
  - boto [required: Any, installed: 2.49.0]
  - distro [required: Any, installed: 1.4.0]
  - setuptools [required: Any, installed: 41.4.0]
gunicorn==19.9.0
healthcheck==1.3.3
humanize==0.5.1
icalendar==4.0.3
  - python-dateutil [required: Any, installed: 2.8.0]
    - six [required: >=1.5, installed: 1.12.0]
  - pytz [required: Any, installed: 2019.3]
marrow.mailer==4.0.3
  - marrow.util [required: <2.0, installed: 1.2.3]
oauth2==1.9.0.post1
  - httplib2 [required: Any, installed: 0.14.0]
omise==0.8.1
  - requests [required: >=2.12.1, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
paypalrestsdk==1.13.1
  - pyopenssl [required: >=0.15, installed: 19.0.0]
    - cryptography [required: >=2.3, installed: 2.8]
      - cffi [required: >=1.8,!=1.11.3, installed: 1.13.0]
        - pycparser [required: Any, installed: 2.19]
      - six [required: >=1.4.1, installed: 1.12.0]
    - six [required: >=1.5.2, installed: 1.12.0]
  - requests [required: >=1.0.0, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
  - six [required: >=1.0.0, installed: 1.12.0]
pipdeptree==0.13.2
  - pip [required: >=6.0.0, installed: 19.3]
psycopg2-binary==2.8.3
pycountry==19.8.18
pycryptodome==3.9.0
python-dotenv==0.10.3
python-geoip-geolite2==2015.303
  - python-geoip [required: Any, installed: 1.2]
python-magic==0.4.15
python-pentabarf-xml==0.19
PyYAML==5.1.2
qrcode==6.1
  - six [required: Any, installed: 1.12.0]
requests-oauthlib==1.2.0
  - oauthlib [required: >=3.0.0, installed: 3.1.0]
  - requests [required: >=2.0.0, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
sendgrid==6.1.0
  - python-http-client [required: >=3.2.1, installed: 3.2.1]
sentry-sdk==0.13.0
  - certifi [required: Any, installed: 2019.9.11]
  - urllib3 [required: >=1.9, installed: 1.25.6]
SQLAlchemy-Continuum==1.3.9
  - SQLAlchemy [required: >=1.0.8, installed: 1.3.10]
  - SQLAlchemy-Utils [required: >=0.30.12, installed: 0.34.2]
    - six [required: Any, installed: 1.12.0]
    - SQLAlchemy [required: >=1.0, installed: 1.3.10]
stripe==2.37.2
  - requests [required: >=2.20, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
unicode-slugify==0.1.3
  - six [required: Any, installed: 1.12.0]
  - unidecode [required: Any, installed: 1.1.1]
wheel==0.33.6
xhtml2pdf==0.2.3
  - html5lib [required: >=1.0, installed: 1.0.1]
    - six [required: >=1.9, installed: 1.12.0]
    - webencodings [required: Any, installed: 0.5.1]
  - Pillow [required: Any, installed: 6.2.0]
  - pyPdf2 [required: Any, installed: 1.26.0]
  - reportlab [required: >=3.0, installed: 3.5.31]
    - pillow [required: >=4.0.0, installed: 6.2.0]
  - six [required: Any, installed: 1.12.0]

19.3.1

Warning!!! Possibly conflicting dependencies found:
* pyOpenSSL==19.0.0
 - cryptography [required: >=2.3, installed: ?]
* kombu==4.6.5
 - importlib-metadata [required: >=0.18, installed: ?]
* icalendar==4.0.3
 - python-dateutil [required: Any, installed: ?]
* Faker==2.0.3
 - python-dateutil [required: >=2.4, installed: ?]
* elasticsearch-dsl==7.0.0
 - python-dateutil [required: Any, installed: ?]
* arrow==0.15.2
 - python-dateutil [required: Any, installed: ?]
* alembic==1.2.1
 - python-dateutil [required: Any, installed: ?]
------------------------------------------------------------------------
APScheduler==3.6.1
  - pytz [required: Any, installed: 2019.3]
  - setuptools [required: >=0.7, installed: 41.4.0]
  - six [required: >=1.4.0, installed: 1.12.0]
  - tzlocal [required: >=1.2, installed: 2.0.0]
    - pytz [required: Any, installed: 2019.3]
arrow==0.15.2
  - python-dateutil [required: Any, installed: ?]
bleach==3.1.0
  - six [required: >=1.9.0, installed: 1.12.0]
  - webencodings [required: Any, installed: 0.5.1]
blinker==1.4
cffi==1.13.0
  - pycparser [required: Any, installed: 2.19]
diff-match-patch==20181111
elasticsearch-dsl==7.0.0
  - elasticsearch [required: >=7.0.0,<8.0.0, installed: 7.0.5]
    - urllib3 [required: >=1.21.1, installed: 1.25.6]
  - python-dateutil [required: Any, installed: ?]
  - six [required: Any, installed: 1.12.0]
envparse==0.2.0
eventlet==0.25.1
  - dnspython [required: >=1.15.0, installed: 1.16.0]
  - greenlet [required: >=0.3, installed: 0.4.15]
  - monotonic [required: >=1.4, installed: 1.5]
  - six [required: >=1.10.0, installed: 1.12.0]
factory-boy==2.12.0
  - Faker [required: >=0.7.0, installed: 2.0.3]
    - python-dateutil [required: >=2.4, installed: ?]
    - six [required: >=1.10, installed: 1.12.0]
    - text-unidecode [required: ==1.3, installed: 1.3]
Flask-Admin==1.5.4
  - Flask [required: >=0.7, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - wtforms [required: Any, installed: 2.2.1]
Flask-Caching==1.7.2
  - Flask [required: Any, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
Flask-CeleryExt==0.3.2
  - celery [required: >=3.1, installed: 4.3.0]
    - billiard [required: >=3.6.0,<4.0, installed: 3.6.1.0]
    - kombu [required: >=4.4.0,<5.0, installed: 4.6.5]
      - amqp [required: ==2.5.1, installed: 2.5.1]
        - vine [required: >=1.1.3,<5.0.0a1, installed: 1.3.0]
      - importlib-metadata [required: >=0.18, installed: ?]
    - pytz [required: >dev, installed: 2019.3]
    - vine [required: >=1.3.0, installed: 1.3.0]
  - celery [required: >=4.3, installed: 4.3.0]
    - billiard [required: >=3.6.0,<4.0, installed: 3.6.1.0]
    - kombu [required: >=4.4.0,<5.0, installed: 4.6.5]
      - amqp [required: ==2.5.1, installed: 2.5.1]
        - vine [required: >=1.1.3,<5.0.0a1, installed: 1.3.0]
      - importlib-metadata [required: >=0.18, installed: ?]
    - pytz [required: >dev, installed: 2019.3]
    - vine [required: >=1.3.0, installed: 1.3.0]
  - Flask [required: >=0.10, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
Flask-Cors==3.0.8
  - Flask [required: >=0.9, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - Six [required: Any, installed: 1.12.0]
Flask-Elasticsearch==0.2.5
  - Elasticsearch [required: Any, installed: 7.0.5]
    - urllib3 [required: >=1.21.1, installed: 1.25.6]
  - Flask [required: Any, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
Flask-JWT-Extended==3.23.0
  - Flask [required: >=1.0, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - PyJWT [required: >=1.6.4, installed: 1.7.1]
  - six [required: Any, installed: 1.12.0]
  - Werkzeug [required: >=0.14, installed: 0.16.0]
Flask-Limiter==1.1.0
  - Flask [required: >=0.8, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - limits [required: Any, installed: 1.3]
    - six [required: >=1.4.1, installed: 1.12.0]
  - six [required: >=1.4.1, installed: 1.12.0]
Flask-Login==0.4.1
  - Flask [required: Any, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
Flask-Migrate==2.5.2
  - alembic [required: >=0.7, installed: 1.2.1]
    - Mako [required: Any, installed: 1.1.0]
      - MarkupSafe [required: >=0.9.2, installed: 1.1.1]
    - python-dateutil [required: Any, installed: ?]
    - python-editor [required: >=0.3, installed: 1.0.4]
    - SQLAlchemy [required: >=1.1.0, installed: 1.3.10]
  - Flask [required: >=0.9, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - Flask-SQLAlchemy [required: >=1.0, installed: 2.4.1]
    - Flask [required: >=0.10, installed: 1.1.1]
      - click [required: >=5.1, installed: 7.0]
      - itsdangerous [required: >=0.24, installed: 1.1.0]
      - Jinja2 [required: >=2.10.1, installed: 2.10.3]
        - MarkupSafe [required: >=0.23, installed: 1.1.1]
      - Werkzeug [required: >=0.15, installed: 0.16.0]
    - SQLAlchemy [required: >=0.8.0, installed: 1.3.10]
Flask-REST-JSONAPI==0.12.6
  - Flask [required: >=0.11, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - marshmallow [required: >=2.15.1,<3, installed: 2.15.6]
  - marshmallow-jsonapi [required: Any, installed: 0.22.0]
    - marshmallow [required: >=2.15.2, installed: 2.15.6]
  - six [required: Any, installed: 1.12.0]
  - sqlalchemy [required: Any, installed: 1.3.10]
Flask-Script==2.0.6
  - Flask [required: Any, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
Flask-Scrypt==0.1.3.6
  - Flask [required: Any, installed: 1.1.1]
    - click [required: >=5.1, installed: 7.0]
    - itsdangerous [required: >=0.24, installed: 1.1.0]
    - Jinja2 [required: >=2.10.1, installed: 2.10.3]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.15, installed: 0.16.0]
  - scrypt [required: Any, installed: 0.8.13]
forex-python==1.5
  - requests [required: Any, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
  - simplejson [required: Any, installed: 3.16.0]
geoip2==2.9.0
  - maxminddb [required: >=1.4.0, installed: 1.5.1]
  - requests [required: >=2.9, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
google-compute-engine==2.8.13
  - boto [required: Any, installed: 2.49.0]
  - distro [required: Any, installed: 1.4.0]
  - setuptools [required: Any, installed: 41.4.0]
gunicorn==19.9.0
healthcheck==1.3.3
humanize==0.5.1
icalendar==4.0.3
  - python-dateutil [required: Any, installed: ?]
  - pytz [required: Any, installed: 2019.3]
marrow.mailer==4.0.3
  - marrow.util [required: <2.0, installed: 1.2.3]
more-itertools==7.2.0
oauth2==1.9.0.post1
  - httplib2 [required: Any, installed: 0.14.0]
omise==0.8.1
  - requests [required: >=2.12.1, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
paypalrestsdk==1.13.1
  - pyopenssl [required: >=0.15, installed: 19.0.0]
    - cryptography [required: >=2.3, installed: ?]
    - six [required: >=1.5.2, installed: 1.12.0]
  - requests [required: >=1.0.0, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
  - six [required: >=1.0.0, installed: 1.12.0]
pipdeptree==0.13.2
  - pip [required: >=6.0.0, installed: 19.3.1]
psycopg2-binary==2.8.3
pycountry==19.8.18
pycryptodome==3.9.0
python-dotenv==0.10.3
python-geoip-geolite2==2015.303
  - python-geoip [required: Any, installed: 1.2]
python-magic==0.4.15
python-pentabarf-xml==0.19
PyYAML==5.1.2
qrcode==6.1
  - six [required: Any, installed: 1.12.0]
redis==3.3.11
requests-oauthlib==1.2.0
  - oauthlib [required: >=3.0.0, installed: 3.1.0]
  - requests [required: >=2.0.0, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
sendgrid==6.1.0
  - python-http-client [required: >=3.2.1, installed: 3.2.1]
sentry-sdk==0.13.0
  - certifi [required: Any, installed: 2019.9.11]
  - urllib3 [required: >=1.9, installed: 1.25.6]
SQLAlchemy-Continuum==1.3.9
  - SQLAlchemy [required: >=1.0.8, installed: 1.3.10]
  - SQLAlchemy-Utils [required: >=0.30.12, installed: 0.34.2]
    - six [required: Any, installed: 1.12.0]
    - SQLAlchemy [required: >=1.0, installed: 1.3.10]
stripe==2.37.2
  - requests [required: >=2.20, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.9.11]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
unicode-slugify==0.1.3
  - six [required: Any, installed: 1.12.0]
  - unidecode [required: Any, installed: 1.1.1]
wheel==0.33.6
xhtml2pdf==0.2.3
  - html5lib [required: >=1.0, installed: 1.0.1]
    - six [required: >=1.9, installed: 1.12.0]
    - webencodings [required: Any, installed: 0.5.1]
  - Pillow [required: Any, installed: 6.2.0]
  - pyPdf2 [required: Any, installed: 1.26.0]
  - reportlab [required: >=3.0, installed: 3.5.31]
    - pillow [required: >=4.0.0, installed: 6.2.0]
  - six [required: Any, installed: 1.12.0]

[pradyunsg]

@iamareebjamal Hi! I've edited your bug report to read more easily (shorter sentences, imperative tone) and to tone down some of the wording used.

Using all caps (equivalent to shouting via text) isn't a good move when you're asking for help from volunteers who do this stuff because they find it fun. :)

If you have any concerns with that, please file a new issue on pip or drop me an email on {username}@gmail.com.

[pradyunsg]

Looking at the diff between the release tags: 19.3...19.3.1

I think what broke your usecase was the fact that we fixed a major bug in PEP 517 based builds. I'd have liked to have made that fix in 19.3 but I didn't have the bandwidth to and AFAICT, that fix was fairly innocuous. I'm not sure there's anything actionable now for pip, other than acknowledging that cached docker containers that were built earlier, should be invalidated since a bug in pip's build process was fixed which affects how we can use the cache.

Sorry if that sounds a little bleak but well, pip's used in so many ways that at this point, every non-refactoring patch breaks something, usually in ways that no one thought of when working on this stuff. 🤷🏻‍♂️

[chrahunt]

@iamareebjamal, in your Dockerfile please pass an explicit --prefix=/install argument to pip itself instead of --install-options='--prefix=/install'. Pip will internally update --install-options as well as take other necessary measures to ensure installed packages respect the --prefix argument. I tested this locally and the resulting image (containing dependencies installed with pip 19.3.1) had the same dependencies as the image built with dependencies installed with 19.3. You'll probably also want to pass --no-warn-script-location to reduce some of the noise which is irrelevant to your use case.

There was a bug fixed in #6606 (released in 19.3.1) as mentioned by @pradyunsg that now correctly builds projects that use PEP 517 without implicitly going through setuptools. This fix was causing an issue in your case because the packages that were now using PEP 517 were not picking up the prefix directory passed via --install-options.

[iamareebjamal]

Sorry if it felt like I was using caps to emphasize or shout. I was simply referring to MAJOR.MINOR.PATCH in semver context, not for shouting