Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unnecessary downloading of dependencies when they are constrained #9232

Closed
potiuk opened this issue Dec 6, 2020 · 3 comments · Fixed by #9241
Closed

Unnecessary downloading of dependencies when they are constrained #9232

potiuk opened this issue Dec 6, 2020 · 3 comments · Fixed by #9241
Labels
type: bug A confirmed bug or unintended behavior

Comments

@potiuk
Copy link
Contributor

potiuk commented Dec 6, 2020

When we try to install 1.10.14rc1 release of Airflow with fully consistent set of requirements, At least google-cloud-bigquery and oauthlib libraries are needlessly downloaded (all versions) despite being properly constrained in https://github.com/apache/airflow/blob/constraints-1.10.14/constraints-3.6.txt

If you look to the file you will see:

  • google-cloud-bigquery==2.5.0
  • oauthlib==3.1.0

Running it with old resolver or pip 20.2.4 works perfectly fine:

pyenv virtualenv 3.6.12 test-new-resolver4
pyenv activate test-new-resolver4
pip --upgrade pip=20.2.4
pip install "apache-airflow[devel_ci]==1.10.14rc1" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1-10/constraints-3.6.txt"

When I run pip check I got:

[jarek:~/code/airflow] [test-new-resolver4] add-per-provider-javascript(+0/-0,+12/-2)+ 1m8s ± pip check
No broken requirements found.

Then it fails with 20.3:

pip install --upgrade pip
pip --version
pip 20.3.1 from /home/jarek/.pyenv/versions/3.6.12/envs/test-new-resolver4/lib/python3.6/site-packages/pip (python 3.6)
pip install "apache-airflow[devel_ci]==1.10.14rc1" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1-10/constraints-3.6.txt"

What we get (despite having both google-cloud-bigquery==2.5.0:

Collecting google-cloud-bigquery[bqstorage,pandas]<3.0.0dev,>=1.11.1
  Using cached google_cloud_bigquery-2.4.0-py2.py3-none-any.whl (210 kB)
  Using cached google_cloud_bigquery-2.3.1-py2.py3-none-any.whl (208 kB)
  Using cached google_cloud_bigquery-2.2.0-py2.py3-none-any.whl (200 kB)
  Using cached google_cloud_bigquery-2.1.0-py2.py3-none-any.whl (190 kB)
  Using cached google_cloud_bigquery-2.0.0-py2.py3-none-any.whl (193 kB)
  Using cached google_cloud_bigquery-1.28.0-py2.py3-none-any.whl (187 kB)
  Using cached google_cloud_bigquery-1.27.2-py2.py3-none-any.whl (172 kB)
  Using cached google_cloud_bigquery-1.26.1-py2.py3-none-any.whl (170 kB)
  Using cached google_cloud_bigquery-1.26.0-py2.py3-none-any.whl (170 kB)
  Using cached google_cloud_bigquery-1.25.0-py2.py3-none-any.whl (169 kB)
  Using cached google_cloud_bigquery-1.24.0-py2.py3-none-any.whl (165 kB)
  Using cached google_cloud_bigquery-1.23.1-py2.py3-none-any.whl (162 kB)
  Using cached google_cloud_bigquery-1.23.0-py2.py3-none-any.whl (162 kB)
  Using cached google_cloud_bigquery-1.22.0-py2.py3-none-any.whl (161 kB)
  Using cached google_cloud_bigquery-1.21.0-py2.py3-none-any.whl (158 kB)
  Using cached google_cloud_bigquery-1.20.0-py2.py3-none-any.whl (154 kB)
  Using cached google_cloud_bigquery-1.19.1-py2.py3-none-any.whl (149 kB)
  Using cached google_cloud_bigquery-1.19.0-py2.py3-none-any.whl (149 kB)
  Using cached google_cloud_bigquery-1.18.1-py2.py3-none-any.whl (146 kB)
  Using cached google_cloud_bigquery-1.18.0-py2.py3-none-any.whl (146 kB)
  Using cached google_cloud_bigquery-1.17.1-py2.py3-none-any.whl (142 kB)
  Using cached google_cloud_bigquery-1.17.0-py2.py3-none-any.whl (140 kB)
  Using cached google_cloud_bigquery-1.16.1-py2.py3-none-any.whl (140 kB)
  Using cached google_cloud_bigquery-1.16.0-py2.py3-none-any.whl (139 kB)
  Using cached google_cloud_bigquery-1.15.1-py2.py3-none-any.whl (137 kB)
  Using cached google_cloud_bigquery-1.15.0-py2.py3-none-any.whl (133 kB)
  Using cached google_cloud_bigquery-1.14.1-py2.py3-none-any.whl (137 kB)
  Using cached google_cloud_bigquery-1.14.0-py2.py3-none-any.whl (132 kB)
  Using cached google_cloud_bigquery-1.13.1-py2.py3-none-any.whl (137 kB)
  Using cached google_cloud_bigquery-1.13.0-py2.py3-none-any.whl (132 kB)
  Using cached google_cloud_bigquery-1.12.2-py2.py3-none-any.whl (137 kB)
  Using cached google_cloud_bigquery-1.12.1-py2.py3-none-any.whl (130 kB)
  Using cached google_cloud_bigquery-1.12.0-py2.py3-none-any.whl (130 kB)
  Using cached google_cloud_bigquery-1.11.4-py2.py3-none-any.whl (134 kB)
  Using cached google_cloud_bigquery-1.11.3-py2.py3-none-any.whl (127 kB)
  Using cached google_cloud_bigquery-1.11.2-py2.py3-none-any.whl (127 kB)
  Using cached google_cloud_bigquery-1.11.1-py2.py3-none-any.whl (127 kB)

And then despite Having oauthlib==3.1.0 in the constrains, we get:

Collecting oauthlib[signedtoken]>=1.0.0
  Using cached oauthlib-3.0.2-py2.py3-none-any.whl (143 kB)
  Using cached oauthlib-3.0.1-py2.py3-none-any.whl (142 kB)
  Using cached oauthlib-3.0.0-py2.py3-none-any.whl (142 kB)
  Using cached oauthlib-2.1.0-py2.py3-none-any.whl (121 kB)
  Using cached oauthlib-2.0.7-py2.py3-none-any.whl (124 kB)
  Using cached oauthlib-2.0.6.tar.gz (127 kB)
  Using cached oauthlib-2.0.5.tar.gz (129 kB)
  Using cached oauthlib-2.0.4.tar.gz (127 kB)
  Using cached oauthlib-2.0.3.tar.gz (127 kB)
  Using cached oauthlib-2.0.2.tar.gz (125 kB)
  Using cached oauthlib-2.0.1.tar.gz (122 kB)
  Using cached oauthlib-2.0.0.tar.gz (122 kB)
  Using cached oauthlib-1.1.2.tar.gz (111 kB)
  Using cached oauthlib-1.1.1.tar.gz (108 kB)
  Downloading oauthlib-1.0.3.tar.gz (109 kB)
     |████████████████████████████████| 109 kB 2.1 MB/s 
  Downloading oauthlib-1.0.2.tar.gz (108 kB)
     |████████████████████████████████| 108 kB 4.7 MB/s 
  Downloading oauthlib-1.0.1.tar.gz (108 kB)
     |████████████████████████████████| 108 kB 4.7 MB/s 
  Downloading oauthlib-1.0.0.tar.gz (108 kB)
     |████████████████████████████████| 108 kB 4.1 MB/s 
  Using cached oauthlib-1.1.0-1.tar.gz (106 kB)
ERROR: Requested oauthlib[signedtoken]>=1.0.0 from https://files.pythonhosted.org/packages/e5/54/4f96c51b171cf3a64a04b8c5167268803205bc5943b5cdf70bd770727b88/oauthlib-1.1.0-1.tar.gz#sha256=0f786c5573248a38efa86c48c59c0c93140ac836ab2a246aeefd8f9039e999ba (from jira->apache-airflow[devel_ci]==1.10.14rc1) has different version in metadata: '1.1.0'

And installation crashes here. This crash was originally raised in #9203 and it is likely workarounded (not solved yet - this is only possible because we asked oauthlib maintainers to yank the release and they kindly agreed) in #9226 .

I would love to know when we can expect fix to that.

@potiuk
Copy link
Contributor Author

potiuk commented Dec 6, 2020

CC: @pradyunsg

@uranusjr
Copy link
Member

uranusjr commented Dec 7, 2020

Fix proposed in #9241.

@potiuk
Copy link
Contributor Author

potiuk commented Dec 7, 2020

Cool!

bors bot referenced this issue in duckinator/emanate Dec 15, 2020
204: Update pip to 20.3.3 r=duckinator a=pyup-bot


This PR updates [pip](https://pypi.org/project/pip) from **20.3.1** to **20.3.3**.



<details>
  <summary>Changelog</summary>
  
  
   ### 20.3.3
   ```
   ===================

Bug Fixes
---------

- Revert &quot;Skip candidate not providing valid metadata&quot;, as that caused pip to be overeager about downloading from the package index. (`9264 &lt;https://github.com/pypa/pip/issues/9264&gt;`_)
   ```
   
  
  
   ### 20.3.2
   ```
   ===================

Features
--------

- New resolver: Resolve direct and pinned (``==`` or ``===``) requirements first
  to improve resolver performance. (`9185 &lt;https://github.com/pypa/pip/issues/9185&gt;`_)
- Add a mechanism to delay resolving certain packages, and use it for setuptools. (`9249 &lt;https://github.com/pypa/pip/issues/9249&gt;`_)

Bug Fixes
---------

- New resolver: The &quot;Requirement already satisfied&quot; log is not printed only once
  for each package during resolution. (`9117 &lt;https://github.com/pypa/pip/issues/9117&gt;`_)
- Fix crash when logic for redacting authentication information from URLs
  in ``--help`` is given a list of strings, instead of a single string. (`9191 &lt;https://github.com/pypa/pip/issues/9191&gt;`_)
- New resolver: Correctly implement PEP 592. Do not return yanked versions from
  an index, unless the version range can only be satisfied by yanked candidates. (`9203 &lt;https://github.com/pypa/pip/issues/9203&gt;`_)
- New resolver: Make constraints also apply to package variants with extras, so
  the resolver correctly avoids backtracking on them. (`9232 &lt;https://github.com/pypa/pip/issues/9232&gt;`_)
- New resolver: Discard a candidate if it fails to provide metadata from source,
  or if the provided metadata is inconsistent, instead of quitting outright. (`9246 &lt;https://github.com/pypa/pip/issues/9246&gt;`_)

Vendored Libraries
------------------

- Update vendoring to 20.8

Improved Documentation
----------------------

- Update documentation to reflect that pip still uses legacy resolver by default in Python 2 environments. (`9269 &lt;https://github.com/pypa/pip/issues/9269&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pip
  - Changelog: https://pyup.io/changelogs/pip/
  - Homepage: https://pip.pypa.io/
</details>



205: Update pytest to 6.2.1 r=duckinator a=pyup-bot


This PR updates [pytest](https://pypi.org/project/pytest) from **6.1.2** to **6.2.1**.



*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pytest
  - Homepage: https://docs.pytest.org/en/latest/
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 6, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
type: bug A confirmed bug or unintended behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants