Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to importlib-metadata #199

Merged
merged 5 commits into from
May 5, 2019
Merged

Switch to importlib-metadata #199

merged 5 commits into from
May 5, 2019

Conversation

asottile
Copy link
Member

@asottile asottile commented Apr 7, 2019

This attempts to switch the backing metadata looking from pkg_resources (slow) to importlib-metadata (fast)

I had to expose one derived property on top of the importlib_metadata Distribution class to make it look like the pkg_resources one -- but otherwise this should be drop-in.

At least for pytest this isn't enough to get the full performance boost since pytest imports pkg_resources itself -- but baby steps! I'll be following up with the pytest branch in pytest-dev/pytest#5063

@asottile
Copy link
Member Author

asottile commented Apr 7, 2019

This is WIP:

  • pytest uses dist.project_name which isn't a thing with importlib-metadata Distribution objects

need to make a shim? dist.metadata['name'] ~= dist.project_name

@asottile asottile changed the title [WIP] Switch to importlib-metadata Switch to importlib-metadata Apr 20, 2019
@asottile
Copy link
Member Author

This is no longer WIP, technically as far as I can tell this version is compatible with pytest as-is (due to the Distribution shim)

@asottile
Copy link
Member Author

switched the last commit from python -m tox to tox directly, it looks like until pluggy is the src layout tox is importing it directly 🤦‍♂️

@codecov-io
Copy link

codecov-io commented Apr 21, 2019

Codecov Report

Merging #199 into master will decrease coverage by 0.08%.
The diff coverage is 92.3%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #199      +/-   ##
==========================================
- Coverage   92.65%   92.57%   -0.09%     
==========================================
  Files          14       14              
  Lines        1553     1549       -4     
  Branches      115      116       +1     
==========================================
- Hits         1439     1434       -5     
- Misses         96       98       +2     
+ Partials       18       17       -1
Impacted Files Coverage Δ
testing/test_pluginmanager.py 95.67% <100%> (-0.12%) ⬇️
pluggy/manager.py 94.56% <84.21%> (-0.36%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 59b3cf2...0706fba. Read the comment docs.

num = pm.load_setuptools_entrypoints("hello")
assert num == 0 # no plugin loaded by this call


def test_load_setuptools_version_conflict(monkeypatch, pm):
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

runtime checking of dependency versions is not possible without the old and slow pkg_resources? ;-(

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmmm I guess that's not the way I saw that. any time I've dealt with pkg_resources the side-effect of looking up package metadata triggering VersionConflict I always saw as a nuisance. It's not really protecting you against anything, you're already in the broken state.

It would be possible to do with importlib-metadata, but traversing the dependency graph is wasted cpu cycles :)

@nicoddemus
Copy link
Member

Anything else to be done here @asottile? Also, can we check how this affects tox and devpy?

cc @gaborbernat @fschulze

@asottile
Copy link
Member Author

asottile commented May 4, 2019

I think testing the integration with the other big users is all that's left here. my pytest PR passes tests when coupled with this PR (unfortunately pytest has some tests which test the full end-to-end integration and worry about some specifics of the implementation of pluggy and so changes were necessary there)

@asottile
Copy link
Member Author

asottile commented May 4, 2019

tox's testsuite passes, there's not much that tox does directly with pkg_resources and would be easy to factor that out 👍

@fschulze
Copy link
Contributor

fschulze commented May 4, 2019

Does it affect users of the pluggy library in any way? By the looks of it, it shouldn't. And the things we use from pkg_resources in devpi should be covered by importlib-metadata as well, so maybe we will also switch. I think only in tests are we using some things that aren't available yet.

@asottile
Copy link
Member Author

asottile commented May 4, 2019

this shouldn't affect users in any way (other than potentially speeding things up if pkg_resources gets factored out of the rest of the things) -- the only changes I had to make for pytest to make the tests pass was adjusting the tests (though I did also factor out most of the rest of the pytest usages of pkg_resources to get the full speed boost)

@nicoddemus
Copy link
Member

Great!

I propose we merge this and make a 0.10 release then. We can then update pytest's PR and get this out with pytest 4.5. What do you think @asottile?

@asottile
Copy link
Member Author

asottile commented May 5, 2019

Sounds good! Let's do this Tuesday at the earliest (busy with pycon today and Monday and would want to be a available for potential fixes)

@gaborbernat
Copy link

tox will follow probably with a release on Thursday, to take advantage of the pytest release lessons 😎

@nicoddemus
Copy link
Member

Let's do this Tuesday at the earliest (busy with pycon today and Monday and would want to be a available for potential fixes)

You mean pluggy, pytest or both? My original intention was to release pluggy soon only. pytest still has a few open PRs I would like to see merged before we release 4.5.

@nicoddemus
Copy link
Member

(btw jealous of you guys at PyCon ❤️ 😁)

@asottile
Copy link
Member Author

asottile commented May 5, 2019

Let's do this Tuesday at the earliest (busy with pycon today and Monday and would want to be a available for potential fixes)

You mean pluggy, pytest or both? My original intention was to release pluggy soon only. pytest still has a few open PRs I would like to see merged before we release 4.5.

I was mostly referring to pluggy (and with the hope that pytest would follow soon after, I can help finishing up the other things we want to get into 4.5 as well :D)

@nicoddemus
Copy link
Member

OK, let's wait until Tuesday then! 👍

@nicoddemus nicoddemus merged commit 8749791 into pytest-dev:master May 5, 2019
@nicoddemus
Copy link
Member

Oops forgot about the changelog entry.

@fschulze
Copy link
Contributor

fschulze commented May 6, 2019

What about making it a 1.0.0? It's been in use for quite a while now.

@goodboy
Copy link
Contributor

goodboy commented May 6, 2019

@fschulze we're trying to get in a bunch of "breaking" changes for 1.0 which I can hopefully get to at some point in the next month. Unfortunately been really busy with non-software stuff atm..

@nicoddemus
Copy link
Member

We are tracking 1.0 here: #140

@asottile asottile deleted the importlib_metadata_v2 branch May 6, 2019 14:08
nicoddemus added a commit that referenced this pull request May 6, 2019
Add changelog for #199
bors bot referenced this pull request in mozilla/normandy May 17, 2019
1887: Scheduled weekly dependency update for week 19 r=rehandalal a=pyup-bot






### Update [botocore](https://pypi.org/project/botocore) from **1.12.142** to **1.12.146**.


<details>
  <summary>Changelog</summary>
  
  
   ### 1.12.146
   ```
   ========

* api-change:``glue``: Update glue client to latest version
* api-change:``sts``: Update sts client to latest version
   ```
   
  
  
   ### 1.12.145
   ```
   ========

* api-change:``servicecatalog``: Update servicecatalog client to latest version
* api-change:``eks``: Update eks client to latest version
* api-change:``sagemaker``: Update sagemaker client to latest version
* api-change:``kinesisanalytics``: Update kinesisanalytics client to latest version
* api-change:``kinesisanalyticsv2``: Update kinesisanalyticsv2 client to latest version
   ```
   
  
  
   ### 1.12.144
   ```
   ========

* api-change:``appsync``: Update appsync client to latest version
* api-change:``storagegateway``: Update storagegateway client to latest version
* api-change:``ssm``: Update ssm client to latest version
* api-change:``alexaforbusiness``: Update alexaforbusiness client to latest version
   ```
   
  
  
   ### 1.12.143
   ```
   ========

* api-change:``config``: Update config client to latest version
* api-change:``iam``: Update iam client to latest version
* api-change:``sts``: Update sts client to latest version
* api-change:``codepipeline``: Update codepipeline client to latest version
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/botocore
  - Changelog: https://pyup.io/changelogs/botocore/
  - Repo: https://github.com/boto/botocore
</details>





### Update [Faker](https://pypi.org/project/Faker) from **1.0.5** to **1.0.6**.


*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/faker
  - Changelog: https://pyup.io/changelogs/faker/
  - Repo: https://github.com/joke2k/faker
</details>





### Update [pluggy](https://pypi.org/project/pluggy) from **0.9.0** to **0.11.0**.


<details>
  <summary>Changelog</summary>
  
  
   ### 0.11.0
   ```
   ==========================

Bug Fixes
---------

- `205 &lt;https://github.com/pytest-dev/pluggy/issues/205&gt;`_: Revert changes made in 0.10.0 release breaking ``.egg`` installs.
   ```
   
  
  
   ### 0.10.0
   ```
   ==========================

Features
--------

- `199 &lt;https://github.com/pytest-dev/pluggy/issues/199&gt;`_: Switch from ``pkg_resources`` to ``importlib-metadata`` for entrypoint detection for improved performance and import time.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pluggy
  - Changelog: https://pyup.io/changelogs/pluggy/
  - Repo: https://github.com/pytest-dev/pluggy
</details>





### Update [Pygments](https://pypi.org/project/Pygments) from **2.3.1** to **2.4.0**.


*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pygments
  - Homepage: http://pygments.org/
</details>





### Update [pyrsistent](https://pypi.org/project/pyrsistent) from **0.15.1** to **0.15.2**.


<details>
  <summary>Changelog</summary>
  
  
   ### 0.15.2
   ```
   * Fix 166, Propagate &#39;ignore_extra&#39; param in hierarchy. Thanks ss18 for this!
 * Fix 167, thaw typing. Thanks nattofriends for this!
 * Fix 154, not possible to insert empty pmap as leaf node with transform.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pyrsistent
  - Changelog: https://pyup.io/changelogs/pyrsistent/
  - Repo: http://github.com/tobgu/pyrsistent/
</details>





### Update [boto3](https://pypi.org/project/boto3) from **1.9.142** to **1.9.146**.


<details>
  <summary>Changelog</summary>
  
  
   ### 1.9.146
   ```
   =======

* api-change:``glue``: [``botocore``] Update glue client to latest version
* api-change:``sts``: [``botocore``] Update sts client to latest version
   ```
   
  
  
   ### 1.9.145
   ```
   =======

* api-change:``servicecatalog``: [``botocore``] Update servicecatalog client to latest version
* api-change:``eks``: [``botocore``] Update eks client to latest version
* api-change:``sagemaker``: [``botocore``] Update sagemaker client to latest version
* api-change:``kinesisanalytics``: [``botocore``] Update kinesisanalytics client to latest version
* api-change:``kinesisanalyticsv2``: [``botocore``] Update kinesisanalyticsv2 client to latest version
   ```
   
  
  
   ### 1.9.144
   ```
   =======

* api-change:``appsync``: [``botocore``] Update appsync client to latest version
* api-change:``storagegateway``: [``botocore``] Update storagegateway client to latest version
* api-change:``ssm``: [``botocore``] Update ssm client to latest version
* api-change:``alexaforbusiness``: [``botocore``] Update alexaforbusiness client to latest version
   ```
   
  
  
   ### 1.9.143
   ```
   =======

* api-change:``config``: [``botocore``] Update config client to latest version
* api-change:``iam``: [``botocore``] Update iam client to latest version
* api-change:``sts``: [``botocore``] Update sts client to latest version
* api-change:``codepipeline``: [``botocore``] Update codepipeline client to latest version
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/boto3
  - Changelog: https://pyup.io/changelogs/boto3/
  - Repo: https://github.com/boto/boto3
</details>





### Update [django-cors-headers](https://pypi.org/project/django-cors-headers) from **2.5.3** to **3.0.1**.


<details>
  <summary>Changelog</summary>
  
  
   ### 3.0.0
   ```
   ------------------

* ``CORS_ORIGIN_WHITELIST`` now requires URI schemes, and optionally ports.
  This is part of the CORS specification
  (`Section 3.2 &lt;https://tools.ietf.org/html/rfc6454section-3.2&gt;`_) that was
  not implemented in this library, except from with the
  ``CORS_ORIGIN_REGEX_WHITELIST`` setting. It fixes a security issue where the
  CORS middleware would allow requests between schemes, for example from
  insecure ``http://`` Origins to a secure ``https://`` site.

  You will need to update your whitelist to include schemes, for example from
  this:

  .. code-block:: python

      CORS_ORIGIN_WHITELIST = [&#39;example.com&#39;]

  ...to this:

  .. code-block:: python

      CORS_ORIGIN_WHITELIST = [&#39;https://example.com&#39;]

* Removed the ``CORS_MODEL`` setting, and associated class. It seems very few,
  or no users were using it, since there were no bug reports since its move to
  abstract in version 2.0.0 (2017-01-07). If you *are* using this
  functionality, you can continue by changing your model to not inherit from
  the abstract one, and add a signal handler for ``check_request_enabled`` that
  reads from your model. Note you&#39;ll need to handle the move to include schemes
  for Origins.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/django-cors-headers
  - Changelog: https://pyup.io/changelogs/django-cors-headers/
  - Repo: https://github.com/ottoyiu/django-cors-headers
</details>





### Update [djangorestframework](https://pypi.org/project/djangorestframework) from **3.9.3** to **3.9.4**.


*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/djangorestframework
  - Changelog: https://pyup.io/changelogs/djangorestframework/
  - Homepage: https://www.django-rest-framework.org/
</details>





### Update [factory_boy](https://pypi.org/project/factory_boy) from **2.11.1** to **2.12.0**.


<details>
  <summary>Changelog</summary>
  
  
   ### 2.12.0
   ```
   -------------------

*New:*

    - Add support for Python 3.7
    - Add support for Django 2.1
    - Add :attr:`~factory.fuzzy.FuzzyChoice.getter` to :class:`~factory.fuzzy.FuzzyChoice` that mimics
      the behavior of ``getter`` in :class:`~factory.Iterator`
    - Make the ``extra_kwargs`` parameter of :meth:`~factory.faker.Faker.generate` optional
    - Add :class:`~factory.RelatedFactoryList` class for one-to-many support, thanks `Sean Harrington &lt;https://github.com/seanharr11&gt;`_.
    - Make the `locale` argument for :class:`~factory.faker.Faker` keyword-only

*Bugfix:*

    - Allow renamed arguments to be optional, thanks to `Justin Crown &lt;https://github.com/mrname&gt;`_.
    - Fix `django_get_or_create` behavior when using multiple fields with `unique=True`, thanks to `YPCrumble &lt;https://github.com/YPCrumble&gt;`
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/factory-boy
  - Changelog: https://pyup.io/changelogs/factory-boy/
  - Repo: https://github.com/FactoryBoy/factory_boy
</details>





### Update [kinto-http](https://pypi.org/project/kinto-http) from **10.3.0** to **10.4.0**.


<details>
  <summary>Changelog</summary>
  
  
   ### 10.4.0
   ```
   ===================

- Add support for Bearer tokens in the CLI utilities.
- Use black for code formatting.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/kinto-http
  - Changelog: https://pyup.io/changelogs/kinto-http/
  - Repo: https://github.com/Kinto/kinto-http.py/
</details>





### Update [pytest](https://pypi.org/project/pytest) from **4.4.1** to **4.5.0**.


<details>
  <summary>Changelog</summary>
  
  
   ### 4.4.2
   ```
   =========================

Bug Fixes
---------

- `5089 &lt;https://github.com/pytest-dev/pytest/issues/5089&gt;`_: Fix crash caused by error in ``__repr__`` function with both ``showlocals`` and verbose output enabled.


- `5139 &lt;https://github.com/pytest-dev/pytest/issues/5139&gt;`_: Eliminate core dependency on &#39;terminal&#39; plugin.


- `5229 &lt;https://github.com/pytest-dev/pytest/issues/5229&gt;`_: Require ``pluggy&gt;=0.11.0`` which reverts a dependency to ``importlib-metadata`` added in ``0.10.0``.
  The ``importlib-metadata`` package cannot be imported when installed as an egg and causes issues when relying on ``setup.py`` to install test dependencies.



Improved Documentation
----------------------

- `5171 &lt;https://github.com/pytest-dev/pytest/issues/5171&gt;`_: Doc: ``pytest_ignore_collect``, ``pytest_collect_directory``, ``pytest_collect_file`` and ``pytest_pycollect_makemodule`` hooks&#39;s &#39;path&#39; parameter documented type is now ``py.path.local``


- `5188 &lt;https://github.com/pytest-dev/pytest/issues/5188&gt;`_: Improve help for ``--runxfail`` flag.



Trivial/Internal Changes
------------------------

- `5182 &lt;https://github.com/pytest-dev/pytest/issues/5182&gt;`_: Removed internal and unused ``_pytest.deprecated.MARK_INFO_ATTRIBUTE``.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pytest
  - Changelog: https://pyup.io/changelogs/pytest/
  - Homepage: https://docs.pytest.org/en/latest/
</details>







Co-authored-by: pyup-bot <github-bot@pyup.io>
Co-authored-by: Michael Cooper <mythmon@gmail.com>
@asottile
Copy link
Member Author

I've re-applied this in #215 🎉

@gaborbernat
Copy link

cool times

cool times

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants