pypy2: EntryPoint object repr results in infinite recursion

[jaraco]

In GitLab by @asottile on Nov 29, 2019, 18:08

>>> from importlib_metadata import EntryPoint
>>> EntryPoint('foo', 'bar', None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>

...

  File "<string>", line 22, in __repr__
  File "<string>", line 22, in __repr__
  File "<string>", line 22, in __repr__
  File "<string>", line 22, in __repr__
  File "<string>", line 22, in __repr__
  File "/home/asottile/workspace/flake8/.tox/pypy/site-packages/importlib_metadata/__init__.py", line 125, in __iter__
    return iter((self.name, self))
RuntimeError: maximum recursion depth exceeded

[jaraco]

In GitLab by @asottile on Nov 29, 2019, 18:17

mentioned in commit asottile/flake8@6b764d0

[jaraco]

In GitLab by @asottile on Nov 29, 2019, 18:18

mentioned in merge request pycqa/flake8!389

[jaraco]

In GitLab by @asottile on Nov 30, 2019, 02:33

I'd be interested in working on this, the root cause is the same as #96 -- generally when extending NamedTuple overriding __iter__ is a bad idea (since things expecting tuples will expect it to work in a certain way). In this case, one of the other tuple member functions makes assumptions about __iter__ (which are violated by the specification in importlib.metadatas EntryPoint

I'm wondering if at this point it's too late to back out __iter__ from EntryPoint (it's provisional in python3.8 right?)

[jaraco]

In GitLab by @jaraco on Nov 30, 2019, 10:24

mentioned in commit 10234d5

[jaraco]

In GitLab by @jaraco on Nov 30, 2019, 10:25

In the aforementioned commit, I added a test. The test only fails on PyPy 2, but passes on PyPy 3.

[jaraco]

In GitLab by @jaraco on Nov 30, 2019, 10:26

changed title from pypy: EntryPoint object repr results in infinite recursion to pypy{+2+}: EntryPoint object repr results in infinite recursion

[jaraco]

In GitLab by @jaraco on Nov 30, 2019, 10:31

In the bugfix/pickleable-ep branch, I have a fix for #96 (I'm just waiting on a fix for #100 before submitting). Given the limited impact (doesn't affect late PyPy 3) of this issue, I'm less inclined to re-design EntryPoint.__iter__ behavior, though I believe that's a possibility. It would be a 2.x release of the backport and might have to wait for Python 3.9 to change for stdlib Python.

[jaraco]

In GitLab by @jaraco on Nov 30, 2019, 10:38

These are the tested assertions that fail when removing __iter__:

importlib_metadata master $ git diff                                                                                                                                      
diff --git a/importlib_metadata/__init__.py b/importlib_metadata/__init__.py
index f0e7eba..ae78b9f 100644
--- a/importlib_metadata/__init__.py
+++ b/importlib_metadata/__init__.py
@@ -118,12 +118,6 @@ class EntryPoint(collections.namedtuple('EntryPointBase', 'name value group')):
             config.readfp(io.StringIO(text))
         return EntryPoint._from_config(config)
 
-    def __iter__(self):
-        """
-        Supply iter so one may construct dicts of EntryPoints easily.
-        """
-        return iter((self.name, self))
-
 
 class PackagePath(pathlib.PurePosixPath):
     """A reference to a path in a package"""
importlib_metadata master $ tox -e py37                                                                                                                                   
py37 develop-inst-noop: /Users/jaraco/code/public/importlib_metadata
py37 installed: -e git+gl://python-devs/importlib_metadata@c471831d2bdda601b32b38295fc893e908cc972a#egg=importlib_metadata,more-itertools==8.0.0,packaging==19.2,pyparsing==2.4.5,six==1.13.0,zipp==0.6.0
py37 run-test-pre: PYTHONHASHSEED='1602908799'
py37 run-test: commands[0] | python -m unittest discover
E..........................E.E........E.....E..
======================================================================
ERROR: test_entry_points (importlib_metadata.tests.test_api.APITests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jaraco/code/public/importlib_metadata/importlib_metadata/tests/test_api.py", line 57, in test_entry_points
    entries = dict(entry_points()['entries'])
ValueError: dictionary update sequence element #0 has length 3; 2 is required

======================================================================
ERROR: test_entrypoint_with_colon_in_name (importlib_metadata.tests.test_main.ImportTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jaraco/code/public/importlib_metadata/importlib_metadata/tests/test_main.py", line 53, in test_entrypoint_with_colon_in_name
    entries = dict(entry_points()['entries'])
ValueError: dictionary update sequence element #0 has length 3; 2 is required

======================================================================
ERROR: test_resolve (importlib_metadata.tests.test_main.ImportTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jaraco/code/public/importlib_metadata/importlib_metadata/tests/test_main.py", line 48, in test_resolve
    entries = dict(entry_points()['entries'])
ValueError: dictionary update sequence element #0 has length 3; 2 is required

======================================================================
ERROR: test_zip_entry_points (importlib_metadata.tests.test_zip.TestEgg)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jaraco/code/public/importlib_metadata/importlib_metadata/tests/test_zip.py", line 38, in test_zip_entry_points
    scripts = dict(entry_points()['console_scripts'])
ValueError: dictionary update sequence element #0 has length 3; 2 is required

======================================================================
ERROR: test_zip_entry_points (importlib_metadata.tests.test_zip.TestZip)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jaraco/code/public/importlib_metadata/importlib_metadata/tests/test_zip.py", line 38, in test_zip_entry_points
    scripts = dict(entry_points()['console_scripts'])
ValueError: dictionary update sequence element #0 has length 3; 2 is required

----------------------------------------------------------------------
Ran 47 tests in 0.203s

FAILED (errors=5)
ERROR: InvocationError for command /Users/jaraco/code/public/importlib_metadata/.tox/py37/bin/python -m unittest discover (exited with code 1)
________________________________________________________________________________ summary _________________________________________________________________________________
ERROR:   py37: commands failed

It feels to me like the value of constructing dicts of EntryPoints is more valuable than the edge cases that it trips up, especially if those edge cases can be addressed directly and cleanly.

[jaraco]

In GitLab by @jaraco on Nov 30, 2019, 10:53

mentioned in commit 7310c08

[jaraco]

In GitLab by @jaraco on Nov 30, 2019, 10:54

In the bugfix/97-entry-point-repr, I've pushed a patch that surgically addresses the issue. I'll plan to submit that for review after #100 is fixed, but I'm open to your thoughts anytime.

[jaraco]

In GitLab by @asottile on Nov 30, 2019, 15:29

Here's an example which is impossible to fix:

>>> entrypoint = EntryPoint('a', 'b', 'c')
>>> json.dumps(ep)
...
RecursionError

[jaraco]

In GitLab by @asottile on Nov 30, 2019, 15:31

(typed that on phone, here's the actual error):

>>> json.dumps(importlib_metadata.EntryPoint('a', 'b', 'c'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
ValueError: Circular reference detected

[jaraco]

In GitLab by @jaraco on Nov 30, 2019, 17:02

mentioned in commit 741ed9eeca75f288ee5a0bbbb341963aff4c6d9a

[jaraco]

In GitLab by @jaraco on Nov 30, 2019, 17:02

mentioned in commit 3b999e408591f26ee14fe9bacccd511a2784c46c

[jaraco]

In GitLab by @jaraco on Nov 30, 2019, 17:02

mentioned in commit b742833623ad7b9d42e65086d5731a6a39625ba5