bpo-39481: Implementation for PEP 585 #18239

gvanrossum · 2020-01-28T15:41:26Z

I consider this a finished prototype now. We've got everything working, and then some (e.g. pickle), and while we found a fair number of minor issues with PEP 585, the main conclusion is that the PEP is quite reasonable to implement.

A number of suggestions for changes PEP 585 that have basically been vetted by @ambv can be found in python/peps#1289.

There are a few places where I still disagree with @ambv. In particular, I don't think it makes sense to even allow isinstance(x, list[int]) and issubclass(C, list[int]) -- IMO these should just raise TypeError, and that's what this PR implements. (If @ambv convinces me to change my mind, it's easy to revert that.) [UPDATE: @ambv agrees and has updated the PEP.]

There are also a few things that ought to be changed:

We should add a nice docstring to GenericAlias.
We made io.IOBase generic. This was a misunderstanding, and we'll revert that.
The __parameters__ attribute can and probably should be computed lazily the first time it's needed.
Maybe we still need a cache, like typing.py uses?
Where should the code live? For now I put it in descrobject.c, but that's debatable (it was just convenient).

https://bugs.python.org/issue39481

Objects/descrobject.c

serhiy-storchaka · 2020-01-28T16:04:00Z

Objects/descrobject.c

+};
+
+static PyMemberDef ga_members[] = {
+    {"__origin__", T_OBJECT, offsetof(gaobject, origin), READONLY},


Maybe T_OBJECT_EX?

I think not. The two attributes should never be allowed to be NULL. (This is a change from an earlier design.)

Objects/descrobject.c

serhiy-storchaka · 2020-01-28T16:14:43Z

Objects/descrobject.c

+static PyObject *
+ga_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
+{
+    if (kwds != NULL) {


It is worth to use Argument Clinic which generates simpler code using non-public C API similar to the following:

if (!_PyArg_NoKeywords("GenericAlias", kwds) || !_PyArg_CheckPositional("GenericAlias", PyTuple_GET_SIZE(args), 2, 2)) { return NULL; }

Objects/descrobject.c

serhiy-storchaka · 2020-01-28T16:27:39Z

Objects/descrobject.c

+};
+
+static PyObject *
+ga_new(PyTypeObject *type, PyObject *args, PyObject *kwds)


Do we need a __new__ method? Why not set tp_new = NULL?

I first had a tp_init. Then calling GenericAlias() somehow worked and produced an object with NULL attributes (or maybe it was some more clever incantation, I forget, but I definitely saw it). I do want the class instantiable from Python (so it can potentially be used from typing.py). Using tp_new instead solved that problem.

gvanrossum · 2020-01-29T04:22:57Z

So I'm looking into the test failures. I've fixed two simple ones, but now I'm stuck with at least two:

test_typing.py
test_genericalias.py (i.e. the very PEP 585 test) crashes on Ubuntu, but not for me locally , see below (UPDATE: fixed)

The problem with test_typing.py is that when we have a class that derives from Tuple[xxx], like so:

from typing import Tuple
class C(Tuple[int]): pass
print(C.mro())  # C, tuple, typing.Generic, object

its MRO contains tuple before Generic, and the new tuple.__class_getitem__ shadows the intended Generic.__class_getitem__. I'm not sure what to do about this. @ambv @ilevkivskyi @serhiy-storchaka

gvanrossum · 2020-01-29T04:25:48Z

Using a debug build I can repro the test_genericalias.py crash. The upper few lines of the stack trace are:

* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0x00007fff72e217fa libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff72edebc1 libsystem_pthread.dylib`pthread_kill + 432
    frame #2: 0x00007fff72da8a1c libsystem_c.dylib`abort + 120
    frame #3: 0x00007fff72da7cd6 libsystem_c.dylib`__assert_rtn + 314
    frame #4: 0x00000001000df936 python.exe`_PyType_Lookup(type=0x0000000100401d30, name=0x00000001035f1d60) at typeobject.c:3155:5
    frame #5: 0x00000001000c0b8f python.exe`_PyObject_GenericGetAttrWithDict(obj=0x0000000100401ec8, name=0x00000001035f1d60, dict=0x0000000000000000, suppress=0) at object.c:1319:13
    frame #6: 0x00000001000c0ab3 python.exe`PyObject_GenericGetAttr(obj=0x0000000100401ec8, name=0x00000001035f1d60) at object.c:1407:12
    frame #7: 0x00000001000c0268 python.exe`PyObject_GetAttr(v=0x0000000100401ec8, name=0x00000001035f1d60) at object.c:1013:16
    frame #8: 0x00000001000c01a4 python.exe`PyObject_GetAttrString(v=0x0000000100401ec8, name="__module__") at object.c:918:11
    frame #9: 0x0000000100056154 python.exe`ga_repr_item(writer=0x00007ffeefbe9078, p=0x0000000100401ec8) at descrobject.c:1828:24
    frame #10: 0x0000000100054015 python.exe`ga_repr(self=0x00000001040349b0) at descrobject.c:1890:13
    frame #11: 0x00000001000beefb python.exe`PyObject_Repr(v=0x00000001040349b0) at object.c:543:11
    frame #12: 0x00000001001e9d49 python.exe`builtin_repr(module=0x0000000101221dd0, obj=0x00000001040349b0) at bltinmodule.c:2141:12

@ethanhs does this give you a clue? Methinks there's a refcount bug in ga_repr(). :-(

gvanrossum · 2020-01-29T04:30:29Z

@ethanhs Update: it's crashing in repr(tuple[int, ...]). I think I know the problem. Fix coming up.

gvanrossum · 2020-01-29T05:38:16Z

Okay, so test_typing.py is the only failing test (see above). But I think the gc_repr() code might be refactored a bit to avoid getting the __origin__ and __parameters__ attributes before we know they are needed.

emmatyping · 2020-01-29T08:52:41Z

But I think the gc_repr() code might be refactored a bit to avoid getting the origin and parameters attributes before we know they are needed.

Ah, are you suggesting that we move those into the branch where we check fro __qualname__ and __module__?

(also thank you for fixing the crash!)

serhiy-storchaka · 2020-01-29T18:05:28Z

Lib/test/test_genericalias.py


 class BaseTest(unittest.TestCase):
    """Test basics."""

    def test_subscriptable(self):
-        for t in tuple, list, dict, set, frozenset, defaultdict, deque:
+        for t in tuple, list, dict, set, frozenset, defaultdict, deque, IOBase, Pattern, Match:


TODO: I suppose IOBase subclasses BufferedIOBase, RawIOBase and TextIOBase should not support indexing.

There is also Python implementation _pyio.IOBase. I am not sure what to do with it. Maybe just ignore for now.

I added __class_getitem__ to _pyio.IOBase in 35cd5f477c9da985880130d72297335e010450e4.

I'm not sure whether it matters much that we take the __class_getitem__ away from the various subclasses (let the static type checkers worry about that).

…list, dict, set, frozenset

... collections.defaultdict

By Ethan Smith.

Co-Authored-By: Serhiy Storchaka <storchaka@gmail.com>

gvanrossum · 2020-04-06T19:16:15Z

What's up with the Docs build failures in CI?

gvanrossum · 2020-04-06T20:45:04Z

Can't say I understand the Windows failures either. Anyway, I'm prepping this for merge by deleting the GitHub workflow edit. I'll be back later.

gvanrossum · 2020-04-06T22:17:45Z

Hey @ethanhs -- can you repro the Windows failure here on test_ctypes?

emmatyping · 2020-04-06T22:42:27Z

@gvanrossum I cannot repro locally. It seems to be unable to find the sqlite shared library, but I have no idea how that could be caused by this changeset.

gvanrossum · 2020-04-06T22:44:11Z

Thanks -- I'll just land.

gvanrossum · 2020-04-06T22:51:21Z

Oops. "Required statuses must pass before merging"

serhiy-storchaka · 2020-04-07T10:29:27Z

Should not we add Py_GenericAlias and/or Py_GenericAliasType in PC/python3.def?

gvanrossum · 2020-04-07T15:49:10Z

Should not we add Py_GenericAlias and/or Py_GenericAliasType in PC/python3.def?

Done, both. And the test failure is resolved after the merge from master, so once this passes I'll land.

bedevere-bot · 2020-04-07T16:50:09Z

@gvanrossum: Please replace # with GH- in the commit message next time. Thanks!

This implements things like `list[int]`, which returns an object of type `types.GenericAlias`. This object mostly acts as a proxy for `list`, but has attributes `__origin__` and `__args__` that allow recovering the parts (with values `list` and `(int,)`. There is also an approximate notion of type variables; e.g. `list[T]` has a `__parameters__` attribute equal to `(T,)`. Type variables are objects of type `typing.TypeVar`.

gvanrossum requested a review from serhiy-storchaka January 28, 2020 15:41

gvanrossum requested review from methane and rhettinger as code owners January 28, 2020 15:41

the-knights-who-say-ni added the CLA signed label Jan 28, 2020

bedevere-bot added the awaiting core review label Jan 28, 2020

gvanrossum removed the request for review from rhettinger January 28, 2020 15:42

serhiy-storchaka reviewed Jan 28, 2020

View reviewed changes

gvanrossum changed the title ~~WIP: PEP 585~~ bpo-39481: WIP: PEP 585 Jan 28, 2020

gvanrossum added the DO-NOT-MERGE label Jan 28, 2020

serhiy-storchaka reviewed Jan 29, 2020

View reviewed changes

gvanrossum requested a review from 1st1 as a code owner January 30, 2020 00:05

gvanrossum and others added 14 commits January 29, 2020 16:29

PEP 585 step 2: Write a Py_GenericAlias() function that returns origin

d557a64

PEP 585 step 3: Add Py_GenericAlias() as __class_getitem__ to tuple, …

da33267

…list, dict, set, frozenset

PEP 585 step 4: Write some tests

447fbe9

Define a GenericAlias (proxy) object (with lots of TODOs still)

83e2ba2

More tests

fb3e8d3

PEP 585 steps 5-6: Finish basic GenericAlias object and test

235180b

Fix bug with class methods

70f3abc

Define tp_new instead of tp_init

e4d1356

Test that list[int][int] fails

8f7a55e

Test that class L(list): pass; L[int] works

f2ba2e5

Implement PEP 585 for collections.deque and ... (#2)

0af74d1

... collections.defaultdict

Respond to Serhiy's code review

b264294

📜🤖 Added by blurb_it.

0ddea56

PEP 585 steps 7-8: Implement repr for GenericAlias (#1)

91edf1c

By Ethan Smith.

gvanrossum and others added 2 commits April 6, 2020 12:01

Use PyObject_Str(qualname)

95b3c2a

Co-Authored-By: Serhiy Storchaka <storchaka@gmail.com>

Merge remote-tracking branch 'upstream/master' into pep585

13d3742

Delete mention of pep585 branch in GitHub workflow

d3839ea

gvanrossum removed the awaiting review label Apr 6, 2020

gvanrossum closed this Apr 6, 2020

gvanrossum reopened this Apr 6, 2020

Move Py_GenericAlias* to genericaliasobject.h

31fd842

gvanrossum added 2 commits April 7, 2020 08:22

Merge remote-tracking branch 'upstream/master' into pep585

6bad315

Add Py_GenericAlias[Type] to PC/python3.ref

cdb46c3

gvanrossum merged commit 48b069a into python:master Apr 7, 2020

bedevere-bot removed the awaiting merge label Apr 7, 2020

gvanrossum deleted the pep585 branch April 7, 2020 16:50

BvB93 mentioned this pull request May 7, 2021

API: Add npt.NDArray, a runtime-subscriptable alias for np.ndarray numpy/numpy#18935

Merged

hoophalab mentioned this pull request Feb 28, 2025

The return type of _get_mutex_context_from_loglevel_timeout is annotated as Any to maintain support for Python 3.7 and 3.8. y-scope/clp-loglib-py#61

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-39481: Implementation for PEP 585 #18239

bpo-39481: Implementation for PEP 585 #18239

gvanrossum commented Jan 28, 2020 •

edited

Loading

serhiy-storchaka Jan 28, 2020

gvanrossum Jan 28, 2020

serhiy-storchaka Jan 28, 2020

serhiy-storchaka Jan 28, 2020

gvanrossum Jan 28, 2020

gvanrossum commented Jan 29, 2020 •

edited

Loading

gvanrossum commented Jan 29, 2020

gvanrossum commented Jan 29, 2020

gvanrossum commented Jan 29, 2020

emmatyping commented Jan 29, 2020 •

edited

Loading

serhiy-storchaka Jan 29, 2020

serhiy-storchaka Jan 29, 2020

gvanrossum Jan 29, 2020 •

edited

Loading

gvanrossum commented Apr 6, 2020

gvanrossum commented Apr 6, 2020

gvanrossum commented Apr 6, 2020

emmatyping commented Apr 6, 2020

gvanrossum commented Apr 6, 2020

gvanrossum commented Apr 6, 2020

serhiy-storchaka commented Apr 7, 2020

gvanrossum commented Apr 7, 2020

bedevere-bot commented Apr 7, 2020

bpo-39481: Implementation for PEP 585 #18239

bpo-39481: Implementation for PEP 585 #18239

Conversation

gvanrossum commented Jan 28, 2020 • edited Loading

serhiy-storchaka Jan 28, 2020

Choose a reason for hiding this comment

gvanrossum Jan 28, 2020

Choose a reason for hiding this comment

serhiy-storchaka Jan 28, 2020

Choose a reason for hiding this comment

serhiy-storchaka Jan 28, 2020

Choose a reason for hiding this comment

gvanrossum Jan 28, 2020

Choose a reason for hiding this comment

gvanrossum commented Jan 29, 2020 • edited Loading

gvanrossum commented Jan 29, 2020

gvanrossum commented Jan 29, 2020

gvanrossum commented Jan 29, 2020

emmatyping commented Jan 29, 2020 • edited Loading

serhiy-storchaka Jan 29, 2020

Choose a reason for hiding this comment

serhiy-storchaka Jan 29, 2020

Choose a reason for hiding this comment

gvanrossum Jan 29, 2020 • edited Loading

Choose a reason for hiding this comment

gvanrossum commented Apr 6, 2020

gvanrossum commented Apr 6, 2020

gvanrossum commented Apr 6, 2020

emmatyping commented Apr 6, 2020

gvanrossum commented Apr 6, 2020

gvanrossum commented Apr 6, 2020

serhiy-storchaka commented Apr 7, 2020

gvanrossum commented Apr 7, 2020

bedevere-bot commented Apr 7, 2020

gvanrossum commented Jan 28, 2020 •

edited

Loading

gvanrossum commented Jan 29, 2020 •

edited

Loading

emmatyping commented Jan 29, 2020 •

edited

Loading

gvanrossum Jan 29, 2020 •

edited

Loading