Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minor edits to the Descriptor HowTo Guide #24901

Merged
merged 25 commits into from
Oct 9, 2022
Merged
Changes from 15 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
58b799d
Fix wrong Sphinx markup preventing syntax highlighting
geryogam Mar 17, 2021
f77b0f9
Fix a bug in the ClassMethod Python equivalent
geryogam Mar 17, 2021
1dbd876
Fix the description of the ClassMethod Python equivalent
geryogam Mar 17, 2021
d0d301e
Raise TypeError for __get__(None, None) calls
geryogam Mar 19, 2021
e87fe37
Fix a typo
geryogam Mar 22, 2021
52a06d0
Use the existing cls variable
geryogam Mar 25, 2021
2f9971b
Allow keyword arguments in Object.__new__
geryogam Mar 31, 2021
2f335ca
Use two more super() for consistency
geryogam Mar 31, 2021
4886dbb
Add Object.__getattribute__ to raise AttributeError with the correct …
geryogam Mar 31, 2021
705c577
Allow attribute lookup from a class in Member.__get__
geryogam Mar 31, 2021
c0e6432
Allow keyword arguments in Type.__new__
geryogam Mar 31, 2021
81ec93c
Raise ValueError for slot names conflicting with class variables
geryogam Apr 1, 2021
34eef3a
Update descriptor.rst
geryogam Apr 1, 2021
ae8c622
Raise TypeError for non-empty __slots__ in subclasses of variable-len…
geryogam Apr 5, 2021
1d5fb96
Raise TypeError for non-empty __slots__ in subclasses of variable-len…
geryogam Apr 5, 2021
4671b91
Apply requested changes and remove changes on the slots emulation
geryogam May 10, 2022
0a132bf
Merge branch 'main' into patch-14
geryogam May 10, 2022
62fe4fc
Update descriptor.rst
geryogam May 10, 2022
1252f4b
Update descriptor.rst
geryogam May 10, 2022
58ae977
Update descriptor.rst
geryogam May 10, 2022
5630c64
Update descriptor.rst
geryogam May 10, 2022
bfb33f7
Merge branch 'main' into patch-14
geryogam May 10, 2022
5d45eaa
Revert the two-argument form of super() and forward variadic arguments
geryogam Oct 4, 2022
c853e6b
Merge branch 'main' into patch-14
rhettinger Oct 7, 2022
bcc9da0
Update descriptor.rst
geryogam Oct 8, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 73 additions & 26 deletions Doc/howto/descriptor.rst
Original file line number Diff line number Diff line change
Expand Up @@ -795,7 +795,7 @@ afterwards, :meth:`__set_name__` will need to be called manually.
ORM example
-----------

The following code is simplified skeleton showing how data descriptors could
The following code is a simplified skeleton showing how data descriptors could
be used to implement an `object relational mapping
<https://en.wikipedia.org/wiki/Object%E2%80%93relational_mapping>`_.

Expand Down Expand Up @@ -940,6 +940,8 @@ here is a pure Python equivalent:
self._name = name

def __get__(self, obj, objtype=None):
if obj is None and objtype is None:
raise TypeError('__get__(None, None) is invalid')
if obj is None:
return self
if self.fget is None:
Expand Down Expand Up @@ -1087,6 +1089,8 @@ during dotted lookup from an instance. Here's how it works:

def __get__(self, obj, objtype=None):
"Simulate func_descr_get() in Objects/funcobject.c"
if obj is None and objtype is None:
raise TypeError('__get__(None, None) is invalid')
if obj is None:
return self
return MethodType(self, obj)
Expand Down Expand Up @@ -1205,7 +1209,7 @@ example calls are unexciting:
Using the non-data descriptor protocol, a pure Python version of
:func:`staticmethod` would look like this:

.. doctest::
.. testcode::

class StaticMethod:
"Emulate PyStaticMethod_Type() in Objects/funcobject.c"
Expand All @@ -1214,6 +1218,8 @@ Using the non-data descriptor protocol, a pure Python version of
self.f = f

def __get__(self, obj, objtype=None):
if obj is None and objtype is None:
raise TypeError('__get__(None, None) is invalid')
return self.f


Expand Down Expand Up @@ -1277,9 +1283,11 @@ Using the non-data descriptor protocol, a pure Python version of
self.f = f

def __get__(self, obj, cls=None):
if obj is None and cls is None:
raise TypeError('__get__(None, None) is invalid')
if cls is None:
cls = type(obj)
if hasattr(obj, '__get__'):
if hasattr(self.f, '__get__'):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out-of-date. This currently reads:


    class ClassMethod:
        "Emulate PyClassMethod_Type() in Objects/funcobject.c"

        def __init__(self, f):
            self.f = f

        def __get__(self, obj, cls=None):
            if cls is None:
                cls = type(obj)
            if hasattr(type(self.f), '__get__'):
                return self.f.__get__(cls, cls)
            return MethodType(self.f, cls)

Copy link
Contributor Author

@geryogam geryogam May 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright. I wish we had a convenient way in Python to bypass instance attributes (with type(self.f), the attribute '__get__' is also looked up in the metaclass). I suggested this on Stack Overflow: https://stackoverflow.com/q/67099920/2326961

return self.f.__get__(cls)
return MethodType(self.f, cls)

Expand All @@ -1303,7 +1311,7 @@ Using the non-data descriptor protocol, a pure Python version of
>>> t.cm(11, 22)
(<class 'T'>, 11, 22)

The code path for ``hasattr(obj, '__get__')`` was added in Python 3.9 and
The code path for ``hasattr(self.f, '__get__')`` was added in Python 3.9 and
makes it possible for :func:`classmethod` to support chained decorators.
For example, a classmethod and property could be chained together:

Expand Down Expand Up @@ -1413,7 +1421,7 @@ It is not possible to create an exact drop-in pure Python version of
``__slots__`` because it requires direct access to C structures and control
over object memory allocation. However, we can build a mostly faithful
simulation where the actual C structure for slots is emulated by a private
``_slotvalues`` list. Reads and writes to that private structure are managed
``_slot_values`` list. Reads and writes to that private structure are managed
by member descriptors:

.. testcode::
Expand All @@ -1432,25 +1440,29 @@ by member descriptors:
def __get__(self, obj, objtype=None):
'Emulate member_get() in Objects/descrobject.c'
# Also see PyMember_GetOne() in Python/structmember.c
value = obj._slotvalues[self.offset]
if obj is None and objtype is None:
raise TypeError('__get__(None, None) is invalid')
if obj is None:
return self
value = obj._slot_values[self.offset]
if value is null:
raise AttributeError(self.name)
return value

def __set__(self, obj, value):
'Emulate member_set() in Objects/descrobject.c'
obj._slotvalues[self.offset] = value
obj._slot_values[self.offset] = value

def __delete__(self, obj):
'Emulate member_delete() in Objects/descrobject.c'
value = obj._slotvalues[self.offset]
value = obj._slot_values[self.offset]
if value is null:
raise AttributeError(self.name)
obj._slotvalues[self.offset] = null
obj._slot_values[self.offset] = null

def __repr__(self):
'Emulate member_repr() in Objects/descrobject.c'
return f'<Member {self.name!r} of {self.clsname!r}>'
return f'<Member {self.name!r} of {self.clsname!r} objects>'

The :meth:`type.__new__` method takes care of adding member objects to class
variables:
Expand All @@ -1460,13 +1472,25 @@ variables:
class Type(type):
'Simulate how the type metaclass adds member objects for slots'

def __new__(mcls, clsname, bases, mapping):
def __new__(mcls, clsname, bases, mapping, **kwargs):
'Emuluate type_new() in Objects/typeobject.c'
# type_new() calls PyTypeReady() which calls add_methods()
slot_names = mapping.get('slot_names', [])
for offset, name in enumerate(slot_names):
if slot_names:
for base in bases:
for cls in base.__mro__:
if cls.__itemsize__:
raise TypeError(
f'nonempty __slots__ not supported for '
f'subtype of {cls.__name__!r}'
)
for offset, name in enumerate(dict.fromkeys(slot_names)):
if name in mapping:
raise ValueError(
f'{name!r} in __slots__ conflicts with class variable'
)
mapping[name] = Member(name, clsname, offset)
return type.__new__(mcls, clsname, bases, mapping)
return super().__new__(mcls, clsname, bases, mapping)

The :meth:`object.__new__` method takes care of creating instances that have
slots instead of an instance dictionary. Here is a rough simulation in pure
Expand All @@ -1477,20 +1501,30 @@ Python:
class Object:
'Simulate how object.__new__() allocates memory for __slots__'

def __new__(cls, *args):
def __new__(cls, *args, **kwargs):
'Emulate object_new() in Objects/typeobject.c'
inst = super().__new__(cls)
if hasattr(cls, 'slot_names'):
empty_slots = [null] * len(cls.slot_names)
object.__setattr__(inst, '_slotvalues', empty_slots)
super().__setattr__(inst, '_slot_values', empty_slots)
return inst

def __getattribute__(self, name):
'Emulate _PyObject_GenericGetAttrWithDict() Objects/object.c'
cls = type(self)
if (hasattr(cls, 'slot_names') and name not in cls.slot_names
and name != '_slot_values'):
raise AttributeError(
f'{cls.__name__!r} object has no attribute {name!r}'
)
return super().__getattribute__(name)

def __setattr__(self, name, value):
'Emulate _PyObject_GenericSetAttrWithDict() Objects/object.c'
cls = type(self)
if hasattr(cls, 'slot_names') and name not in cls.slot_names:
raise AttributeError(
f'{type(self).__name__!r} object has no attribute {name!r}'
f'{cls.__name__!r} object has no attribute {name!r}'
)
super().__setattr__(name, value)

Expand All @@ -1499,7 +1533,7 @@ Python:
cls = type(self)
if hasattr(cls, 'slot_names') and name not in cls.slot_names:
raise AttributeError(
f'{type(self).__name__!r} object has no attribute {name!r}'
f'{cls.__name__!r} object has no attribute {name!r}'
)
super().__delattr__(name)

Expand Down Expand Up @@ -1533,31 +1567,44 @@ At this point, the metaclass has loaded member objects for *x* and *y*::

# We test this separately because the preceding section is not
# doctestable due to the hex memory address for the __init__ function
>>> isinstance(vars(H)['x'], Member)
>>> isinstance(H.x, Member)
True
>>> isinstance(vars(H)['y'], Member)
>>> isinstance(H.y, Member)
True

When instances are created, they have a ``slot_values`` list where the
When instances are created, they have a ``_slot_values`` list where the
attributes are stored:

.. doctest::

>>> h = H(10, 20)
>>> vars(h)
{'_slotvalues': [10, 20]}
>>> h._slot_values
[10, 20]
>>> h.x = 55
>>> vars(h)
{'_slotvalues': [55, 20]}
>>> h._slot_values
[55, 20]

Misspelled or unassigned attributes will raise an exception:

.. doctest::

>>> h.xz
>>> vars(h)
Traceback (most recent call last):
...
TypeError: vars() argument must have __dict__ attribute
>>> h.__dict__
Traceback (most recent call last):
...
AttributeError: 'H' object has no attribute '__dict__'
>>> h.z
Traceback (most recent call last):
...
AttributeError: 'H' object has no attribute 'z'
>>> del h.y
>>> h.y
Traceback (most recent call last):
...
AttributeError: 'H' object has no attribute 'xz'
AttributeError: 'y'

.. doctest::
:hide:
Expand Down