Document descriptors

[JukkaL]

We probably want to first make sure that mypy supports some of the most common real-world descriptor use cases.

[pawelswiecki]

Is the documentation available somewhere? I cannot find it.

[ilevkivskyi]

@pawelswiecki No, the issue is still open. The general Python docs are here, from this you can guess how it translates to static typing.

[pawelswiecki]

Thanks for the answer.

I have the following problem, to which I did not find an answer to in the Descriptor HowTo Guide you've linked to. So, running mypy 0.700 on the following code...

# desc.py

class StringField:
    def __get__(self, obj, objtype) -> str:
        return "string"

class MyClass:
    x: str = StringField()

reveal_type(MyClass.x)

m = MyClass()
reveal_type(m.x)

... gives me the following output:

desc.py:8: error: Incompatible types in assignment (expression has type "StringField", variable has type "str")
desc.py:10: error: Revealed type is 'builtins.str'
desc.py:13: error: Revealed type is 'builtins.str'

So mypy correctly reveals the type of the x field (at least in case of dealing with an instance of MyClass), but annotation still gives the "Incompatible types in assignment" error.

When I remove the annotation relveal_type mypy gives the same outputs and the error disappears.

That is why I thought I need the documentation, because I must be doing something wrong. Or the feature is not yet implemented? Or is it a bug? Let me know if it makes sense to create a separate issue.

[ilevkivskyi]

@pawelswiecki mypy is perfectly right here and your annotation is wrong. Descriptor protocol is invoked only when accessing an attribute, not inside the class body. To understand it, try this:

class MyClass:
    x = StringField()
    print(x)  # <__main__.StringField object at ...>

print(MyClass.x)  # "string"

[pawelswiecki]

Right, so how do I tell mypy that x as an accessed attribute is str? Just like in dataclasses.

[pawelswiecki]

I'd like to avoid using cast.

Like something like this:

from typing import TypeVar, cast

T = TypeVar("T")

class _StringField:
    def __get__(self, obj, objtype) -> str:
        return "string"

def StringField() -> T:
    return cast(T, _StringField())

class MyClass:
    x: str = StringField()  # no error
    y: int = StringField()  # no error

reveal_type(MyClass.x)  # 'builtins.str'

m = MyClass()
reveal_type(m.x)  # 'builtins.str'

This is a partial solucion, since return type of __get__ is ignored (no surprise here).

[ilevkivskyi]

Just an hour ago you wrote

When I remove the annotation reveal_type mypy gives the same outputs and the error disappears.

class StringField:
    def __get__(self, obj, objtype) -> str:
        return "string"

class MyClass:
    x = StringField()

y = MyClass()
y.x = "test"  # OK
y.x = 42  # Error: Incompatible types in assignment (expression has type "int", variable has type "str")

What else do you need?

Also please stop spamming in this issue. If you have questions about mypy, there is a Gitter channel for this.

[pawelswiecki]

What you wrote looks like the solution. I just wanted to use the same typing pattern as dataclasses (it seemed natural at the time), which is just not adequate in the descriptor context, as you pointed out. Thank you very much for your responses. And sorry for the overabundance of comments, I'm stopping here.

[JukkaL]

Increased priority to high since this is a pretty fundamental Python feature and annotating descriptors is non-obvious.

[ilevkivskyi]

We should also add an example with an overloaded __get__ (for access on class vs instance), because it is a very common pattern.

[kawing-chiu]

I'm trying to define a simple descriptor base class, but after 3 hours of trying, still can't make all the annotations work.....It's just so hard for a normal user. This is the best I came up with:

from __future__ import annotations                                                                                       
from typing import TypeVar, Generic, Type, Any  
from weakref import WeakKeyDictionary  
  
S = TypeVar('S', bound='Descriptorable')  
  
class Descriptorable:
    # This is the most precise one, but I can't make it to work:                                                        
    # _descriptor_instances: WeakKeyDictionary[Any, S] 

    # Also not working, although it looks innocent to me:
    # _descriptor_instances: WeakKeyDictionary[Any, Descriptorable]

    _descriptor_instances: WeakKeyDictionary
  
    def __init__(self):  
        self._descriptor_instances = WeakKeyDictionary()  

    # This is not precise enough, see reveal_type()s below
    # def __get__(self, instance, owner) -> Descriptorable:                                                                                                               
    def __get__(self: S, instance, owner) -> S:                                                                          
        if instance is None:                                                                                             
            return self                                                                                                  
        if instance in self._descriptor_instances:                                                                       
            descriptor_instance = self._descriptor_instances[instance]                                                   
        else:                                                                                                            
            descriptor_instance = self._create_descriptor_instance(instance)                                             
            self._descriptor_instances[instance] = descriptor_instance                                                   
        return descriptor_instance                                                                                       
                                                                                                                         
    def _create_descriptor_instance(self, instance):                                                                     
        raise NotImplementedError                                                                                        

T = TypeVar('T')                                                                                                         
                                                                                                                         
# This class subclasses Descriptorable, so it can be used as descriptor.                                             
class DataContainer(Descriptorable, Generic[T]):  
    data_type: Type[T]  
    data: T  
  
    def __init__(self, data: T, data_type: Type[T]):  
        super().__init__()  
        self.data_type = data_type  
        self.data = data  
  
    def _create_descriptor_instance(self, instance) -> DataContainer[T]:  
        return DataContainer(self.data, self.data_type)

    def get_data(self) -> T:
        return self.data

    def set_data(self, data: T) -> None:
        self.data = data

class MyClass:
    data = DataContainer(2.0, float)

m = MyClass()
reveal_type(m.data)    # Should be: DataContainer[float]
reveal_type(m.data._descriptor_instances)   # Should be WeakKeyDictionary[Any, DataContainer[float]]     

The hardest part is how to annotate _descriptor_instances: WeakKeyDictionary[Any, S] (see comments in the above code). I've also tried making the Descriptorable class generic, but with no success.

Any help would be much appreciated. And I really think that descriptors deserve multiple paragraphs, if not a whole page, in the documentation.

[eric-wieser]

Was the verdict here that code using the dataclass-style pattern is wrong by design?

class _StringField:
    def __get__(self, obj, objtype) -> str:
        return "string"

class MyClass:
    x: str = StringField()  # what is the ruling on this annotation

That is, is the marked annotation quite deliberately rejected by mypy now and for all time, or is there a possibility that a patch to make this legal would be considered?

[msullivan]

I think we'd probably consider it?

[jmehnle]

Because it took me about 90 minutes trying to figure out how to do this today, I'm leaving this here for posterity:

from typing import Any, Generic, TypeVar, Union, overload

T = TypeVar("T")

class MyDescriptor(Generic[T]):
    ...

    @overload
    def __get__(self, instance: None, owner: Any) -> "MyDescriptor": ...

    @overload
    def __get__(self, instance: object, owner: Any) -> T: ...

    def __get__(self, instance: Any, owner: Any) -> Union["MyDescriptor", T]:
        # actual implementation
        ...

[starhel]

Dataclasses and descriptors are described in https://docs.python.org/3/library/dataclasses.html#descriptor-typed-fields
Unfortunately mypy reports an error:

Argument 1 to "InventoryItem" has incompatible type "str"; expected "IntConversionDescriptor"

I've tried many suggestions from this issue and SO threads, but nothing is working correctly. How dataclasses with descriptors should be annotated?

[bszonye]

@starhel I'm encountering this now too. I have a generic descriptor class ProfileValue[T] with appropriate type hints on __get__ and __set__, but mypy reports errors with either of these dataclass field definitions:

@dataclass
class Profile:
    name: ProfileValue[str] = ProfileValue(default="TODO")

test.py: Argument "name" to "Profile" has incompatible type "str"; expected "ProfileValue[str]"

or

@dataclass
class Profile:
    name: str = ProfileValue(default="TODO")

profile.py: Incompatible types in assignment (expression has type "ProfileValue[str]", variable has type "str")

The first one seems to be the preferred declaration, based on the example in the dataclasses docs, but mypy isn't inferring type correctly from the descriptor's __set__ method (#13856). The second one works at point of instantiation but produces an error on the field definition. For what it's worth, pyright accepts the first one but not the second.