Protocol with nested class

[finswimmer]

Bug Report

Hey,

I don't know if this a bug report, feature request or simply a question about how to do it right 😄

For proper typing I need to overload the type annotation based on the argument types. One type involved is a Protocol. Everything works as expected as long as the Protocol consists of properties or method. But as soon as I'm looking for a class within the Protocol, mypy cannot find an overload variant. (See below)

(A clear and concise description of what the bug is.)

To Reproduce

This works:

https://mypy-play.net/?mypy=latest&python=3.11&gist=d750eefc830c2080d989a7437ca518b6

This fails:

https://mypy-play.net/?mypy=latest&python=3.11&gist=0e424fbd22d314764a0d13ac9f8fe0b5

main.py:40: error: No overload variant of "create" of "Client" matches argument type "Content"  [call-overload]
main.py:40: note: Possible overload variants:
main.py:40: note:     def [_SupportsConfig <: SupportsConfig] create(self, content: _SupportsConfig) -> _SupportsConfig
main.py:40: note:     def [_SupportsConfig <: SupportsConfig] create(self, content: _SupportsConfig, result_class: None) -> _SupportsConfig
main.py:40: note:     def [_SupportsConfig <: SupportsConfig, _T] create(self, content: _SupportsConfig, result_class: type[_T]) -> _T

So, do I have to achieve my goal?

fin swimmer

[JelleZijlstra]

Having a nested class in a Protocol is weird at best and I'm not sure how I'd expect it to behave. I'd probably try to do this by defining another Protocol _ConfigProtocol for the configuration, then putting Config: _ConfigProtocol in the outer protocol.

[finswimmer]

Hey,

Having a nested class in a Protocol is weird at best and I'm not sure how I'd expect it to behave.

I agree. Unfortunately the classes that matches those protocols in my use case are generated (https://xsdata.readthedocs.io/en/latest/). Postprocessing them to inherit from a base class or doing something else is not suitable :(

Yesterday I was under the wrong impression that this issue only appears when using @overload. But actually mypy have problems with it in general. See https://mypy-play.net/?mypy=latest&python=3.11&gist=ec2cc4c5b3b726dde3790e9dc3589ad8 (PyCharm is fine with this code, that's why I thought it was ok).

I'd probably try to do this by defining another Protocol _ConfigProtocol for the configuration, then putting Config: _ConfigProtocol in the outer protocol.

I tried this, but it doesn't seem to work either: https://mypy-play.net/?mypy=latest&python=3.11&gist=ab20fa06fabac0a347abcb514fa21f5c

fin swimmer

[erictraut]

I think mypy is doing the right thing here. If the class declared in the protocol is a nominal class (not a protocol), then type compatibility rules dictate that it must match that class exactly to satisfy that protocol. It can't simply be another class that happens to have the same name.

[finswimmer]

If the class declared in the protocol is a nominal class (not a protocol), then type compatibility rules dictate that it must match that class exactly to satisfy that protocol.

I think I can agree with this. But then I would still expect, if I have an attribute within a Protocol that is another Protocol, that this is fine. But as shown in https://mypy-play.net/?mypy=latest&python=3.11&gist=ab20fa06fabac0a347abcb514fa21f5c this is not the case at the moment.

[erictraut]

@finswimmer, there are two problems with the code sample above. First, the type of Config in the protocol needs to be type[ConfigProtocol], not ConfigProtocol (since it's a class, not an instance of the class). Second, it needs to be marked read-only. This can be done by making it into a property (without a setter) within the protocol definition.

class SupportsConfig(Protocol):
    @property
    def Config(self) -> type[ConfigProtocol]:
        ...

This now type checks without errors.

[finswimmer]

Awesome, thanks @erictraut 👍 (Even it is not clear to me, why we need to mark Config as read-only.)

Trying to implement this into my real code, I've found a slightly different use case for this, which is still problematic: If I pass a class (not an instance) as a parameter to a function and try to access the nested class within this function, mypy is complaining again:

https://mypy-play.net/?mypy=latest&python=3.11&gist=f1eb051d0176aed9a0ae0b373a45e056

(It somehow makes sense, as Config is now in marked as in instance attribute due to @property).

Any clue how to cover this use case?

[erictraut]

The @property trick in my previous response works only for instance variables, not for class variables.

Config needs to be marked as read-only because if it's writable, then it is considered invariant, and the type must match exactly in the protocol. You want it to be covariant so subtypes of type[ConfigProtocol] are accepted.

I've played around with a bunch of ideas (e.g. using ClassVar or Final), but I haven't found any solutions that work. Maybe someone else who is more creative than me can come up with some ideas.

I think we can conclude that nested classes don't play well with the Python static type system. Is there a possibility that you can move away from nested classes and move to something like a class variable that refers to a TypedDict or a dataclass? Or are you stuck with a legacy interface?

In recent discussions around PEP 705, I proposed adding a ReadOnly modifier for TypedDict fields. I even prototyped this feature in pyright. I suggested that ReadOnly could be supported for protocol field definitions. That could be a potential solution for nested classes in the future.