Resolution of same name attributes on intersection types.

[baluyotraf]

Sorry if this has been discussed before.

Currently, the draft PEP says that the Intersection[...] element order for the intersection does not matter, and create overloads for the intersection types. However the file under notes.txt says that the intersection of class types should follow the MRO, which relies on the order the classes are provided.

In general, this means that if the order and @overloading of the attributes is how the specification should work, the way the typing works is different from multiple inheritance actually works in Python. Given the example below:

class A:
  def __init__(self, a: int):
    self.a = a

class B:
  def __init__(self, a: str, b: int):
    self.a = a
    self.b = b

class AB(A, B):
   # Will use A's __init__ via MRO 

T_AB = A & B # Very different assumptions to the actual multiple inheritance case
# Equivalent to:
class T_AB:
  @overload
  def __init__(self, a: int)
    ...
  
  @overload
  def __init__(self, a: str, b: int)
    ...

Means that the type checker will make assumptions on code interfaces that won't exist during run time as using the second overload will result in an error.

[mikeshardmind]

Yes, intersections are different from multiple inheritance. Multiple inheritance picks an implementation based on mro, intersections declare compatibility with both uses. This is a case that needs to be detected as incompatible. There's an additional obscuring problem here in that constructors are currently special cased in subclassing. If this were not the case, your multiple inheritance example should error. (a real world case based on it almost certainly does, because B's attributes are missing from AB, and attempts to use it as AB("a", 1), compatibly with B break)

[baluyotraf]

I agree with your point that multiple inheritance and intersection types are indeed different, from a theoretical perspective.

Which makes me think if we need a full intersection types in Python or would something like a SuperType[...] or MRO[...] should be enough to cover most use cases, as intersections are mostly defined in practice using inheritance.

[mikeshardmind]

Intersections are useful outside of multiple inheritance, we have motivating examples involving protocol composition (structural subtyping) and class transforming decorators. ((T) -> (T & BehaviorAddedProtocol)) If this was only about inheritance, I agree that we probably wouldn't need full intersections, making type Generic over a typevartuple of types in resolution order would work.

[baluyotraf]

That's fair actually. And that seems to be more aligned with what's written in the notes.txt which is to follow the MRO for real classes, and actual intersection for Protocols which sort of relates on how the inheritance syntax resolves from both implementation (real classes) and typing perspective (Protocol). This means that Intersection[A, B, C, D, ...] resolves to how class SuperType(A, B, C, D, ...) resolves.

Though as you said, this is actually not how intersection works in theory.

Though going back, I'm not sure which one to actually follow, since the notes.txt and the specification.rst seems to have conflicting inputs.

[mark-todd]

@baluyotraf If you're interested in picking this up again, I might recommend reading this version of the spec too: https://github.com/CarliJoy/intersection_examples/blob/intersection-pep-new/docs/specification.rst

We all went back and forth on these quite a bit, and I drafted this up as something that expresses a "happy medium", where we have something that is essentially MRO based, but still allows for the use cases of TypedDict, Protocols and class decorators (some of the points of discussions). Couldn't reach agreement on this, but it might provide a jumping off point for a way forward.

You can also see discussions of the PR here: #49

[baluyotraf]

From what I understand, a inheritance based approach was also discussed in #38 and the current issue is how to resolve it with Any

[mikeshardmind]

Sorry, there's a lot of history of discussion on this. The approach in #38 doesn't work.

The short version:
Current best is back to fully implementing set-theoretic intersections, where:

T & Any is not a reducible expression
T & object reduces to T
T & Never reduces to Never

Resolution for overlapping attributes is to intersect the types of the attributes for existing members.

The open questions on this are much fewer now, and the majority of hangups actually involve other poorly defined parts of how typing should handle the python data model. In particular, collections.abc.Hashable has the potential to be a very large nuisance with user denotable intersections (intersections are already in the type system via TypeIs, but they are not user-denotable), because it is removable in subclasses. If this is solved in a way existing contributors to the typing ecosystem are satisfied with, I'd be willing to start drafting a formal proposal for user denotable intersections. The ground work on this blocker was taken over to discourse, though discussion on it left me with the impression that people had some issues with the method I had in mind.

[baluyotraf]

Thanks for giving me a quick summary of what's been happening so far.

The proposal on Hashable looks good to me as well. If I set __hash__ = None on a class, I'm assuming that the consumer of that class treats that as an unhashable. Other people can of course override the hashing behavior, but as the writer of the base class, the explicit setting of __hash__ = None means that I can add other internal workings inside the object that won't hash properly, making the inheritance of my code unsafe.

I think the reason why removing hashability should not be done is relatively obvious from an inheritance perspective.

Other things on intersections:

• Required/NotRequired: I guess this just resolves to Required[A & B]? The constructs represent the existence of the field, rather than the type anyway.
• Annotated: I think this just resolved to Annotated[A & B, <Annotations from A and B>]

I think it would be nice to have a place where the current status has been summarized. I know you guys have done a lot of work and thinking in this matter already. I can put some work to go through everything. I would imagine most of them are in the issues in this repository.

If someone has something that I can start with that would be nice, otherwise start with the discussions I find interesting and start from there.

[mark-todd]

Most up to date discussion is on this issue: #48 - so I would probably start there. I wouldn't try reading all of the old issues tbh as there's just an extremely large amount of discussion (most of which is now outdated or resolved).

The ordered layers spec was just only way I could find to describe a consistent interpretation at the time - I think based on what Mike's saying we're probably at the point an unordered one would be possible which would be cool!

That said there are sections in the ordered layers spec that mention previous discussions, so it's probably a good way to get the spark notes of the hot topics (without trawling hundreds of comments around Any & :) ). Also each of the headings essentially addresses each of the major issues - it might be now that we have another way to resolve each of these, but you can see at least see the issues we were trying to solve here.

From what I understand, a inheritance based approach was also discussed in #38 and the current issue is how to resolve it with Any

The Ordered layers version basically came out of the discussion on 38 - I adapted it from one Mike drafted around that discussion. It does present a solution to Any & etc but it may not be the preferable one now.

https://github.com/CarliJoy/intersection_examples/blob/intersection-pep-new/docs/specification.rst

[mark-todd]

Also re Required/NotRequired I think these weren't in Python last time around so may be new discussion.

Re Annotated I think we might have discussed this before but only briefly so might be worthy of a new issue

[mikeshardmind]

Required/NotRequired: I guess this just resolves to Required[A & B]? The constructs represent the existence of the field, rather than the type anyway.

We don't have rules for type qualifiers yet. There are a few that are obvious-ish like Final and ClassVar that should just propagate.

Required[A] & NotRequired[B] isn't possible in my interpretation no matter what A and B are due to the nature of structural subtyping, and NotRequired saying it's possible to remove that key/value pair, where Required says it must remain.

[mark-todd]

`Required[A] & NotRequired[B] isn't possible in my interpretation no matter what A and B are due to the nature of structural subtyping, and NotRequired saying it's possible to remove that key/value pair, where Required says it must remain.

I would suggest that an example that might be worth considering is something like this:

class A(TypedDict):
    x: Required[int]
    y: str

class B(TypedDict):
    x: NotRequired[int]
    z: str
a: A & B

E.g. in the above, should we ban A & B essentially because the attribute x can't exist? I would think maybe giving x the type Never could work?

[mikeshardmind]

A & B needs to be detected as invalid here in some way, otherwise this "looks" fine to the user until they go to access x.

[mikeshardmind]

I can start drafting a rougher form of what should become the formal proposal as it would exist under the assumption the hashability question will have a usable resolution (doesn't need to be the exact resolution linked) which should help serve as a point to refocus rather than ask people to dredge up all of the existing discussion.

[mark-todd]

A & B needs to be detected as invalid here in some way, otherwise this "looks" fine to the user until they go to access x.

I suppose I'm thinking about the use case of a user with two large typed dicts let's say, and they want to merge them. Thinking perhaps x -> Never would be more ergonomic? As essentially it would force them to redefine the entire class if the merge is banned.