Self type for a method parameter is an automatic LSP violation

[decorator-factory]

Example code:

from typing import Self

class Animal:
    def hug(self, friend: Self) -> None:
        ...

class Cat(Animal): ...
class Duck(Animal): ...

def hugs(a: Animal, b: Animal) -> None:
    a.hug(b)
    b.hug(a)

cat = Cat()
duck = Duck()

cat.hug(duck)  # not allowed
#^ error: Argument 1 to "hug" of "Animal" has incompatible type "Duck"; expected "Cat"

duck.hug(cat)  # not allowed
#^ error: Argument 1 to "hug" of "Animal" has incompatible type "Cat"; expected "Duck"

hugs(duck, cat)  # allowed

Essentially, the Animal.hug method has this property: "you can call it with any Animal". But as soon as you subclass Animal, that property is automatically violated, because only instances of that subclass are now allowed in this method. For example, if Cat.hug assumed that the friend is also a Cat, it might fail at runtime when friend is Duck.
Using a TypeVar (which Self is essentially syntax sugar for) would lead to similar issues.

Maybe Self should not be allowed as a parameter type in methods?

[erictraut]

I'm not convinced this is a problem. If it is a problem, it's definitely not unique to Self.

Self is effectively a type variable that has an upper bound of the class that contains it. Your code sample is equivalent to the following:

class Animal:
    def hug[S: Animal](self: S, friend: S) -> None: ...

Note that my code sample doesn't involve Self, yet the behavior is the same.

[decorator-factory]

Yes, maybe I should have expanded on

Using a TypeVar (which Self is essentially syntax sugar for) would lead to similar issues.

Why do you think this is not a problem? Type checkers usually flag incompatible overrides of methods, and using Self as a parameter type (or a typevar as you showed) ensures the child method will be incompatible. If the user is okay with incompatible overrides, that's fine, they can turn the reportIncompatibleMethodOverride option (in pyright) off

[carljm]

The typevar example only has the same behavior if you annotate self: S, binding the typevar to the type of self, which is a narrower type in subclasses. The explicit : Animal bound isn't even relevant to the example, since it doesn't vary.

So this is not a general problem with bound typevars, it is a problem specific to the Self type, whether that type is spelled explicitly or implicitly via typevar bound to self. The problem is because the self type (unlike a normal typevar with fixed bound) inherently varies covariantly in subclasses (i.e. the self type of Cat is a subtype of the self type of Animal.)

Since classes are contravariant in their argument types, (subclass method arguments must accept a type that is assignable from the type of the corresponding superclass method argument), I think the OP is correct that annotating any method argument other than self with the Self type (explicitly, or equivalently via typevar bound to type of self) inherently creates an invalid method override if any subclass exists, and should be flagged as an error unless the class is @final.

The same is not true for its use as a return type, since method return types are covariant, which works fine with the inherent covariance of the self type. (Unless of course you put Self inside an invariant container in the return type.)

[carljm]

After reading @grievejia's comment below and experimenting with Pyre's behavior, I think the issue here is not the specification of Self (though maybe it needs to be clearer) but rather pyright's implementation specifically.

Pyre treats Self as a type var with upper bound of Animal (and that upper bound is not implicitly narrowed in subclasses that don't override the method), which means that in the call c.hug(d) it instead resolves Self for that call to Cat | Duck and does not error on the call, which avoids the invalid override. (It also means that a method returning Self, when called on a subclass, may not be inferred to return precisely that subclass, if there are other appearances of Self in the method signature that also must be considered in resolving the typevar.)

[Akuli]

Maybe Self should not be allowed as a parameter type in methods?

Banning self in parameter types is not enough to fix this because of variance issues. Basically, return values can act similarly to parameters.

For example, consider a method that returns a list. You can then append to that list. It acts as a parameter even though it is a return value.

from typing import Self

class Animal:
    def __init__(self) -> None:
        self.relatives: list[Self] = [self]
    def get_relatives(self) -> list[Self]:
        return self.relatives

class Cat(Animal):
    def meow(self) -> None:
        ...
class Duck(Animal): ...

cat = Cat()

a: Animal = cat
a.get_relatives().append(Duck())

print(a.relatives)  # cat and duck
for relative in cat.relatives:
    relative.meow()  # AttributeError: 'Duck' object has no attribute 'meow'

From a practical point of view, I think this doesn't matter much. It is fine to return a list[Self] from a method, because it's usually a copy, and the object itself won't break if you append something else to the returned list.

[decorator-factory]

Yes, animal.get_relatives() returns a list[Animal], and cat.get_relatives() returns a list[Cat]. list[Cat] is not assignable to list[Animal], so the child method will be incompatible with the parent.

I suppose right now this kind of compatibility is simply not considered when using type variables?

[decorator-factory]

I think I'm not quite understanding the consensus on practicality/soundness tradeoffs in type checkers. If you go down the practical point of view road, you could do something like TypeScript and make list covariant, but maybe that's not very good.

[Akuli]

If you go down the practical point of view road, you could do something like TypeScript and make list covariant, but maybe that's not very good.

It looks like you do understand it :) Type checking features should be practical enough for people to use them, and sound enough to spot bugs.

Appending the wrong type to a list is a bug that happens somewhat often, and causes a lot of confusion. It doesn't help that the 3 places where the list is created, appended, and used are sometimes all over a large codebase, far apart from each other. It also doesn't help that some other languages copy lists implicitly. (I'm looking at you, C++.) So we want type checkers to complain about that.

But this corner case is more niche, doesn't come up very often in practice, and apparently would be painful to implement correctly.

[carljm]

I don't think this unsoundness is necessary for practical reasons. I think there is a practical, sound, and implementable approach to this problem, which Pyre already implements. See #1761 (reply in thread)

[grievejia]

I remember discussing this issue with Pradeep during the drafting stage of PEP 647.

Having the Self parameter type automatically "lowered" in subclasses requires type checkers implement what is called F-bounded polymorphism. We think it is a non-trivial type system feature to implement (at least in Pyre) and the added benefit was relatively small, so eventually it was decided to go with the bounded type variable approach which does not permit this kind of pattern.

If implementation complexity is not a concern, then I think it can indeed be argued that the suggested behavior of Self type is more intuitive and should be permitted. As @carljm pointed out, the current approach of "Self is effectively a type variable that has an upper bound of the class that contains it" really only works well when Self type appears co-variantly on the method signature (the linked paper names this case "positive recursion"). It's too coarse-grained to handle the case where Self appears contra-variantly, and therefore needs to be replaced with alternative formulation in the spec if we want proper support of F-bounds. I don't really have a good sense of whether the current limitation of Self is worth such a spec revision though -- if not, then we may have to admit that the type system today offers no type-safe way to express the idea of, e.g., "my method takes & only takes the 'current' class as argument".

[carljm]

I'm not sure I understand this comment entirely. "Having the Self parameter type automatically 'lowered' in subclasses" -- if I understand accurately what you mean by that -- seems to be implemented already by all type-checkers (including Pyre), and important for one of the key motivating use cases for Self (a class method that always returns the type it is called on, rather than the type it is defined on.)

What doesn't seem to be implemented is the variance checking that ought to occur, given this definition of Self.

[carljm]

Pyre's behavior does differ from pyright's, and is more consistent, in that it doesn't complain about c.hug(d) or d.hug(c). But I'm not sure how to square this with the fact that it also considers the return type of c.hug(...) to be Cat, not Animal. Is it treating it as some kind of gradual form, where it can pick the most "favorable" binding in different contexts?

EDIT: Oh, I see! It considers the return type of c.hug(d) to be Cat | Duck, and c.hug(c) to be Cat, and c.hug(a) where a is an Animal to be Animal. So it really is treating it as a typevar with upper bound of Animal.

I think that behavior is consistent and correct, and the issue here is just a pyright problem.

[grievejia]

We may be talking about the same underlying problem just with different languages and perspectives.

I remember Pradeep's original intention of PEP 673 was to dictate that this code snippet

class Base:
  def foo(self, other: Self) -> None: ...

should be treated as equivalent to this code snippet

class Base:
  def foo[T: Base](self:T, other: T) -> None: ...

It is how Self gets implemented within Pyre today -- there's literally a syntactical transformation pass for it. According to what's described here, Pyright seems to also implements what's said in the PEP faithfully.

I have not looked into details what gets implemented in other type checkers. Assume they follow the spec strictly, then I would expect the following code:

class Base:
  def foo(self, other: Self) -> None: ...
class Derived(Base):
  def foo(self, other: Self) -> None: ...

should not type check, because in the "bounded-typevar-transformed" version of the code, Base.foo() requires it can handle others being any subtype of Base, whereas Derived.foo() can only handle others being subtypes of Derived. I've verified that at least trunk Pyre would report an inconsistent override on this example.

Self type on return position works as expected though -- to clarify, that's what I meant by saying that the limitation is "having the Self parameter type automatically 'lowered' in subclasses", instead of "having the Self return type automatically 'lowered' in subclasses".

I think we both agree that modeling typing.Self with bounded typevar naively does not work well due to the inherent co-variance of Self. But my interpretation is that such kind of modeling is intended by the PEP, as opposed to being implementation issues of individual type checkers.

[carljm]

I think Pyre and Pyright both do implement the typevar equivalence you mention above, but their behavior is different on both the Self example and the equivalent typevar example, in a key way. There is a difference in how they interpret what it means for a typevar to be bound to the type of self.

Referring again to this playground example:

Pyre is willing to resolve the Self typevar to Cat | Duck in the call cat.hug(duck), or to Animal on the call cat.hug(animal). This means there is no variance problem with using Self on a method argument: Cat remains substitutable for Animal, with no LSP violation. (Note that Cat and Duck do not explicitly override the hug method; there is no issue with type-checking the implementation of Animal.hug with an upper bound of Animal for Self.)

In this same case, pyright says that when called on an instance of Cat, Self must have an upper bound of Cat, even for a method defined only on Animal. This leads it to reject the calls cat.hug(duck) and cat.hug(animal). I think in these cases Pyright's behavior leads to LSP violation (a Cat is no longer substitutable for an Animal, because Cat.hug won't accept all the arguments Animal.hug will), and Pyre's is sound.

When we get to defining a Horse subclass that does override Animal.hug and use Self types, here I'm not so sure about Pyre's behavior either. I think Pyre should flag these as incompatible overrides (because Self on a method defined on Horse must have an upper bound of Horse, not Animal, so the explicit method override is invalid because it violates contravariance of argument types.)

[grievejia]

Ah thanks for the explanation! I think I finally get your point here.

IIUC, the key difference between Pyre and Pyright here is that if a method inherits a Self signature from its parent, whether we should treat it as being bounded by the parent class or being bounded by the current class. Totally agree that the Pyre approach (i.e. inheriting the bound from parent) leads to generally more consistent behavior.

When the method gets explicitly overridden (e.g. in your Horse example), I also agree that an inconsistent override error should be reported. The reason why Pyre playground did not report the issue today is that the problem only got fixed very recently and the patch was not yet deployed to the playground.

Whether typing.Self should be modeled as a bounded typevar is an independent issue unrelated to OP's question. That part was a misunderstanding from my side. Apologies about the confusion!