Covariance and contravariance problem?

[hunterhogan]

Analyzing a diagnostic error

I think I finally understand the following situation. Can someone either correct my misunderstandings or shore up the concepts I describe?

Error and troubleshooting

Given this TypeAlias and this (edited) diagnostic error:

type hasDOTvalue_exprOrNone = ast.AnnAssign | ast.Return | ast.Yield
...
Argument of type 
    "(node: hasDOTvalue_exprOrNone) -> (ast.expr | None)" 
    cannot be assigned to parameter "doThat" of type 
    "(ast.Return) -> (ast.Tuple | None)"
    in function "__init__"

If I change the argument type, the error goes away.

From

(ast.Return) -> (ast.Tuple | None)

To

(ast.Return) -> (ast.expr | None)

That is still not identical to

(node: hasDOTvalue_exprOrNone) -> (ast.expr | None)

But there isn't an error.

Analysis

So, I think this means

1. ast.Return is kosher because I can pass a Callable that has a parameter of hasDOTvalue_exprOrNone or any subset of hasDOTvalue_exprOrNone, which I think is called "covariance."
2. ast.Tuple | None is not kosher because the passed Callable must be able to handle all possible return types, which is ast.expr | None, and while ast.Tuple is a subclass of ast.expr, the problem is that the passed function does not handle all possible returns: i.e., the other subclasses of ast.expr. I think this is called "contravariance."

Full error message

Argument of type "Overload[
    (node: hasDOTvalue_boolOrNone) -> (bool | None)
    , (node: hasDOTvalue_ConstantValueType) -> ConstantValueType, (node: hasDOTvalue_expr) -> expr
    , (node: hasDOTvalue_exprOrNone) -> (expr | None)]" 
    cannot be assigned to parameter "doThat" of type 
    "(Return) -> (Tuple | None)" in function "__init__"
  No overloaded function matches type "(Return) -> (Tuple | None)"

TypeAlias

type hasDOTvalue_boolOrNone = ast.MatchSingleton
type hasDOTvalue_ConstantValueType = ast.Constant
type hasDOTvalue_expr=ast.Assign | ast.Attribute | ast.AugAssign | ast.Await | ast.DictComp | ast.Expr | ast.FormattedValue \
    | ast.keyword | ast.MatchValue | ast.NamedExpr | ast.Starred | ast.Subscript | ast.TypeAlias | ast.YieldFrom
type hasDOTvalue_exprOrNone = ast.AnnAssign | ast.Return | ast.Yield
type hasDOTvalue = hasDOTvalue_boolOrNone | hasDOTvalue_ConstantValueType | hasDOTvalue_expr | hasDOTvalue_exprOrNone

Code snippet

astTuple: ast.Tuple = raiseIfNone(
	NodeTourist[ast.Return, ast.Tuple | None](
		findThis=Be.Return.valueIs(Be.Tuple)
		, doThat=Then.extractIt(DOT.value)
	).captureLastMatch(ingredientsFunction.astFunctionDef)
)

raiseIfNone

def raiseIfNone[TypeSansNone](returnTarget: TypeSansNone | None, errorMessage: str | None = None) -> TypeSansNone:
    if returnTarget is None:
        raise ValueError(errorMessage)
    return returnTarget

NodeTourist

木 = typing_TypeVar('木', bound=ast.AST, covariant=True) # "mu"
归个 = typing_TypeVar('归个', covariant=True) # "gui-ge"

class NodeTourist(ast.NodeVisitor, Generic[木, 归个]):
    def __init__(self, findThis: Callable[[ast.AST], TypeIs[木] | bool], doThat: Callable[[木], 归个]) -> None:
        self.findThis = findThis
        self.doThat = doThat
        self.nodeCaptured: 归个 | None = None

    def visit(self, node: ast.AST) -> None:
        if self.findThis(node):
            self.nodeCaptured = self.doThat(cast("木", node))
        self.generic_visit(node)

    def captureLastMatch(self, node: ast.AST) -> 归个 | None:
        self.nodeCaptured = None
        self.visit(node)
        return self.nodeCaptured

Then

class Then:
    @staticmethod
    def extractIt[个](node: 个) -> 个:
        return node

DOT

class DOT:
    @staticmethod
    @overload
    def value(node: hasDOTvalue_boolOrNone) -> bool | None:...

    @staticmethod
    @overload
    def value(node: hasDOTvalue_ConstantValueType) -> ConstantValueType:...

    @staticmethod
    @overload
    def value(node: hasDOTvalue_expr) -> ast.expr:...

    @staticmethod
    @overload
    def value(node: hasDOTvalue_exprOrNone) -> ast.expr | None:...

    @staticmethod
    def value(node: hasDOTvalue) -> ast.expr | bool | ConstantValueType | None:
        return node.value

Follow-up: new paradigm?

If I am right, then I am pretty sure there is no way to tweak my current system of checking and passing type information: in some areas, I need a new paradigm to transfer type information. I have no idea, however, what I can do in this case.

The predicate has the type information (i.e., node.value has type ast.Tuple), but I am not aware of any mechanism for transferring that information from the predicate to the action (doThat). If there is a mechanism, I am nearly certain I am not capturing the information in the predicate. I use TypeIs to capture the type of node, but even though I check the type of the attribute of node (i.e., node.value), I don't think I am capturing the information, and I don't know how to capture it.

[hunterhogan]

2. ast.Tuple | None is not kosher because the passed Callable must be able to handle all possible return types, which is ast.expr | None, and while ast.Tuple is a subclass of ast.expr, the problem is that the passed function does not handle all possible returns: i.e., the other subclasses of ast.expr. I think this is called "contravariance."

This is not quite right. I think I need to frame it from a different perspective.

Pylance is saying:

1. You assigned a Callable that might return ast.expr | None,
2. but you wrote that you expect a return of ast.Tuple " None.

Pylance is implying: That's a problem because there are values that could be ast.expr without being ast.Tuple.

Follow-up: new paradigm?

Therefore, I guess I want to find a way to "narrow" the expected return type from ast.expr to ast.Tuple. The overload definitions effectively narrow the expected return type from ast.expr | bool | ConstantValueType | None to ast.expr | None and raiseIfNone effectively narrows that to ast.expr. The only way I know to narrow it further would require ~1800 definitions covering each combination of: ast.class.attribute * (each possible type in the ast.class.attribute's annotation, including each subclass of an abstract ast.AST subclass). I don't think that is a good idea, but I could easily generate the code.

idk.