Unpacking an iterable in a list comprehension leads to the type inferred as list[Any]

[hvenev]

Bug Report

When a value of a user-defined class with __iter__ is converted into a list using [*value], the type of the list is inferred as list[Any].

To Reproduce

import typing

class Spam:
    def __iter__(self, /) -> typing.Iterator[int]:
        yield 1

a = Spam()

# list[int]
reveal_type(list(a))

# list[int]
reveal_type([i for i in a])

# list[int]
reveal_type([*(i for i in a)])

# list[int]
reveal_type([*a.__iter__()])

# list[Any] ???
reveal_type([*a])

b, = a
# int
reveal_type(b)

Expected Behavior

The type of [*a] is list[int]

Actual Behavior

example.py:22: note: Revealed type is "builtins.list[Any]"

Your Environment

• Mypy version used: 1.4.1
• Mypy command-line flags: --strict
• Mypy configuration options from mypy.ini (and other config files):
• Python version used: 3.11.4

[kamilturek]

I'll give it a try.

[JaylenLuc]

Are you still working on it? I can give it a try as well. Any insights to the issue?

[kamilturek]

@JaylenLuc Please feel free to pick it up. I couldn't come up with a solution to it. What I got is that in the example posted in the issue, [*a] is treated as a generic callable call and its type is inferred somewhere around this place (sorry for the impreciseness, I looked at it a few weeks ago).

mypy/mypy/checkexpr.py

Lines 1660 to 1687 in 9e1f4df

	if callee.is_generic():
	need_refresh = any(
	isinstance(v, (ParamSpecType, TypeVarTupleType)) for v in callee.variables
	)
	callee = freshen_function_type_vars(callee)
	callee = self.infer_function_type_arguments_using_context(callee, context)
	if need_refresh:
	# Argument kinds etc. may have changed due to
	# ParamSpec or TypeVarTuple variables being replaced with an arbitrary
	# number of arguments; recalculate actual-to-formal map
	formal_to_actual = map_actuals_to_formals(
	arg_kinds,
	arg_names,
	callee.arg_kinds,
	callee.arg_names,
	lambda i: self.accept(args[i]),
	)
	callee = self.infer_function_type_arguments(
	callee, args, arg_kinds, arg_names, formal_to_actual, need_refresh, context
	)
	if need_refresh:
	formal_to_actual = map_actuals_to_formals(
	arg_kinds,
	arg_names,
	callee.arg_kinds,
	callee.arg_names,
	lambda i: self.accept(args[i]),
	)

[JaylenLuc]

working on it. made progress, ill get back to you with my findings

[JaylenLuc]

preface: I'm a first time open source contributor :)

In the infer_constraints_for_callable() function in constraints.py it calls infer_constraints() towards the end of the function which produces "[1 :> Any]" when it should produce: "[1 :> builtins.int]".

It may be because of a previous function call not far before it in the same function called mapper.expand_actual_type() that produces "Any" instead of builtins.int. This may be because its first argument "actual_arg_type" may be incorrect. Why I say this is because "actual_arg_type" is "Spam " when i think it should be "typing.Iterator[builtins.int]" or typing.generator[builtins.int].

The aforementioned "actual_arg_type" is infer_constraints_for_callable()'s second positional argument : "arg_types: Sequence[Type | None]". This argument is supplied by infer_function_type_arguments as "arg_types" in infer.py which is called by another function of the same name in checkexpr.py. The erroneous type (Spam instead of builtins.generator) is supplied by infer_arg_types_in_context() called in the same function before in infer_function_type_arguments(). This infer_arg_types_in_context() function calls accept which actually interprets the type correctly when it calls visit_int_expr() defined in checkexpr.py which calls infer_literal_expr_type() also defined in checkexpr.py.

I dont know what is causing the arg_types parameter to be Spam instead of int in infer_function_type_arguments in infer.py. This may be a problem that has to do with Python instead of MyPy but I have no clue. But it may not because Python actually knows what the yield type of generators are when you use the builtin : type() function on *a ( type(*a) --> <class 'int'>)

Should the generator unpacking expression that is erroneous be a mypy.nodes.NameExpr instead of mypy.nodes.NameExpr?
And should the argument kind be a ArgKind.ARG_POS instead of ArgKind.ARG_STAR?

How do you all reckon about this issue? @hauntsaninja @kamilturek @hvenev

[JaylenLuc]

im trying to figure out how the * unpacking unary operator works. is [*a.iter()] the same as reveal_type([*a])?

[hauntsaninja]

I didn't understand that comment, but my guess is you'd need to trace the code into argmap.py

[JaylenLuc]

I didn't understand that comment, but my guess is you'd need to trace the code into argmap.py
thanks, working on it

[JaylenLuc]

in checkexpr.py, the function infer_arg_types_in_context() on line 1985 returns [testing.Spam] when it should return typing.Iterator[builtins.int].

mypy/mypy/checkexpr.py

Lines 1984 to 2008 in 8b73cc2

	with self.msg.filter_errors():
	arg_types = self.infer_arg_types_in_context(
	callee_type, args, arg_kinds, formal_to_actual
	)
	arg_pass_nums = self.get_arg_infer_passes(
	callee_type, args, arg_types, formal_to_actual, len(args)
	)
	pass1_args: list[Type | None] = []
	for i, arg in enumerate(arg_types):
	if arg_pass_nums[i] > 1:
	pass1_args.append(None)
	else:
	pass1_args.append(arg)
	inferred_args, _ = infer_function_type_arguments(
	callee_type,
	pass1_args,
	arg_kinds,
	arg_names,
	formal_to_actual,
	context=self.argument_infer_context(),
	strict=self.chk.in_checked_function(),
	)

This testing.Spam object is an Instance object where testing.Spam.args length is 0. This means that MyPy thinks testing.Spam has no arguments when it checks in argmap.py line 192 and returns AnyType(TypeOfAny.from_error) on line 217 instead of the result from map_instance_to_supertype().

mypy/mypy/argmap.py

Lines 191 to 219 in 8b73cc2

	if actual_kind == nodes.ARG_STAR:
	if isinstance(actual_type, Instance) and actual_type.args:
	from mypy.subtypes import is_subtype
	if is_subtype(actual_type, self.context.iterable_type):
	return map_instance_to_supertype(
	actual_type, self.context.iterable_type.type
	).args[0]
	else:
	# We cannot properly unpack anything other
	# than `Iterable` type with `*`.
	# Just return `Any`, other parts of code would raise
	# a different error for improper use.
	return AnyType(TypeOfAny.from_error)
	elif isinstance(actual_type, TupleType):
	# Get the next tuple item of a tuple *arg.
	if self.tuple_index >= len(actual_type.items):
	# Exhausted a tuple -- continue to the next *args.
	self.tuple_index = 1
	else:
	self.tuple_index += 1
	return actual_type.items[self.tuple_index - 1]
	elif isinstance(actual_type, ParamSpecType):
	# ParamSpec is valid in *args but it can't be unpacked.
	return actual_type
	else:
	return AnyType(TypeOfAny.from_error)
	elif actual_kind == nodes.ARG_STAR2:
	from mypy.subtypes import is_subtype

The main issue is that [*a] cannot be unpacked and thus the types of what the Spam object yields is not unpacked.

how do you all reckon?

[hauntsaninja]

Yeah, I'm guessing that code dates back before structural types and Protocols. Probably want to swap out the stuff in argmap for some code from checker that analyses the return type of __iter__, like analyze_iterable_item_type. Might need to thread the checker through to make this work

[JaylenLuc]

submitted a PR. if you check the PR comment I made, the only issues I'm having are with dmypy_server.py which is strange since I don't know how my changes are affecting it.

[flaeppe]

I think #14470 reports the same thing as well

[flaeppe]

And I'm not sure if my quite naive attempt in #14496 could provide any additional insights in how to resolve the issue (or if it resolves the repro case found here)