"Statement is unreachable" false positive when calling `callable` on union with `object`

[DetachHead]

def foo(value: object | str) -> None:
    if callable(value):
        print(1) # Statement is unreachable  [unreachable]

https://mypy-play.net/?mypy=master&python=3.10&flags=show-error-codes%2Callow-redefinition%2Cstrict%2Ccheck-untyped-defs%2Cdisallow-any-decorated%2Cdisallow-any-expr%2Cdisallow-any-explicit%2Cdisallow-any-generics%2Cdisallow-any-unimported%2Cdisallow-incomplete-defs%2Cdisallow-subclassing-any%2Cdisallow-untyped-calls%2Cdisallow-untyped-decorators%2Cdisallow-untyped-defs%2Cno-implicit-optional%2Cno-implicit-reexport%2Cstrict-equality%2Cwarn-incomplete-stub%2Cwarn-redundant-casts%2Cwarn-return-any%2Cwarn-unreachable%2Cwarn-unused-configs%2Cwarn-unused-ignores&gist=a72eb2ed08576c38916daccddd89fed7

[DetachHead]

oops didn't mean to click feature but i can't remove it

[sobolevn]

Sorry, but this is true. object is not callable:

>>> o = object()
>>> o()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'object' object is not callable

But, Type[object] is callable.

Or am I missing something?

[DetachHead]

sure object isn't, but a subtype of object could be

def foo(value: object | str) -> None:
    if callable(value):
        print(1) # Statement is unreachable  [unreachable]
        
foo(lambda: ...) # no error because a `Callable` is a subtype of `object`

callable(lambda: ...) # True

[sobolevn]

Subtype of any type can be callable 🙂

Do you mean that object should be treated specially, because it is a base type?

[DetachHead]

no it shouldn't be treated specifically, instead callable should narrow it. for example see how it's done in typescript

function foo(value: unknown | string) { //unknown is the typescript equivalent of object
    if (typeof value === 'function') {
        value() //narrowed to Function
    }
}

https://www.typescriptlang.org/play?#code/GYVwdgxgLglg9mABMOcAUA3AhgGxAUwC5FwBrMOAdyQB9EBnKAJxjAHMBKRAbwFgAoREMQxgiNFACeAB3xwx2PPkQBeNYgDkoSLAQaufQcOOKCaLgHoLYLEyZV8AE0RQ4iAGLho8MAOMBfAX8gA

if you look at the signature of callable, that's actually what it should be doing, since it uses a TypeGuard:

def callable(__obj: object) -> TypeGuard[Callable[..., object]]: ...

in fact, if we just copy that signature and make our own callable, the issue goes away:

from typing import TypeGuard, Callable

def callable2(__obj: object) -> TypeGuard[Callable[..., object]]: ...

def foo(value: object | str) -> None:
    if callable2(value):
        reveal_type(value) # note: Revealed type is "def (*Any, **Any) -> builtins.object"
        print(1)

def bar(value: object | str) -> None:
    if callable(value):
        reveal_type(value) # error: Statement is unreachable  [unreachable]

https://mypy-play.net/?mypy=latest&python=3.10&flags=warn-unreachable&gist=cc5b40496ef4e540a6fc7789332269ab

so it seems that the problem is that there's some sort of intrinsic behavior on callable that should just be removed?

[sobolevn]

Oh, now I can see a problem:

from typing import TypeGuard, Callable

def callable2(__obj: object) -> TypeGuard[Callable[..., object]]: ...

def foo(value: object | str) -> None:
    if callable2(value):
        reveal_type(value) # note: Revealed type is "def (*Any, **Any) -> builtins.object"
        print(1)

def bar(value: object | str) -> None:
    if callable(value):
        reveal_type(value) # error: Statement is unreachable  [unreachable]

This is inconsistent and probably should be solved. I will take a look, thanks a lot for your explanation! 👍

[sobolevn]

Right now in typeshed callable is defined as def callable(__obj: object) -> TypeGuard[Callable[..., object]]: ....

But, mypy does some in-depth analysis of callable() function. During this process we do partition by callable / non callable types to narrow types more precisely. Source:

mypy/mypy/checker.py

Lines 4062 to 4142 in 56684e4

	def partition_by_callable(self, typ: Type,
	unsound_partition: bool) -> Tuple[List[Type], List[Type]]:
	"""Partitions a type into callable subtypes and uncallable subtypes.
	Thus, given:
	`callables, uncallables = partition_by_callable(type)`
	If we assert `callable(type)` then `type` has type Union[*callables], and
	If we assert `not callable(type)` then `type` has type Union[*uncallables]
	If unsound_partition is set, assume that anything that is not
	clearly callable is in fact not callable. Otherwise we generate a
	new subtype that *is* callable.
	Guaranteed to not return [], [].
	"""
	typ = get_proper_type(typ)
	if isinstance(typ, FunctionLike) or isinstance(typ, TypeType):
	return [typ], []
	if isinstance(typ, AnyType):
	return [typ], [typ]
	if isinstance(typ, NoneType):
	return [], [typ]
	if isinstance(typ, UnionType):
	callables = []
	uncallables = []
	for subtype in typ.items:
	# Use unsound_partition when handling unions in order to
	# allow the expected type discrimination.
	subcallables, subuncallables = self.partition_by_callable(subtype,
	unsound_partition=True)
	callables.extend(subcallables)
	uncallables.extend(subuncallables)
	return callables, uncallables
	if isinstance(typ, TypeVarType):
	# We could do better probably?
	# Refine the the type variable's bound as our type in the case that
	# callable() is true. This unfortunately loses the information that
	# the type is a type variable in that branch.
	# This matches what is done for isinstance, but it may be possible to
	# do better.
	# If it is possible for the false branch to execute, return the original
	# type to avoid losing type information.
	callables, uncallables = self.partition_by_callable(erase_to_union_or_bound(typ),
	unsound_partition)
	uncallables = [typ] if len(uncallables) else []
	return callables, uncallables
	# A TupleType is callable if its fallback is, but needs special handling
	# when we dummy up a new type.
	ityp = typ
	if isinstance(typ, TupleType):
	ityp = tuple_fallback(typ)
	if isinstance(ityp, Instance):
	method = ityp.type.get_method('__call__')
	if method and method.type:
	callables, uncallables = self.partition_by_callable(method.type,
	unsound_partition=False)
	if len(callables) and not len(uncallables):
	# Only consider the type callable if its __call__ method is
	# definitely callable.
	return [typ], []
	if not unsound_partition:
	fake = self.make_fake_callable(ityp)
	if isinstance(typ, TupleType):
	fake.type.tuple_type = TupleType(typ.items, fake)
	return [fake.type.tuple_type], [typ]
	return [fake], [typ]
	if unsound_partition:
	return [], [typ]
	else:
	# We don't know how properly make the type callable.
	return [typ], [typ]

I am not sure how to solve this problem 🤔

On one hand, I understand that some subtypes of any types can be callble. And TypeGuard case shows us this. On the other hand, using types that do not declare __call__ as their contract and still use them is not very good. I would probably what to see object | str | SomeCallableProtocol as the argument type here.

[finite-state-machine]

An Optional[object] (i.e., a Union[object, NoneType]) also triggers this issue.

This was particularly surprising to me because reveal_type(callable(...)) gives bool, not Literal[False].

mypy-play.net

from __future__ import annotations
from typing import Optional


def func1(value: Optional[object]) -> None:
    if callable(value):
        _ = True  # got: Statement is unreachable [unreachable]
                  # expected: no error


# ADDITIONAL MATERIAL - NOT NECESSARY TO REPRODUCE:

def func2(value: object) -> None:
    if callable(value):
        _ = True  # no error (as expected)
        
obj1: object
reveal_type(callable(obj1))  # Revealed type is 'bool' (as expected)

obj2: Optional[object]
reveal_type(callable(obj2))  # Revealed type is 'bool' (as expected)

[KotlinIsland]

Could we just simplify the union here?

@finite-state-machine Optional[object] is identical to object, NoneType is a subtype of object.

[finite-state-machine]

@KotlinIsland wrote:

Optional[object] is identical to object, NoneType is a subtype of object.

Oh, I agree, there's no point in writing Optional[object]. For me, this came up with a TypeVar with its bound implicitly set to object (or so I assume), in code that looked roughly like this:

from __future__ import annotations
from typing import *

T = TypeVar('T')

class SomeClass:

    def __init__(self, magic_test: T):
        self.magic_test = magic_test

    def is_magic(self, value: object) -> bool:
        if callable(self.magic_test):
            return self.magic_test(value)  # "unreachable"
        return value is self.magic_test

An earlier revision of @finite-state-machine had the idea – probably not very good one, in retrospect – to allow "magic" tests such as the following:

# where values can't be 'None' anyway:

inst = SomeClass(magic_test=None)


# a specific, undifferentiated 'object' might act as the sentinal (again, using the 'is' test):

definitely_magic = object()
inst = SomeClass(magic_test=definitely_magic)


# where it's helpful to have more than one distinct value considered 'magic':

@dataclass
class MagicClass:
    detail_a: int
    detail_b: str
    
inst = SomeClass(magic_test=lambda _value: isinstance(_value, MagicClass))

While I certainly agree that:

value: Union[CallableThing, NotObviouslyCallableThing]
if callable(value):
    ...
else:
    ...

... should narrow the type for both the if and else branches, my instinct is to say object should never be narrowed until/unless it becomes <nothing>/NoReturn.