AnyOf - Union for return types

[srittau]

Sometimes a function or method can return one of several types, depending on the passed in arguments or external factors. Best example is probably open(), but there are other examples in the standard library, like shutil.copy()[1]. In many of those cases, the caller knows what return type to expect.

Currently there are several options, but none of them is really satisfactory:

• Use @overload. This is the best solution if it can be used. But that is often not the case, like in the examples above.
• Use a mypy plugin. This solution does not scale to outside the standard library (and arguably not even inside it) and is mypy-specific.
• Use Union as the return type. This is usually not recommended, since it means that the caller needs to use isinstance() to use the return type.
• Use Any as the return type. This is currently best practice in those cases, but of course provides no type safety at all.

Therefore, I propose to add another type, for example AnyOf[...] that acts like Union, but can be used everywhere any of its type arguments could be used.

from datetime import date
x: AnyOf[str, date] = ...
s: str = x  # ok
dt: date = x  # ok
i: int = x  # type error
u: Union[str, bytes] = x  # ok
x = u  # type error (although the type checker could do something smart here and infer that u can only be str here)

I also think that the documentation should make it clear that using AnyOf is a code smell.

[1] Currently the type behaviour in shutil is broken in my opinion, but that does not change the fact that currently it is as it is.

[srittau]

Another use case - at least util we get a more flexible Callable syntax - is for optional arguments in callback functions, like WSGI's start_response():

StartResponse = AnyOf[
    Callable[[str, List[(str, str)]], None],
    Callable[[str, List[(str, str)], ExcInfo], None]
]

def create_start_response() -> StartResponse:
    ...

create_start_response()("200 OK", [])

Using Union this causes a type error. (Too few arguments.)

[gvanrossum]

This would be as unsafe as Any though right? E.g. def foo() -> Union[int, str] -- we have no idea whether foo() + 1 is safe or not. Sure, it can tell you that foo().append(1) is definitely wrong, but that's pretty minor.

Similarly we won't know if create_start_response()("200 OK", [], sys.exc_info()) will be accepted or not. If you meant to describe a situation where it returns a callback that can be called with or without the extra argument, there's already a solution: https://mypy.readthedocs.io/en/latest/additional_features.html?highlight=mypy_extensions#extended-callable-types.

[srittau]

Good to know that there is a proper solution for the callback case!

Personally, I think the improvement in type safety over just returning Any would be worth it. It surely can't catch all problematic cases, but some is better than none. And the issue seems to crop up from time to time, for example just after I opened this in python/typeshed#2271. That the problem of returning Unions is also explicitly mentioned in the docs is also noteworthy, I think.

[ilevkivskyi]

In my experience I never had a situation where I needed unsafe unions. Anyway, I could imagine some people might want it. However, the problem is that the benefits of unsafe unions are incomparable with the amount of work to implement them. Adding a new kind of types to mypy e.g. could take months of intense work. This one will be as hard as introducing intersection types, and while the later are more powerful (they cover most of use cases of unsafe unions) we still hesitate to start working on it.

[JukkaL]

I'm with Ivan, and this is something we've considered earlier -- see the discussion at python/mypy#1693, for example. The relatively minor benefits don't really seem worth the extra implementation work and complexity.

This would only be a potentially good idea for legacy APIs that can't be typed precisely right now, and the most popular of those can be special cased by tools (e.g. through mypy plugins). Mypy already special cases open and a few other stdlib functions. Alternatively, we might be able to use some other type system improvements, such as literal types, once they become available. For new code and new APIs the recommended way is to avoid signatures that would require the use of AnyOf anyway.

Ad-hoc extensions have the benefit of being easy to implement. They are also modular, don't complicate the rest of the type system, and they potentially allow inferring precise return types.

There is also often a simple workaround -- write a wrapper function around the library function with an Any return that has a precise return type, by restricting the arguments so that the return type can be known statically. Example:

def open_text(path: str, mode: str) -> TextIO:
    assert 'b' not in mode
    return open(path, mode)

def open_binary(path: str, mode: str) -> BinaryIO:
    assert 'b' not in mode
    return open(path, mode + 'b')

[JukkaL]

Some use cases (such as open) can now be supported pretty well by using overloads and literal types (PEP 586).