[v8.0] Annotate ReturnValues.py

[chaen]

I originally just wanted to silent pylint for complaining about unsubscriptable-object when using convertToReturnValue. Turns out to be trickier than expected: pylint-dev/pylint#7292

Anyway, I added the type hints. Not sure whether we want to merge it already or wait for pylint to be fixed. I don't mind.

Also, as of python 3.10, we will be able to forward the type hint through the elipse https://peps.python.org/pep-0612/

Update from @cburr: This now adds type hints to the entire ReturnValues module and can propogate the type of retVal["Value"] through S_OK/convertToReturnValue/returnValueOrRaise. We can't actually use mypy for checking this until python/mypy#13389 is merged and releasd however I've tested locally with that branch and it works as expected and is able to catch a bunch of errors.

Click me to see examples of errors than can now be caught with mypy

from typing import Union

from DIRAC.Core.Utilities.ReturnValues import convertToReturnValue, DReturnType, DOKReturnType, S_OK, returnValueOrRaise


def do_thing0(x: int) -> DReturnType[int]:
    return S_OK(x * 2)


def do_thing1(x: int) -> DOKReturnType[int]:
    return S_OK(x * 2)


@convertToReturnValue
def do_thing2(x: int) -> int:
    return x * 2


@convertToReturnValue
def do_thing3(x: str) -> str:
    return x * 2


def do_thing4(x: int) -> DReturnType[int]:
    if x > 20:
        return S_OK(x)
    return S_OK()
    # ↑ error: Incompatible return value type (got "DOKReturnType[None]", expected "Union[DOKReturnType[int], DErrorReturnType]")


def do_thing5(x: int) -> Union[DReturnType[int], DReturnType[None]]:
    if x > 20:
        return S_OK(x)
    return S_OK()


retVal = do_thing0(5)
if retVal["OK"]:
    print(retVal["Value"] + 1)
    print(retVal["Value"] + "1")
    # ↑ error: Unsupported operand types for + ("int" and "str")
else:
    print(retVal["Message"])


retVal = do_thing0(5)
print(retVal["Value"] + 1)
# ↑ error: TypedDict "DErrorReturnType" has no key "Value"

retVal = do_thing1(5)
print(retVal["Value"] + 1)  # Allowed as do_thing1 can't return DErrorReturnType


retVal = do_thing2(5)
if retVal["OK"]:
    print(retVal["Value"] + 1)
    print(retVal["Value"] + "1")
    # ↑ error: Unsupported operand types for + ("int" and "str")
else:
    print(retVal["Message"])


xxx = do_thing0(5)
thing0 = returnValueOrRaise(xxx)
print(thing0 + 1)
print(thing0 + "1")
# ↑ error: Unsupported operand types for + ("int" and "str")


thing2: int = returnValueOrRaise(do_thing2(5))
print(thing2 + 1)
print(thing2 + "1")
# ↑ error: Unsupported operand types for + ("int" and "str")


retVal = do_thing3("2")
if retVal["OK"]:
    print(retVal["Value"] + "1")
    print(retVal["Value"] + 1)
    # ↑ error: Unsupported operand types for + ("str" and "int")
else:
    print(retVal["Message"])


print(retVal["Value"])
# ↑ error: TypedDict "DErrorReturnType" has no key "Value"
print(retVal["Message"])
# ↑ error: TypedDict "DOKReturnType[str]" has no key "Message"


x = do_thing5(5)
if x["OK"]:
    value = x["Value"]
    if value:
        print(value + 3)
    print(value + 3)
    # ↑ error: Unsupported operand types for + ("None" and "int")
    # ↑ note: Left operand is of type "Optional[int]"

BEGINRELEASENOTES
*Core
NEW: Introduce DReturnType/DOKReturnType/DErrorReturnType for DIRAC return structures

ENDRELEASENOTES

[fstagni]

Why is this needed?

[chrisburr]

It fixes a handful of annoying quirks when adding types. It was planned to be the default in 3.10 and then 3.11 but it has now been delayed twice due to having a different set of issues. PEP-649 has some more details but for now I think from __future__ import annotations is the best course of action for DIRAC as it's more limited than the original approach. If we change our mind removing the future imports should be easier than the opposite approach.

Technically what it does it change the type hints to being evaluated on demand rather than at function definition time. For example:

class C:
    @classmethod
    def factory_func(cls) -> C:
        return cls()

By default this fails with NameError: name 'C' is not defined. Alteratively you can make the type hint a string which causes it's evaluation to be delayed:

class C:
    @classmethod
    def factory_func(cls) -> "C":
        return cls()

But this is quite ugly and annoying and makes evaluating the type hints harder. Using from __future__ import annotations makes it so that the type hint evaluation is always delayed. See [PEP-563](https://peps.python.org/pep-0563/ for details).

(Note this example isn't quite correct if you want correct typing for subclasses of C.)

[fstagni]

Is this needed now? Are you just adding to not forget later on?

[chrisburr]

I defer to @chaen

[chaen]

if pylint isn't complaining anymore, then we can remove it :-)