improve generic typing

[davidism]

Improves things a bit more based on feedback in #374.

[Daverball]

Suggested change

	def loads(self, s: t.AnyStr) -> t.Any: ...
	def dumps(self, obj: t.Any, *args: t.Any, **kwargs: t.Any) -> t.AnyStr: ...
	def loads(self, s: t.AnyStr, /) -> t.Any: ...
	def dumps(self, obj: t.Any, /, *args: t.Any, **kwargs: t.Any) -> t.AnyStr: ...

The names of these arguments should not matter, so it's better to make them positional only, the actual implementation can give the same argument any name it wants to.

[davidism]

This just makes mypy dislike modules as well, instead of only pyright.

src/itsdangerous/serializer.py:79: error: Incompatible types in assignment (expression has type Module, variable has type "_PDataSerializer[Any]")  [assignment]
        default_serializer: _PDataSerializer[t.Any] = json  # pyright: ignore
                                                      ^~~~
src/itsdangerous/serializer.py:79: note: Following member(s) of Module "json" have conflicts:
src/itsdangerous/serializer.py:79: note:     Expected:
src/itsdangerous/serializer.py:79: note:         def dumps(Any, /, *args: Any, **kwargs: Any) -> Any
src/itsdangerous/serializer.py:79: note:     Got:
src/itsdangerous/serializer.py:79: note:         def dumps(obj: Any, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: Optional[Type[JSONEncoder]] = ..., indent: Union[None, int, str] = ..., separators: Optional[Tuple[str, str]] = ..., default: Optional[Callable[[Any], Any]] = ..., sort_keys: bool = ..., **kwds: Any) -> str
Found 1 error in 1 file (checked 8 source files)

[Daverball]

Right, that probably avoids the special casing. The thing is, this is pretty fragile, because it's not part of the typing spec. I think it might be better to go with something like this, as ugly as it is:
Code sample in pyright playground

import typing as t
import typing_extensions as te

class _PDataSerializer(t.Protocol[t.AnyStr]):
    def loads(self, s: t.AnyStr, /) -> t.Any: ...
    @property
    def dumps(self) -> t.Callable[te.Concatenate[t.Any, ...], t.AnyStr]: ...

import json
import pickle

x: _PDataSerializer[str] = json
y: _PDataSerializer[bytes] = pickle

[Daverball]

This is the kind of black magic you have to use, if you want to force a partially gradual callable signature.

[davidism]

Which part isn't part of the typing spec? I'd rather stay within the spec, even if it's less precise or reports errors, and have users report issues to the tools if that's not enough, rather than putting in weird fixes that I then need to keep track of.

[Daverball]

@davidism looks like Concatenate support with ... was not quite correct yet in mypy 1.6. All newer versions are fine. If you want to support earlier versions you can use just ... on its own. It's only slightly less type safe.

[davidism]

I have mypy 1.9 installed, that's what's failing. I'm really not clear what I should do at this point. I appreciate your help. But what started out as a "simple" request has turned into hours of me once again being frustrated by the inaccuracy/imprecision/unexpressiveness of typing compared to how Python actually works. I know you're involved in typing, so it would probably be helpful to take this field report back to the spec team or tool maintainers.

[Daverball]

My involvement in typing goes only so far, that I am a power user that occasionally contributes to typeshed and typing discussions. I agree that in particular the cases, where you need more precise gradual types than Any are often frustrating and difficult to get correct and in APIs that were written before type hints were a thing you are trapped in a situation where you can't change the API to make it easier for tools to understand.

It took me a really long time and a good portion of my sanity to come up with type stubs for WTForms that were reasonably user-friendly, so I can relate to the frustration you are experiencing. But I also like puzzles and hate giving up, so here I am.

Concatenate[Foo, ...] is a relatively new trick in the toolbox, so it is very much possible it doesn't always reliably work. Looking back at my original issue, I did suggest adding a fallback overload for the cases where the structural type matching fails, so you can still help the type checker out by providing an explicit type, rather than getting overload errors.

I will create a follow-up PR based on this one that should address the remaining issues, I will make sure to thoroughly test that everything type checks including some simple usage examples. Thanks for putting up with this in the meantime.

[Daverball]

Alright I think the problem was that I didn't try this with --strict enabled. I'm guessing --extra-checks specifically is what's biting us here, since it changes the behavior of Concatenate and even though we're matching structurally it forces an exact signature match, so if the argument isn't positional only in the implementation, that's an error. This sounds like a mypy bug to me, since that's not how Protocol is supposed to work, I will report this, if there's not already a report.

In the meantime I will use ... and add a comment referring to the mypy issue that's blocking this.

[Daverball]

@davidism continued in #377