redo type annotations

[davidism]

Rewrite the type annotations by hand. The previous annotations generated from MonkeyType were too inaccurate, and trying to modify them after the fact was frustrating, as fixing one part would cause errors in others. Wrote a RedBaron script to remove all existing annotations.

Tried to enable as many Mypy checks as possible, but had to make some compromises in the interest of time. Mypy is nice overall, but it has a lot of corner cases or unimplemented features that Werkzeug's code seems to trigger quite a lot. This can be addressed by future contributors.

• allow_redefinition should be removed.
• disallow_untyped_defs and disallow_incomplete_defs should be enabled (although can be disabled for tests).
• warn_return_any should be enabled.
• no_implicit_reexport should be enabled.
• other things related to untyped code and Any should be enabled
• remove as many type: ignore comments and cast() calls as possible. Ignore comments can hide other issues on the same line.
• It would be interesting to see how other typing tools such as pyright and pyre handle the current annotations.

I tried to modify as little code as possible (besides the annotations). Occasionally I had to rewrite an instance or none check, or rename variables to avoid redefinition. There were a few places that typing seemed to catch a legitimate bug (such as calling dict.get and getting None). All tests still pass.

I removed the werkzeug.types module, as it didn't end up being necessary. I imported typing as t to make the annotations shorter (t.Optional instead of typing.Optional). That also avoids importing a ton of names from typing into every module. Typeshed provides a wsgiref.types stub that came in handy for annotating WSGI environs and callables. Flake8 (through PyFlakes) doesn't understand @t.overload, it only understands @typing.overload (PyCQA/pyflakes#561), so in some files there's both import typing and import typing as t.

Inheriting from typing.Generic makes instantiation many times slower until Python 3.9. This is unacceptable for Werkzeug where I estimate ~10 are created per request. I wanted the data structures to be generic, so that code is typed in a stub pyi file instead. Unfortunately, that disables all type checking on the py file and only consider the pyi file (python/mypy#5028), so there was no type checking done on the datastructures.py code itself. I considered this acceptable, as I already know Werkzeug's code is passing tests, and I'd rather be able to do MultiDict[str, str] in user code instead.

Much of the complexity came from Werkzeug allowing both str and bytes in many places, even where it no longer makes much sense since Python 2 compatibility is gone. Additionally, there are some odd code paths that don't appear to be used anywhere. Should look at deprecating these types of things in a future release.

[graingert]

@davidism it looks like you could backport the 3.9 typing.Generic into a compat.py and use it like:

if TYPE_CHECKING:
    from typing import Generic
else:
    from .compat import Generic

[davidism]

Thanks, but I'm not interested in maintaining compat files anymore. See: removal of Python 2. All that does is mean I have to pull in code that's not about Werkzeug, then keep track of removing it years later, and make sure it keeps working with Mypy and other tools in the mean time.