[red-knot] Stricter parsing for type[] type expressions

[AlexWaygood]

Summary

This PR:

• Adds stricter parsing of type[] type expressions, so that invalid type forms such as type[Protocol] and type[Callable] are flagged
• Adds a helpful subdiagnostic to some invalid-type-form diagnostics that links to the typing spec grammar for type expressions and annotation expressions

Test Plan

Mdtests extended and updated

[github-actions]

mypy_primer results

Changes were detected when running on open source projects

bidict (https://github.com/jab/bidict)
+ warning[lint:redundant-cast] /tmp/mypy_primer/projects/bidict/bidict/_base.py:491:16: Value is already of type `Unknown`
- Found 4 diagnostics
+ Found 5 diagnostics

attrs (https://github.com/python-attrs/attrs)
+ error[lint:invalid-assignment] /tmp/mypy_primer/projects/attrs/tests/test_make.py:2563:13: Implicit shadowing of class `C`
+ error[lint:unsupported-operator] /tmp/mypy_primer/projects/attrs/tests/test_make.py:2573:16: Operator `<` is not supported for types `C` and `C`, in comparing `C | Unknown` with `C | Unknown`
+ error[lint:unsupported-operator] /tmp/mypy_primer/projects/attrs/tests/test_make.py:2578:16: Operator `>` is not supported for types `C` and `C`, in comparing `C | Unknown` with `C | Unknown`
- Found 679 diagnostics
+ Found 682 diagnostics

python-htmlgen (https://github.com/srittau/python-htmlgen)
- warning[lint:unused-ignore-comment] /tmp/mypy_primer/projects/python-htmlgen/test_htmlgen/attribute.py:271:55: Unused blanket `type: ignore` directive
- Found 102 diagnostics
+ Found 101 diagnostics

scrapy (https://github.com/scrapy/scrapy)
- warning[lint:unused-ignore-comment] /tmp/mypy_primer/projects/scrapy/scrapy/squeues.py:28:45: Unused blanket `type: ignore` directive
- warning[lint:unused-ignore-comment] /tmp/mypy_primer/projects/scrapy/scrapy/squeues.py:43:44: Unused blanket `type: ignore` directive
- warning[lint:unused-ignore-comment] /tmp/mypy_primer/projects/scrapy/scrapy/squeues.py:77:45: Unused blanket `type: ignore` directive
- warning[lint:unused-ignore-comment] /tmp/mypy_primer/projects/scrapy/scrapy/squeues.py:116:45: Unused blanket `type: ignore` directive
- Found 1489 diagnostics
+ Found 1485 diagnostics

typeshed-stats (https://github.com/AlexWaygood/typeshed-stats)
+ error[lint:invalid-argument-type] /tmp/mypy_primer/projects/typeshed-stats/src/typeshed_stats/serialize.py:74:41: Argument to this function is incorrect: Expected `type[Unknown]`, found `GenericAlias`
+ error[lint:invalid-argument-type] /tmp/mypy_primer/projects/typeshed-stats/src/typeshed_stats/serialize.py:89:41: Argument to this function is incorrect: Expected `type[Unknown]`, found `GenericAlias`
- warning[lint:unused-ignore-comment] /tmp/mypy_primer/projects/typeshed-stats/src/typeshed_stats/serialize.py:173:52: Unused blanket `type: ignore` directive
- Found 23 diagnostics
+ Found 24 diagnostics

operator (https://github.com/canonical/operator)
+ error[lint:invalid-type-form] /tmp/mypy_primer/projects/operator/ops/main.py:31:28: Variable of type `Never` is not allowed in a type expression
+ error[lint:invalid-argument-type] /tmp/mypy_primer/projects/operator/ops/_private/yaml.py:28:30: Argument to this function is incorrect: Expected `type[Unknown]`, found `Unknown | None`
+ error[lint:invalid-type-form] /tmp/mypy_primer/projects/operator/ops/__init__.py:353:33: Variable of type `Never` is not allowed in a type expression
+ error[lint:invalid-type-form] /tmp/mypy_primer/projects/operator/ops/__init__.py:358:33: Variable of type `Never` is not allowed in a type expression
+ error[lint:invalid-type-form] /tmp/mypy_primer/projects/operator/ops/_main.py:463:28: Variable of type `Never` is not allowed in a type expression
- Found 253 diagnostics
+ Found 258 diagnostics

Expression (https://github.com/cognitedata/Expression)
+ error[lint:invalid-argument-type] /tmp/mypy_primer/projects/Expression/tests/test_typing.py:93:22: Argument to this function is incorrect: Expected `type[@Todo(Support for `typing.TypeVar` instances in type expressions)]`, found `GenericAlias`
- Found 565 diagnostics
+ Found 566 diagnostics

comtypes (https://github.com/enthought/comtypes)
+ error[lint:no-matching-overload] /tmp/mypy_primer/projects/comtypes/comtypes/client/_create.py:144:12: No overload of function `CoGetObject` matches arguments
- Found 711 diagnostics
+ Found 712 diagnostics

psycopg (https://github.com/psycopg/psycopg)
+ error[lint:invalid-exception-caught] /tmp/mypy_primer/projects/psycopg/psycopg_pool/psycopg_pool/null_pool.py:109:20: Cannot catch object of type `Unknown | Literal[DummyConnectionTimeout] | type[Unknown]` in an exception handler (must be a `BaseException` subclass or a tuple of `BaseException` subclasses)
+ error[lint:invalid-exception-caught] /tmp/mypy_primer/projects/psycopg/psycopg_pool/psycopg_pool/null_pool_async.py:104:20: Cannot catch object of type `Unknown | Literal[DummyConnectionTimeout] | type[Unknown]` in an exception handler (must be a `BaseException` subclass or a tuple of `BaseException` subclasses)
+ error[lint:invalid-type-form] /tmp/mypy_primer/projects/psycopg/psycopg/psycopg/pq/__init__.py:42:14: Variable of type `Never` is not allowed in a type expression
+ error[lint:invalid-type-form] /tmp/mypy_primer/projects/psycopg/psycopg/psycopg/pq/__init__.py:43:16: Variable of type `Never` is not allowed in a type expression
+ error[lint:invalid-type-form] /tmp/mypy_primer/projects/psycopg/psycopg/psycopg/pq/__init__.py:44:16: Variable of type `Never` is not allowed in a type expression
+ error[lint:invalid-type-form] /tmp/mypy_primer/projects/psycopg/psycopg/psycopg/pq/__init__.py:45:16: Variable of type `Never` is not allowed in a type expression
+ error[lint:invalid-type-form] /tmp/mypy_primer/projects/psycopg/psycopg/psycopg/pq/__init__.py:46:16: Variable of type `Never` is not allowed in a type expression
+ error[lint:invalid-type-form] /tmp/mypy_primer/projects/psycopg/psycopg/psycopg/pq/__init__.py:47:20: Variable of type `Never` is not allowed in a type expression
- error[lint:invalid-exception-caught] /tmp/mypy_primer/projects/psycopg/psycopg_pool/psycopg_pool/null_pool_async.py:104:20: Cannot catch object of type `Unknown | Literal[DummyConnectionTimeout]` in an exception handler (must be a `BaseException` subclass or a tuple of `BaseException` subclasses)
- error[lint:invalid-exception-caught] /tmp/mypy_primer/projects/psycopg/psycopg_pool/psycopg_pool/null_pool.py:109:20: Cannot catch object of type `Unknown | Literal[DummyConnectionTimeout]` in an exception handler (must be a `BaseException` subclass or a tuple of `BaseException` subclasses)
- Found 1293 diagnostics
+ Found 1299 diagnostics

pydantic (https://github.com/pydantic/pydantic)
- error[lint:invalid-return-type] /tmp/mypy_primer/projects/pydantic/pydantic/dataclasses.py:282:12: Return type does not match returned value: Expected `((*args: @Todo(todo signature *args), **kwargs: @Todo(todo signature **kwargs)) -> type[PydanticDataclass]) | type[PydanticDataclass]`, found `(def create_dataclass(cls: type[Any]) -> type[PydanticDataclass]) | type[PydanticDataclass]`
+ error[lint:invalid-argument-type] /tmp/mypy_primer/projects/pydantic/pydantic/v1/generics.py:214:63: Argument to this function is incorrect: Expected `type[@Todo(Support for `typing.TypeVar` instances in type expressions)]`, found `tuple`
+ error[lint:invalid-argument-type] /tmp/mypy_primer/projects/pydantic/pydantic/experimental/pipeline.py:162:69: Argument to this function is incorrect: Expected `type[Any]`, found `type[Unknown] | Unknown | ellipsis`
- warning[lint:unused-ignore-comment] /tmp/mypy_primer/projects/pydantic/pydantic/v1/types.py:166:38: Unused blanket `type: ignore` directive
- error[lint:invalid-return-type] /tmp/mypy_primer/projects/pydantic/pydantic/v1/types.py:318:12: Return type does not match returned value: Expected `type[float]`, found `type`
+ error[lint:invalid-return-type] /tmp/mypy_primer/projects/pydantic/pydantic/v1/types.py:318:12: Return type does not match returned value: Expected `type[int] | type[float]`, found `type`
- error[lint:invalid-argument-type] /tmp/mypy_primer/projects/pydantic/pydantic/experimental/pipeline.py:162:69: Argument to this function is incorrect: Expected `type[Any]`, found `@Todo(unsupported type[X] special form) | ellipsis`
- Found 923 diagnostics
+ Found 922 diagnostics

beartype (https://github.com/beartype/beartype)
- warning[lint:unused-ignore-comment] /tmp/mypy_primer/projects/beartype/beartype/_util/cache/pool/utilcachepoolinstance.py:74:45: Unused blanket `type: ignore` directive
- warning[lint:unused-ignore-comment] /tmp/mypy_primer/projects/beartype/beartype/_util/cache/utilcachemeta.py:79:41: Unused blanket `type: ignore` directive
+ error[lint:invalid-super-argument] /tmp/mypy_primer/projects/beartype/beartype/typing/_typingpep544.py:192:15: `type[@Todo(Support for `typing.TypeVar` instances in type expressions)]` is not an instance or subclass of `Literal[_CachingProtocolMeta]` in `super(Literal[_CachingProtocolMeta], type[@Todo(Support for `typing.TypeVar` instances in type expressions)])` call
- Found 574 diagnostics
+ Found 573 diagnostics

mongo-python-driver (https://github.com/mongodb/mongo-python-driver)
+ error[lint:invalid-parameter-default] /tmp/mypy_primer/projects/mongo-python-driver/bson/codec_options.py:249:13: Default value of type `ellipsis` is not assignable to annotated parameter type `type[Unknown] | None`
- Found 557 diagnostics
+ Found 558 diagnostics

hydra-zen (https://github.com/mit-ll-responsible-ai/hydra-zen)
- error[lint:invalid-return-type] /tmp/mypy_primer/projects/hydra-zen/src/hydra_zen/structured_configs/_implementations.py:532:12: Return type does not match returned value: Expected `(*args: @Todo(todo signature *args), **kwargs: @Todo(todo signature **kwargs)) -> @Todo(unsupported type[X] special form)`, found `def wrapper(decorated_obj: Any) -> Any`
- error[lint:invalid-assignment] /tmp/mypy_primer/projects/hydra-zen/src/hydra_zen/structured_configs/_implementations.py:3474:9: Object of type `@Todo(unsupported type[X] special form) | dict` is not assignable to `DataclassOptions`
+ error[lint:invalid-assignment] /tmp/mypy_primer/projects/hydra-zen/src/hydra_zen/structured_configs/_implementations.py:3474:9: Object of type `@Todo(Support for `typing.Self`) | dict` is not assignable to `DataclassOptions`
- Found 631 diagnostics
+ Found 630 diagnostics

django-stubs (https://github.com/typeddjango/django-stubs)
- error[lint:type-assertion-failure] /tmp/mypy_primer/projects/django-stubs/tests/assert_type/views/test_generic.py:14:1: Actual type `@Todo(unsupported type[X] special form)` is not the same as asserted type `type[MyModel]`
- error[lint:type-assertion-failure] /tmp/mypy_primer/projects/django-stubs/tests/assert_type/views/test_generic.py:24:1: Actual type `@Todo(unsupported type[X] special form) | None` is not the same as asserted type `type[MyModel] | None`
+ error[lint:invalid-base] /tmp/mypy_primer/projects/django-stubs/django-stubs/db/models/manager.pyi:148:15: Invalid class base with type `type[@Todo(Support for `typing.Self`)]` (all bases must be a class, `Any`, `Unknown` or `Todo`)
+ error[lint:call-non-callable] /tmp/mypy_primer/projects/django-stubs/django-stubs/contrib/sitemaps/__init__.pyi:42:44: Method `__class_getitem__` of type `def __class_getitem__(cls, item: type[@Todo(Support for `typing.TypeVar` instances in type expressions)]) -> @Todo(Support for `typing.Self`)` is not callable on object of type `Literal[QuerySet]`
+ error[lint:type-assertion-failure] /tmp/mypy_primer/projects/django-stubs/tests/assert_type/views/test_generic.py:14:1: Actual type `@Todo(Support for `typing.TypeVar` instances in type expressions)` is not the same as asserted type `type[MyModel]`
+ error[lint:type-assertion-failure] /tmp/mypy_primer/projects/django-stubs/tests/assert_type/views/test_generic.py:24:1: Actual type `@Todo(Support for `typing.TypeVar` instances in type expressions) | None` is not the same as asserted type `type[MyModel] | None`
+ error[lint:call-non-callable] /tmp/mypy_primer/projects/django-stubs/django-stubs/contrib/admin/options.pyi:148:66: Method `__class_getitem__` of type `def __class_getitem__(cls, item: type[@Todo(Support for `typing.TypeVar` instances in type expressions)]) -> @Todo(Support for `typing.Self`)` is not callable on object of type `Literal[QuerySet]`
- Found 1415 diagnostics
+ Found 1418 diagnostics

rotki (https://github.com/rotki/rotki)
- warning[lint:unused-ignore-comment] /tmp/mypy_primer/projects/rotki/rotkehlchen/utils/mixins/enums.py:13:64: Unused blanket `type: ignore` directive
- Found 2139 diagnostics
+ Found 2138 diagnostics

static-frame (https://github.com/static-frame/static-frame)
+ error[lint:invalid-argument-type] /tmp/mypy_primer/projects/static-frame/static_frame/core/store_xlsx.py:535:25: Argument to this function is incorrect: Expected `(...) -> @Todo(Support for `typing.TypeVar` instances in type expressions)`, found `Unknown | Literal[Index]`
- error[lint:invalid-argument-type] /tmp/mypy_primer/projects/static-frame/static_frame/core/store_xlsx.py:535:25: Argument to this function is incorrect: Expected `(...) -> @Todo(Support for `typing.TypeVar` instances in type expressions)`, found `@Todo(unsupported type[X] special form) | Literal[Index]`
- warning[lint:unused-ignore-comment] /tmp/mypy_primer/projects/static-frame/static_frame/test/unit/test_store_zip.py:57:60: Unused blanket `type: ignore` directive
- Found 4751 diagnostics
+ Found 4750 diagnostics

jax (https://github.com/google/jax)
- error[lint:invalid-return-type] /tmp/mypy_primer/projects/jax/jax/_src/util.py:614:10: Return type does not match returned value: Expected `(*args: @Todo(todo signature *args), **kwargs: @Todo(todo signature **kwargs)) -> @Todo(unsupported type[X] special form)`, found `def wrapper(cls) -> Unknown`
- warning[lint:possibly-unbound-attribute] /tmp/mypy_primer/projects/jax/jax/experimental/mosaic/gpu/dialect_lowering.py:162:48: Attribute `OPERATION_NAME` on type `str | @Todo(unsupported type[X] special form) | None` is possibly unbound
+ warning[lint:possibly-unbound-attribute] /tmp/mypy_primer/projects/jax/jax/experimental/mosaic/gpu/dialect_lowering.py:162:48: Attribute `OPERATION_NAME` on type `str | type[Unknown] | None` is possibly unbound
+ error[lint:invalid-assignment] /tmp/mypy_primer/projects/jax/jax/experimental/transfer.py:45:3: Implicit shadowing of class `TransferConnection`
+ error[lint:invalid-assignment] /tmp/mypy_primer/projects/jax/jax/experimental/transfer.py:70:3: Implicit shadowing of class `TransferServer`
+ error[lint:invalid-assignment] /tmp/mypy_primer/projects/jax/jax/_src/basearray.py:174:1: Implicit shadowing of class `Array`
+ error[lint:invalid-assignment] /tmp/mypy_primer/projects/jax/jax/_src/array.py:673:3: Implicit shadowing of class `ArrayImpl`
- Found 5506 diagnostics
+ Found 5509 diagnostics

zulip (https://github.com/zulip/zulip)
- error[lint:invalid-return-type] /tmp/mypy_primer/projects/zulip/zerver/worker/base.py:65:12: Return type does not match returned value: Expected `(*args: @Todo(todo signature *args), **kwargs: @Todo(todo signature **kwargs)) -> @Todo(unsupported type[X] special form)`, found `def decorate(clazz: @Todo(unsupported type[X] special form)) -> @Todo(unsupported type[X] special form)`
- Found 4339 diagnostics
+ Found 4338 diagnostics

[AlexWaygood]

Might need to park this for now; it's not urgent and there's quite a few false positives in the mypy_primer report there.