Do not report higher-kinded types as Anys with --disallow-any=expr

[ilinum]

Do not report higher-kinded types as Anys with --disallow-any=expr as well as in --any-exprs-report and --html-report

We can't fully represent higher-kinded types (e.g. calls to NewType) in mypy's type system,
so we treat them as 'Any' and give specific errors for each higher-kinded type (e.g. NewType).

Another design I considered is either returning object type or having a new HigherKindedType type.
The problem with that approach is that because now a call to NewType() does not have type 'Any'
if something is wrong with the call, it reports redundant errors.
Consider the following situation:

C = NewType('C', int)
C = NewType('C', str)

If NewType(...) has type object, in addition to all the expected errors,
it will also report Incompatible types in assignment error.

[ilinum]

I am not sure this is the best solution but I've been thinking about it and couldn't come up with anything better.

[ilevkivskyi]

As a random thought, I think NewType can be quite well approximated as a generic Callable[[str, Type[T]], Type[T]].

[JukkaL]

As a random thought, I think NewType can be quite well approximated as a generic Callable[[str, Type[T]], Type[T]].

Hmm I'm not sure that such approximations would be generally useful -- they may just be confusing.

[JukkaL]

Looks mostly fine, but I'm not excited about the naming.

[JukkaL]

I'd rename this to something more general, since we don't really take advantage of the fact that these represent type constructors anywhere, and the concept of "higher-kinded type" sounds kind of complicated. My suggestion would be special_form, which can be applied for things other than type constructors that don't have meaningful types but that also aren't imprecisely typed.

[ilinum]

Yeah, that does make sense. Updated!

[gvanrossum]

I'm confused by the PR description. What is the problem you are trying to solve, and what is your solution? The answers to these questions aren't so clear from the current text.

[ilinum]

Hmm, wow, yeah, I missed a big part of explanation. Thanks for pointing it out!

Higher-kinded types (e.g. type of call to NewType) don't fit in mypy's type system, so we represent them as Any.
The problem is that --disallow-any=expr (and the `--html-report) reports these expressions as being untyped when they actually aren't.

The solution proposed in this PR is to tag these Anys as "special form" and then don't report such Anys in --disallow-any=expr flag.

[gvanrossum]

I still don't understand -- in what sense doesn't NewType() fit in the type system? Where does it get turned into Any? When I do reveal_type(NewType('C', int)) I see Type[int], essentially, which seems fine and not at all Any.

[ilevkivskyi]

I think he means something like just reveal_type (NewType).

[ilinum]

No, I do mean type of call to NewType. So the way mypy does this is not very intuitive: NewType('C', int) by itself is just a CallExpr and according to the stubs it returns Type[int], so reveal_type(NewType('C', int)) outputs Type[int].
However, in an assignment C = NewType('C', int)) that last part after the assignment is actually a NewTypeExpr.
So if you run html report on the following file:

from typing import NewType

C = NewType('C', int)

line with the NewType declaration would be colored red because NewType('C', int) has type Any.

Now, if we change NewTypeExpr to have type Type[<whatever parameter is>] we start getting redundant errors if something is wrong with the declaration of NewTypeExpr.

[ilevkivskyi]

@JukkaL

Hmm I'm not sure that such approximations would be generally useful -- they may just be confusing.

OOC I just checked and it is already defined in typeshed that way. So I didn't invent anything new here :-)

@ilinum

However, in an assignment C = NewType('C', int)) that last part after the assignment is actually a NewTypeExpr.

Maybe I am missing something, but IIRC it is not. The NewTypeExpr (and other similar nodes) are stored in various .analyzed attributes (on calls, class definitions, index expressions etc). So maybe a right solution will be just not look into there while searching for Anys?

[ilinum]

@ilevkivskyi that is an interesting suggestion. If that works, it might better than what's currently in the PR.

My concern is that we can miss some Anys if we don't look in CallExpr.analyzed.

Moreover, for --disallow-any=expr we are running a TypeQuery over a type to see if it contains an Any. So by that time CallExpr.analyzed is already analyzed and that CallExpr's type comes from `analyzed.

[JukkaL]

@ilinum Let's keep the implementation as is (with the renaming that I suggested above), at least if you want to get it merged before release. We can refine this afterwards, but the idea to use callables introduces additional complexity which I'd rather not deal with this late in the release process.

[ddfisher]

LGTM!