What to do about inferring types from multiple scopes?

[JukkaL]

Typically mypy infers the type of the variable from the first assignment. In some cases mypy infers the type from two assignments. The first assignment creates internally a partial type, which is missing some information. The second assignment completes the type. Example:

x = []  # x gets a partial List type
x.append(1)  # x gets complete type List[int]

In some cases the assignments can be in two different scopes. Example:

x = []

def f() -> None:
    x.append(1)

# type of x is List[int]

This feature has apparently been supported a while, but it has some issues:

• This behavior is somewhat messy to support in fine-grained incremental mode (Refresh of partial type produces the wrong type in fine-grained incremental mode #4464).
• It can make it hard to figure out the inferred type of a variable, since the two assignments can be separated from each other by hundreds of lines and many functions.
• Moving functions around can affect the type of a module/class variable in cases where there are two potential functions that can complete a partial type.
• If the second assignment is missing or removed, and the first assignment initializes the variable with None, the inferred type will become None, which is usually not what the user wanted.
• It seems a bit fragile with some issues that took a long time to notice (Inconsistent types for variable with partial type #4484, Inference from unannotated functions #4455, Invalid type inferred for attribute initialized to None in class body #4416).

There are a few alternatives which might be reasonable, instead of the current behavior:

1. Require an annotation for the variable in cases such as the above, where the assignment that completes a partial type is in a different scope. This would break compatibility with previous mypy releases. Based on a quick experiment, the S internal codebase at Dropbox would require about 40 additional type annotations if we make this change.
2. Ignore assignments in nested scopes and assume Any for any missing partial type information. For example, if the initial assignment is x = [] and the only additional assignments are in nested scopes, infer List[Any] as the type of x. This way the type of x is defined entirely by the contents of a single scope, which solves the above problems. This also introduces an additional problem -- implicit Any types. We might want to have a strictness flag that falls back to requiring a type annotation (as in option 1 above).

Currently my biggest concern is fine-grained incremental mode. There it may be possible to support the current behavior by introducing a new concept: "bound targets". I'll add a writeup about this to #4464.

@gvanrossum @msullivan @ilevkivskyi Thoughts?

[ilevkivskyi]

I am in favor of option 1. I think I mentioned this some time ago, and I didn't change my mind. My argument is simple: I don't like implicit Any types. Currently PEP 484 defines small set of situations where Any is assumed (like non-annotated functions and missing generic arguments) and we have flags to prohibit those. Adding one more source of implicit Anys seems not worth it (especially that users might not be aware of it and will therefore get false negatives).

Also the number 40 seems relatively small.

[ilevkivskyi]

An additional argument for option 1 is that the function where the second assignment happens may be non-annotated. In this case the inferred type will be e.g. Optional[Any]. This can lead to some hard to spot false negatives. This has been also discussed in #4455

[gvanrossum]

I think option 1 is the most reasonable. Partial types are already something of a stretch (there are lots of situations where they don't trigger properly) and the cross-scope behavior is just too fragile to care much about. While adding 40 new annotations may be a bit painful (I've had to do similar things to that codebase for other mypy changes, and sometimes the hard thing is figuring out what type to put in), it seems to be the most in line with common sense.

[JukkaL]

Ok, let's go with option 1.

[JukkaL]

Turns out that my figure of about 40 annotations was too optimistic -- I wasn't counting all kinds of partial types. A better figure is about 80 annotations for the S codebase. I think that the adaptation work is still reasonable and option 1 is still the best one.

[JukkaL]

I found some further cases. It looks like the C codebase needs around 40 annotations and S around 140. The most common case in S is an attribute initialized to None in the class body.

I'm about half way through the migration of Dropbox codebases. It has been enough work that it might make sense to make it easier. First, we could have a per-file option that ignores missing annotations (and will just infer Any components for missing collection item types, and so on). Second, we could try to infer a type and suggest it in some of the most common cases. Example:

foo.py:234: error: Need type annotation for 'things' (suggestion: "List[str]")

[gvanrossum]

Off-line we decided on a compromise solution. IIRC it's something like add a global flag to enable or disable the current behavior, and force disabling the behavior in fine-grained mode; and when the current behavior is disabled, show an error with a suggestion for the type. E.g.

class C:
    x = None
    def __init__(self) -> None:
        self.x = 0

This would either be allowed (default in non-fine-grained mode) or produce an error on x = None with the suggested type int. The suggested types would be formatted the same as types in messages from reveal_type(), and the phrasing would make it clear that the suggestion is just that, not a guarantee.

[JukkaL]

Closing in favor of #4703 which is about suggesting type annotations. Requiring an explicit annotation was implemented in #4553.