Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generated classes does not work with forward references #1701

Closed
MeGotsThis opened this issue Jun 11, 2016 · 5 comments · Fixed by #3952
Closed

Generated classes does not work with forward references #1701

MeGotsThis opened this issue Jun 11, 2016 · 5 comments · Fixed by #3952

Comments

@MeGotsThis
Copy link

It seems mypy has trouble with generated classes such as NamedTuple.

a.py:

import b
from typing import Any, NamedTuple

class A:
    def a(self, b: 'b.B') -> str:
        return 'a'

ATuple = NamedTuple('ATuple', [('a', Any)])

b.py:

import a

class B:
    def b(self, a: 'a.A') -> str:
        return 'b'

    def aWithTuple(self, atuple: 'a.ATuple') -> str:
        return 'a'

mypy a.py output

a.py:1: note: In module imported here:
b.py: note: In function "aWithTuple":
b.py:7: error: Invalid type "a.ATuple"

mypy b.py works just fine

@gvanrossum
Copy link
Member

gvanrossum commented Jun 12, 2016 via email

@gvanrossum
Copy link
Member

The NamedTuple also plays a role -- if I change a.py to define

class ATuple:
    a = None  # type: Any

then the error disappears.

The problem may not be unique to NamedTuple but it is caused by the late processing of the definition of ATuple when using NamedTuple; with a regular class definition the first pass of the semantic analysis (viz., semanal.FirstPass.visit_class_def()) creates the definition of ATuple, but when ATuple is the result of a NamedTuple() expression is created during the "regular" semantic analysis (viz., semanal.SemanticAnalyzer.process_namedtuple_definition(), called from visit_assignment_stmt() in the same class).

None of the improvements to import cycle processing currently being considered help (neither #2167 nor #2264). I'm guessing some form of deferred processing in the semantic analyzer might have to be devised in order to handle this properly, or we need to move part of the NamedTuple analysis to the FirstPass. (I believe we can't move all of it, since it may contain references to other types; but we may be able to construct a placeholder in the FirstPass that gets populated in the second pass, similar to the way regular class definitions are processed.)

@gvanrossum gvanrossum added this to the 0.5 milestone Oct 20, 2016
@gvanrossum
Copy link
Member

This feels important but fairly tricky.

@JukkaL
Copy link
Collaborator

JukkaL commented Oct 21, 2016

Here are some additional thoughts. I think that this problem probably affects named tuples, type aliases, typed dicts and type variable definitions (e.g. bounds). I've hit this problem myself and I agree that this looks important.

1) Splitting second semantic analysis pass

Another option is to split the main semantic analysis pass into two passes, the first of which would resolve named tuples and type aliases, and the second one would resolve type annotations. However, this could slow things down as both passes would have to keep track of defined names, etc.

2) Deferred processing of AST nodes

Deferred processing, similar to what we have in the type checker, would likely work, but I think that we'd need to take a copy of almost all of the semantic analyzer state to do full deferred processing, and we'd have to ensure that running semantic analysis twice on the same nodes doesn't cause trouble. I'd rather avoid this if possible.

3) Deferred processing of types only

A less general deferred processing seems the most promising approach to me. We'd create a special kind of partially defined type when we encounter a type reference where the target hasn't been processed yet. This type would record the fully qualified name of the target type. In the third pass we'd patch up these partially defined types (or generate errors if they still don't refer to types). The slightly tricky bit with this approach is that we'd need to translate all types that might have these partially defined types as components, but we can skip this step when there's nothing to do. For example, we could collect all deferred types in a list, and if the list is empty for a set of modules, we'd skip the patching-up step.

@mikeyhew
Copy link

Here is another example that doesn't work with NamedTuple, but does without it:

from typing import List, NamedTuple

class GraphNode:
    def __init__(self, name: str) -> None:
        self.name = name
        self.edges: List['GraphEdge'] = []

class GraphEdge(NamedTuple):
    dest: GraphNode
    cost: int

Error message from MyPy:

mypy --fast-parser --python 3.6 foo.py
foo.py:6: error: Incompatible types in assignment (expression has type List[None], variable has type List[GraphEdge])

Also note that If I re-order the definitions so GraphEdge comes first, the error is not raised.

@gvanrossum gvanrossum removed this from the 0.5 milestone Mar 29, 2017
JukkaL pushed a commit that referenced this issue Sep 27, 2017
Forward references didn't work with anything apart from classes, for example 
this didn't work:

```
x: A
A = NamedTuple('A', [('x', int)])
```

The same situation was with `TypedDict`, `NewType`, and type aliases. The 
root problem is that these synthetic types are neither detected in first pass, 
nor fixed in third pass. In certain cases this can lead to crashes (first six issues 
below are various crash scenarios). This fixes these crashes by applying some 
additional patches after third pass.

Here is the summary of the PR:

* New simple wrapper type `ForwardRef` with only one field `link` is introduced 
  (with updates to type visitors)
* When an unknown type is found in second pass, the corresponding 
  `UnboundType` is wrapped in `ForwardRef`, it is given a "second chance" in 
  third pass.
* After third pass I record the "suspicious" nodes, where forward references and 
  synthetic types have been encountered and append patches (callbacks) to fix 
  them after third pass. Patches use the new visitor `TypeReplacer` (which is the 
  core of this PR).

Fixes #3340
Fixes #3419
Fixes #3674
Fixes #3685
Fixes #3799
Fixes #3836
Fixes #3881
Fixes #867
Fixes #2241
Fixes #2399
Fixes #1701
Fixes #3016
Fixes #3054
Fixes #2762
Fixes #3575
Fixes #3990
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants