Allow generics in runtime expressions

[ilevkivskyi]

As discussed in python/typing#303 generics in typing.py are now cached and are therefore much faster at runtime than before. Here I propose (and it looks like Guido is not against this) to allow generic types as runtime expressions.

@JukkaL Please take a look.

[codecov-io]

Current coverage is 83.13% (diff: 100%)

Merging #2302 into master will increase coverage by 0.04%

@@             master      #2302   diff @@
==========================================
  Files            72         72          
  Lines         19032      19036     +4   
  Methods           0          0          
  Messages          0          0          
  Branches       3917       3919     +2   
==========================================
+ Hits          15814      15826    +12   
+ Misses         2618       2611     -7   
+ Partials        600        599     -1   

Powered by Codecov. Last update 3ff04c4...1cbd4c5

[gvanrossum]

Looks great, no need for Jukka to review.

[gvanrossum]

I just got a crash when running this on some code of ours. I'm still trying to distill the code that triggers it into something small, but here's the traceback:

ui/common/xui/wizkit.py:452: error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/guido/src/mypy/mypy/__main__.py", line 5, in <module>
    main(None)
  File "/Users/guido/src/mypy/mypy/main.py", line 38, in main
    res = type_check_only(sources, bin_dir, options)
  File "/Users/guido/src/mypy/mypy/main.py", line 79, in type_check_only
    options=options)
  File "/Users/guido/src/mypy/mypy/build.py", line 184, in build
    dispatch(sources, manager)
  File "/Users/guido/src/mypy/mypy/build.py", line 1494, in dispatch
    process_graph(graph, manager)
  File "/Users/guido/src/mypy/mypy/build.py", line 1674, in process_graph
    process_stale_scc(graph, scc)
  File "/Users/guido/src/mypy/mypy/build.py", line 1753, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/Users/guido/src/mypy/mypy/build.py", line 1422, in type_check_first_pass
    self.type_checker.check_first_pass()
  File "/Users/guido/src/mypy/mypy/checker.py", line 175, in check_first_pass
    self.accept(d)
  File "/Users/guido/src/mypy/mypy/checker.py", line 234, in accept
    typ = node.accept(self)
  File "/Users/guido/src/mypy/mypy/nodes.py", line 713, in accept
    return visitor.visit_class_def(self)
  File "/Users/guido/src/mypy/mypy/checker.py", line 968, in visit_class_def
    self.accept(defn.defs)
  File "/Users/guido/src/mypy/mypy/checker.py", line 234, in accept
    typ = node.accept(self)
  File "/Users/guido/src/mypy/mypy/nodes.py", line 774, in accept
    return visitor.visit_block(self)
  File "/Users/guido/src/mypy/mypy/checker.py", line 1059, in visit_block
    self.accept(s)
  File "/Users/guido/src/mypy/mypy/checker.py", line 234, in accept
    typ = node.accept(self)
  File "/Users/guido/src/mypy/mypy/nodes.py", line 532, in accept
    return visitor.visit_func_def(self)
  File "/Users/guido/src/mypy/mypy/checker.py", line 445, in visit_func_def
    self.check_func_item(defn, name=defn.name())
  File "/Users/guido/src/mypy/mypy/checker.py", line 508, in check_func_item
    self.check_func_def(defn, typ, name)
  File "/Users/guido/src/mypy/mypy/checker.py", line 628, in check_func_def
    self.accept(item.body)
  File "/Users/guido/src/mypy/mypy/checker.py", line 234, in accept
    typ = node.accept(self)
  File "/Users/guido/src/mypy/mypy/nodes.py", line 774, in accept
    return visitor.visit_block(self)
  File "/Users/guido/src/mypy/mypy/checker.py", line 1059, in visit_block
    self.accept(s)
  File "/Users/guido/src/mypy/mypy/checker.py", line 234, in accept
    typ = node.accept(self)
  File "/Users/guido/src/mypy/mypy/nodes.py", line 875, in accept
    return visitor.visit_return_stmt(self)
  File "/Users/guido/src/mypy/mypy/checker.py", line 1496, in visit_return_stmt
    self.check_return_stmt(s)
  File "/Users/guido/src/mypy/mypy/checker.py", line 1510, in check_return_stmt
    typ = self.accept(s.expr, return_type)
  File "/Users/guido/src/mypy/mypy/checker.py", line 234, in accept
    typ = node.accept(self)
  File "/Users/guido/src/mypy/mypy/nodes.py", line 1250, in accept
    return visitor.visit_call_expr(self)
  File "/Users/guido/src/mypy/mypy/checker.py", line 1940, in visit_call_expr
    return self.expr_checker.visit_call_expr(e)
  File "/Users/guido/src/mypy/mypy/checkexpr.py", line 178, in visit_call_expr
    return self.check_call_expr_with_callee_type(callee_type, e)
  File "/Users/guido/src/mypy/mypy/checkexpr.py", line 238, in check_call_expr_with_callee_type
    e.arg_names, callable_node=e.callee)[0]
  File "/Users/guido/src/mypy/mypy/checkexpr.py", line 313, in check_call
    self.infer_arg_types_in_context(None, args)
  File "/Users/guido/src/mypy/mypy/checkexpr.py", line 395, in infer_arg_types_in_context
    arg_type = self.accept(arg, ctx)
  File "/Users/guido/src/mypy/mypy/checkexpr.py", line 1755, in accept
    return self.chk.accept(node, context)
  File "/Users/guido/src/mypy/mypy/checker.py", line 234, in accept
    typ = node.accept(self)
  File "/Users/guido/src/mypy/mypy/nodes.py", line 1297, in accept
    return visitor.visit_index_expr(self)
  File "/Users/guido/src/mypy/mypy/checker.py", line 2067, in visit_index_expr
    return self.expr_checker.visit_index_expr(e)
  File "/Users/guido/src/mypy/mypy/checkexpr.py", line 1271, in visit_index_expr
    result = self.visit_index_expr_helper(e)
  File "/Users/guido/src/mypy/mypy/checkexpr.py", line 1277, in visit_index_expr_helper
    return self.accept(e.analyzed)
  File "/Users/guido/src/mypy/mypy/checkexpr.py", line 1755, in accept
    return self.chk.accept(node, context)
  File "/Users/guido/src/mypy/mypy/checker.py", line 234, in accept
    typ = node.accept(self)
  File "/Users/guido/src/mypy/mypy/nodes.py", line 1663, in accept
    return visitor.visit_type_application(self)
  File "/Users/guido/src/mypy/mypy/checker.py", line 2079, in visit_type_application
    return self.expr_checker.visit_type_application(e)
  File "/Users/guido/src/mypy/mypy/checkexpr.py", line 1377, in visit_type_application
    return self.apply_generic_arguments(tp, tapp.types, tapp)
  File "/Users/guido/src/mypy/mypy/checkexpr.py", line 885, in apply_generic_arguments
    return applytype.apply_generic_arguments(callable, types, self.msg, context)
  File "/Users/guido/src/mypy/mypy/applytype.py", line 20, in apply_generic_arguments
    tvars = callable.variables
AttributeError: 'AnyType' object has no attribute 'variables'
ui/common/xui/wizkit.py:452: note: use --pdb to drop into pdb

The line triggering it has:

        # This magical WizkitStrings metaclasss confuses mypy
        return self.account.render_string(WizkitStrings[string], # type: ignore
                flip_role=flip_role, max_len=max_len, **kw)

The class WizkitStrings inherits from another class which has a metaclass defining __getitem__. The metaclass is added using @add_metaclass(...). There may also be an import cycle or a suppressed import (-s) involved.

[ilevkivskyi]

@gvanrossum
I think I understand what is going on. The class is not a "real" generic, but WizkitString[string] is parsed as TypeApplication. After #type: ignore it becomes Any. It looks like an easy fix would be to add check if isinstance(tp, AnyType): return AnyType() in visit_type_application. Will try this soon. Should I open a separate PR for this?

[gvanrossum]

I figured it was something like that. If you can come up with a test and a fix for that test, a separate PR would be great.

[ilevkivskyi]

@gvanrossum Sorry, it is more complex than I thought. I still could not reproduce this crash.
But I believe that #2309 should help (in any case I think it is safer to check for AnyType() or something invalid just to avoid crash and continue checking with AnyType() inferred).
Could you please try that PR with your code?

I will continue to look for a good test for this.