Add assertion passes

[elazarg]

(Stealing text from #2150)

It would be nice to have "assertion passes" between the different passes, to verify and document the post-conditions of each pass and pre-conditions of the next, without adding more clutter to the code. Examples (due to Jukka):

We could, for example, verify that unbound types don't leak from semantic analysis, and that every name expression refers to something after semantic analysis.

In addition, currently breaking something in e.g. the semantic analysis pass, is likely to result in a seemingly unrelated error message in latter phases. Of course it cannot be completely avoided, but it can be reduced.

I don't have any concrete design in mind. My intention is that the scope, nature, design and other details are to be fleshed out in this thread. The idea is very half-baked, so I'm looking forward for your comments.

[elazarg]

Some random notes and thoughts:

• I think it is important to make sure that no other part on the program depends on the pass in any way, now or in the future.
• There should be some differentiation between assertions over inner state of an object and assertions over the whole state (as in Jukka's examples above).
• (Jukka): The assertion passes could be enabled in test cases and maybe skipped otherwise to avoid performance overhead.
• There's no need to overlap typechecking, but if there are holes in the typing of the previous pass then this is the place to test for type errors. Including expensive detection of nested subtyping.
• It almost goes without saying, but the assertions should be about what is desired state of affairs (e.g. assert is_iterable(x)), not about what happens to be the state in current implementation (e.g. assert isinstance(x, list)).
• An assertion pass is not a unit test for the earlier pass, and should avoid using code from it. It should test that the current state is well formed.
• Again goes without saying, no mutation of state or any permanent side effect.

[JukkaL]

Sounds reasonable.

I'd recommend starting with a simple TraverserVisitor that goes through the AST and checks for just one or two simple things, such as the ones I mentioned earlier (unbound types and name expressions referring to something). Maybe also check if full names are populated. This is may well result in test failures -- if so, initially this could always be off by default, until we can get a clean run. Then we can iterate on it and add additional checks one by one.

[Michael0x2a]

Just a few extra thoughts:

• It might be nice to have the assert checks be optionally enabled or disabled via a flag, rather then just having them be on only for tests. The tests are pretty good, but not comprehensive, and many of the stranger bugs tend to manifest only in large codebases with lots of weirdness.
• Perhaps this could tie in with @ddfisher's work on making mypy pass strict optional checks? I vaguely remember him talking about how we assume different fields may be null/may not be null at different points in the lifecycle and how that made making mypy pass strict optional challenging. Perhaps it might be useful to formally verify those assumptions as a part of the checks? I don't really know if that'll directly help mypy pass strict optional, but it might make analyzing the code progressively easier.

[JukkaL]

Agreed, it would be super useful to be able to run mypy against a codebase with the checks on.

[elazarg]

I think I've found a type error running a very naive pass, doing nothing more than str(self.tree):

  File "../mypy/mypy/strconv.py", line 437, in visit_newtype_expr
    return 'NewTypeExpr:{}({}, {})'.format(o.line, o.fullname(), self.dump([o.value], o))
AttributeError: 'NewTypeExpr' object has no attribute 'fullname'

(Please correct me if I'm missing anything here)

[gvanrossum]

Good catch! I guess there's no unit test for this one. :-(

[elazarg]

And no type annotations. PR: #2162