Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New semantic analyzer: Clean-up logic for classes in functions #6422

Open
ilevkivskyi opened this issue Feb 17, 2019 · 5 comments
Open

New semantic analyzer: Clean-up logic for classes in functions #6422

ilevkivskyi opened this issue Feb 17, 2019 · 5 comments
Labels
priority-1-normal refactoring Changing mypy's internals semantic-analyzer Problems that happen during semantic analysis

Comments

@ilevkivskyi
Copy link
Member

While working on #6421 I looked again at logic for storing symbols for local (function- or method-level) classes in prepare_class_def(). The logic is too ad-hoc and needs to be cleaned up. Here are some points that should be fixed:

  • defn.fullname and info.fullname() should always coincide
  • info.fullname() prefix should coincide with the symbol table where the info is actually stored (for serialization)
  • things should work for double nested classes
  • things should work in fine-grained mode

I think the logic should be as following. Consider this example:

class Out:
    def meth(self) -> None:
        class In:
            def meth(self) -> None:
                class C: pass

The class C should be stored in its local namespace under name 'C'. In addition it should be stored in the namespace of __main__.Out under name 'C@5'. info.fullname() and defn.fullname should be both '__main__.Out.C@5'.

@ilevkivskyi ilevkivskyi added refactoring Changing mypy's internals priority-1-normal semantic-analyzer Problems that happen during semantic analysis labels Feb 17, 2019
@ilevkivskyi ilevkivskyi self-assigned this Feb 19, 2019
ilevkivskyi added a commit that referenced this issue Mar 15, 2019
…rward (#6527)

Fixes #6412 
Fixes #6453 

In this PR:
* Wait until we can classify r.h.s. of an assignment as normal, type alias, named tuple etc.
* Choose the corresponding branch instead of performing all of them (as it used to be before)
* Never create (or rely on creation of) temporary `Var`s
* Add every symbol to symbol table only once (as it is done for classes)
* Don't "patch" existing symbols (with one exception for type variables, see below)
* Add and update docstrings and comments

Notes:
* I allow placeholders for type alias targets and NewTypes targets, they essentially behave as base classes. In general I make logic for type aliases and NewTypes much more similar to logic of class definitions (especially NewTypes), this allows to support a whole bunch of tricky forward references
* I don't store types with placeholders in annotations, and instead if an alias target got updated (e.g. placeholders are resolved), I set `progress=True`

Follow-ups:
* There is still a small hack to keep compatibility with old analyzer in type checker, but now it is basically "don't use binder for l.h.s. of special forms"
* Type variable declarations still have a bug that duplicate type variable definition is not reported as error, second one silently wins, this will be fixed in a separate PR
* This PR doesn't aim to fix aspects of assignment analysis related to #6422, that will be also a separate PR

Test updates:
* Add and enable more tests for type aliases and NewTypes
* Add test for cyclic definition error messages
* Change order of error messages on the same line in few tests
* Enable one test file where previously only some tests were opted in (I think it was not enabled by mistake in one of the previous PRs, I enable it here since order of errors in one test in that file is affected by this PR.)
@ilevkivskyi ilevkivskyi removed their assignment Dec 9, 2019
@MatthieuDartiailh
Copy link

What is the current status of this ? I saw that a couple of PRs addressed part of the issue but #7281 is not fixed and I am wondering how much of the initial comment still applies after three PR related to this issue. I added typing to a project but this issue will prevent downstream package to benefit of it so I may be interested in helping.

@MatthieuDartiailh
Copy link

Also how should the system behaves for a class defined in a function ? Should the symbol be stored on the function or at the global level ?

@JukkaL
Copy link
Collaborator

JukkaL commented Jun 8, 2020

I'm not sure if anybody is up to date on what still needs to be done (if anything). To get started with this, you could try to document the current behavior and then we can discuss how it might need to be modified.

@MatthieuDartiailh
Copy link

Thanks @JukkaL ! Any pointer on the best way to inspect mypy internal after parsing a file, I am new to mypy internals ?

@MatthieuDartiailh
Copy link

MatthieuDartiailh commented Jun 10, 2020

I inspected the nodes stored in the result object returned by mypy.build.build run on the following example(using mypy at 05571b1, point at which I cloned the repo):

from collections import namedtuple

class A:
    pass


def make_class() -> type:
    class Inner:
        pass

    return Inner


B = make_class()


class Out:
    def meth(self) -> type:
        class In:
            def meth(self) -> type:
                class C(namedtuple("_C", ("a")), A): pass
                return C

        return In

D = Out().meth()

E = Out().meth()().meth()

I see that the following is true:

  • defn.name is always the class short name
  • defn.fullname is always the fully qualified name including the line number for nested classes
  • those matches what is found on the node

I am not sure where to look for the info and to figure out where things are stored, since there was a concern of global storage vs storage on the surrounding class. Any pointer appreciated.

patrick91 added a commit to mcsheehan/strawberry that referenced this issue Nov 23, 2021
Apparently mypy has some issues when using classes inside functions,
as we do in our tests, issue here:
python/mypy#6422
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority-1-normal refactoring Changing mypy's internals semantic-analyzer Problems that happen during semantic analysis
Projects
None yet
Development

No branches or pull requests

3 participants