Eval/exec and comprehension scopes unclear in documentation

[brunoloff]

BPO	43605
Nosy	@terryjreedy, @congma
PRs	bpo-43605: Improve the documentation to exec() and eval() #25039

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-03-23.17:26:04.893>
labels = ['type-bug', '3.10', 'docs']
title = 'Eval/exec and comprehension scopes unclear in documentation'
updated_at = <Date 2021-03-27.15:06:53.886>
user = 'https://bugs.python.org/brunoloff'

bugs.python.org fields:

activity = <Date 2021-03-27.15:06:53.886>
actor = 'congma'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2021-03-23.17:26:04.893>
creator = 'bruno.loff'
dependencies = []
files = []
hgrepos = []
issue_num = 43605
keywords = ['patch']
message_count = 9.0
messages = ['389397', '389469', '389565', '389569', '389592', '389593', '389595', '389597', '389599']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'docs@python', 'congma', 'bruno.loff']
pr_nums = ['25039']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43605'
versions = ['Python 3.10']

Linked PRs

• gh-87771: Improve documentation of eval() #100003

[brunoloff]

Python 3.9.2 seems to be giving me some unexpected difficulty evaluating generators inside evals. Here is the example:

def func(l):
    def get(i):
        return l[i]


    print(sum(get(i) for i in range(len(l)))) # works as expected, prints 10
    print(eval("get(0) + get(1) + get(2) + get(3)")) # works just fine, prints 10

    # if __globals is set to locals(), it still works, prints 10
    print(eval("sum(get(i) for i in range(len(l)))", locals()))

    # This will complain
    print(eval("sum(get(i) for i in range(len(l)))"))

func([1,2,3,4])

The last line gives the following error

Traceback (most recent call last):
  File "/something/test_eval.py", line 28, in <module>
    func([1,2,3,4])
  File "/something/test_eval.py", line 10, in func
    print(eval("sum(get(i) for i in range(len(l)))"))  # this does not work... bug?
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <genexpr>

NameError: name 'get' is not defined

Any kind of generator-based code wont work. The following lines would give the same an error:

print(eval("sum(get(i) for i in range(len(l)))"), globals(), locals())
print(eval("[get(i) for i in range(len(l))]"))
print(eval("{i:get(i) for i in range(len(l))}"))

Any clue what is happening? The documentation on eval seems to give no insight on why this behavior is as is. This really feels like an issue, at the very least, it's an issue in the documentation.

[congma]

I think this is in the same class of behaviours as

def func(l):
    def get(i):
        return l[i]
    print(eval("(lambda x: get(x))(0)"))  # Call anonymous lambda with the constant 0 as argument

Calls like func(["spam"]) will not "work", and NameError is raised.

In this case, inside the lambda's body the name "get" can't be resolved. For the lambda body, the name "get" is a nonlocal but there's no way to access a nonlocal in a lambda.

The comprehensions, like lambdas, are in their own nested scope.

[terryjreedy]

This is not an execution bug.

https://docs.python.org/3/reference/expressions.html#displays-for-lists-sets-and-dictionaries

"However, aside from the iterable expression in the leftmost for clause, the comprehension is executed in a separate implicitly nested scope. This ensures that names assigned to in the target list don’t “leak” into the enclosing scope."

So when the passed in locals is not the passed in globals, that means a separate local scope.

https://docs.python.org/3/reference/expressions.html#generator-expressions is a little less clear.

"Variables used in the generator expression are evaluated lazily when the __next__() method is called for the generator object (in the same fashion as normal generators). However, the iterable expression in the leftmost for clause is immediately evaluated, so that an error produced by it will be emitted at the point where the generator expression is defined, rather than at the point where the first value is retrieved. Subsequent for clauses and any filter condition in the leftmost for clause cannot be evaluated in the enclosing scope as they may depend on the values obtained from the leftmost iterable."

By implication, the value expressions are also not evaluated in the enclosing local scope.

I am thinking about adding something to the eval/exec doc, but this issue overlaps with another one about their doc.

https://docs.python.org/3/reference/expressions.html#generator-expressions

[brunoloff]

Hmm... OK, if I understand correctly, the evaluation procedure for a (e.g.) list comprehension, as described in the documentation you linked in, is as follows:

• The generator in the leftmost for expression is evaluated in the current local scope. (doc: "The iterable expression in the leftmost for clause is evaluated directly in the enclosing scope and then passed as an argument to the implicitly nested scope.")
• A new nested scope is created, and the iterator object is passed to that new scope as an argument.
• The other for and if clauses are evaluated in the new scope.

As you mention, the documentation does not properly describe in which scope is evaluated the leftmost expression that generates each object (to be placed in the list).

You say that the leftmost expression is also not evaluated in the enclosing scope (which I take means that it is being evaluated in the new nested scope). But notice that sometimes it *seems to be*.

There are two inconsistent behaviors:

1. If evaluated outside of an eval, using local objects works just fine. It is then reasonable to assume that the leftmost expression has access to the locals() of the function it is in. (see the line with the comment # works as expected in the first post I made).

2. If evaluated inside an eval the leftmost expression cannot access the locals (see the example I gave). This happens even when locals() is passed to the __locals argument of the call to eval.

It seems reasonable that behavior (1) should happen in both cases. In fact, I think it is *very* reasonable to expect that calling eval() on a string should have the exact same effect as if the code that is inside the eval had been written as part of the source code. Don't you think? I believe that this is exactly what happens, e.g., in Lisp.

I would guess this is just a bug, but maybe there is some technical reason why this is not possible. If that is (sadly) the case, this should be explained in the documentation.

Currently I don't really have a mental model for what happens when I call eval.

[congma]

I think it is *very* reasonable to expect that calling eval() on a string should have the exact same effect as if the code that is inside the eval had been written as part of the source code.

I don't think Python's execution model is defined this way. The documentation on execution model says:

The eval() and exec() functions do not have access to the full environment for resolving names. Names may be resolved in the local and global namespaces of the caller. Free variables are not resolved in the nearest enclosing namespace, but in the global namespace.
footnote: This limitation occurs because the code that is executed by these operations is not available at the time the module is compiled.

https://docs.python.org/3/reference/executionmodel.html#interaction-with-dynamic-features

[congma]

I'm preparing an update to the documentation of eval/exec. There are several issues, but chiefly I'll link to the appropriate sections in the Language Reference, and clean up some other inaccuracies. When it's ready I'll submit a PR for core devs to review.

[brunoloff]

Hmm yes, some more words in the documentation might help.

Does anyone understand why it happens, though?

Specifically, note that

sum(get(i) for i in range(len(l)))

or

eval("get(0) + get(1) + get(2) + get(3)")

or

eval("sum(get(i) for i in range(len(l)))", locals())

work just fine, but

eval("sum(get(i) for i in range(len(l)))")

fails, which is really confusing. I have no mental model of what is happening that allows for the first thing to work, but disallows for the second thing. I understand it has something to do with the creation of a new subscope when the comprehension is run, but other than that, I don't really understand.

Also, ideally, the last thing should work, too.

I am teaching programming to college students and I could not explain to them why the first three worked but the last one failed. There was simply nothing I could say that would give them a good mental model for the execution.

[congma]

sum(get(i) for i in range(len(l)))

This expression inside the body of func() references the name "get" and "l" (ell), both are local to the scope introduced by func(). More specifically, these two names are referenced in the unnamed inner scope introduced by the generator-expression (get(i) for i in range(len(l))). It's as if you've passed into that inner scope the locals already introduced in func() by argument passing, e.g.

def func(...):
    get = ...
    ell = ...
    def genexpr(a, b):
        return <expression using a and b>
    sum(genexpr(get, ell))

eval("get(0) + get(1) + get(2) + get(3)")

The expression in the string doesn't introduce its own scope. The name "get" is resolved because without additional arguments, eval() gets the locals from the calling scope's locals, which is where the name "get" came.

eval("sum(get(i) for i in range(len(l)))", locals())

This tells eval() to use the calling scope's locals (the value returned by the call locals()) as the globals for the evaluation of the expression in the string. When eval() executes the compiled code, it's as if that piece of code lives in an environment where names like "gets" and "l" (ell) are top-level. Therefore these names are resolved.

eval("sum(get(i) for i in range(len(l)))")

Without explicitly telling eval() which globals/locals namespaces to use, eval() uses the current calling scope's. This is as if it were called like

eval("sum(get(i) for i in range(len(l)))", globals(), locals())

A problem arises. The generator expression in the string introduces an anonymous inner scope (let's call that scope "the box"). Inside the box, the name "i" is a local, there's no problem. But for the name "get", it's not local to the box, and it's not a global. Unlike other kinds of enclosed scope (for example, one introduced by an inner def block), "the box" has no way to look up names in enclosing scopes. This is the limitation referred to by the Language Reference's section on dynamic execution.

These are my attempts to explain why something works while others don't, based on my own understanding. I hope this helps somewhat, and if I made a mistake anywhere please correct them.

[congma]

Some more context: bpo-37646. The demo in that one was "eval inside list-comprehension-scope", while this one is the other way around.

Perhaps another example may better illustrate the interplay between eval and the execution environment:

def f():
    x = 1
    def g():
        return eval("x")
    return g
enc = f()
enc()

We get NameError: name 'x' is not defined.

The reason is that, during compilation the compiler doesn't and cannot care about what the string "x" means as an argument to eval(). To the compiler it's just a string constant passed to a function, and it's not much different from

        return print("x")

The compiler decides that the enclosed function g() has no locals in its block. And since there's no global with the name x either, when the dynamic expression is evaluated in eval() in that environment, the name doesn't resolve, because "eval() doesn't have access to the enclosing scope".

But the following is different:

def f():
    x = 1
    def g():
        x  # <----- 
        return eval("x")
    return g
enc = f()
enc()  # return value: 1

The marked line introduces name x as a local by virtue of merely having it in an expression-statement. Inside the function block of g(), we can imagine that the name resolution "goes up one level" into the enclosing block of f() where it is bound to the int object. When eval() is called there, the name does resolve.

I'm trying to think up a mental model but I'm afraid I can't find a simple one, except "once compiled, it's compiled, and eval() must learn to work with the already-compiled code". A much more in-depth description of name binding and execution in CPython is given here:

https://tenthousandmeters.com/blog/python-behind-the-scenes-5-how-variables-are-implemented-in-cpython/

especially in the section "LOAD_DEREF and STORE_DEREF".

[ncoghlan]

Python 3.13 adds the following note to the exec and eval documentation (due to #119235 and its follow-ups):

.. note::
      When ``exec`` gets two separate objects as *globals* and *locals*, the
      code will be executed as if it were embedded in a class definition. This
      means functions and classes defined in the executed code will not be able
      to access variables assigned at the top level (as the "top level"
      variables are treated as class variables in a class definition).

Comprehensions execute as if they're a nested function (with only the outermost iterable expression being evaluated in the calling scope rather than the nested scope), so this note applies to them as well.