Python 3.12.0a7 can fail to catch exceptions when an iterator is involved

[mattclay]

Bug report

Python 3.12.0a7 (at least as of commit d65ed69) can fail to catch exceptions when an iterator is involved. The following code works on Python 3.11 but the ValueError is uncaught on Python 3.12:

#!/usr/bin/env python

def do_work():
    yield
    raise ValueError()


def main():
    try:
        for _ in do_work():
            if True is False:
                return
    except ValueError:
        pass


if __name__ == '__main__':
    main()

With this code in repro.py the following traceback is shown on Python 3.12:

Traceback (most recent call last):
  File "//repro.py", line 18, in <module>
    main()
  File "//repro.py", line 12, in main
    return
  File "//repro.py", line 5, in do_work
    raise ValueError()
ValueError

No error or output occurs under Python 3.11.

Your environment

Tested on Ubuntu 20.04.5 using the deadsnakes nightly build: python3.12 - 3.12.0~a7-98-gd65ed693a8-1+focal1

Linked PRs

• GH-103488: Use return-offset, not yield-offset. #103502

[terryjreedy]

Verified on latest 3.11/2 releases on Windows.
Edit revised: same if yield something but error in yield expression or before yield is caught. IE, Error must be after yield statement.

[pochmann]

Line 12 being in the traceback instead of line 10 also seems wrong.

[terryjreedy]

If return is changed to pass or continue, the exception is caught when the loop asks for the next value and gets an exception when executing the code after yield. I am slightly surprised that breaking (also not caught) or returning does not just gc the generator without executing the remaining (buggy) code. (Revised twice after more experiments.)

[terryjreedy]

@mattclay Inserting 1/0 or better, an unbound name such as z is an easier way to generate an exception and easier to move or delete. 'NameError' should be the shortest exception name.

[gvanrossum]

Can you post the reduced reproducer? And maybe the code object disassembly (python -m dis <file>)? It smells like a failure to generate or interpret the exception table correctly. Disassembly on the original reproducer shows odd gaps in the exception table that aren't there in 3.11.

[terryjreedy]

@gvanrossum Who is 'who'? I reduced the raise and catch a bit but also added 3 debug prints to trace execution.

[mattclay]

I simplified the reproducer a bit using the suggestion from @terryjreedy:

#!/usr/bin/env python

def do_work():
    yield
    x


def main():
    try:
        for _ in do_work():
            if True is False:
                return
    except NameError:
        pass


if __name__ == '__main__':
    main()

I've also attached the code object disassembly from Python 3.11 and 3.12 for the revised reproducer.

dis311.txt
dis312.txt (3.12.0a6)
dis312a7.txt (3.12.0a7, commit d65ed69)

[carljm]

It seems like this symptom could also result from a change in the timing of finalization of the do_work() generator on the stack in main(). If the generator is not exhausted, then the code after the yield will only run when the generator object is finalized, IIRC.

[carljm]

The change in behavior bisects to 4a1c58d

cc @markshannon