sys.monitoring `PY_UNWIND` not being delivered when a generator is closed/garbage collected

[jaltmayerpizzorno]

Bug report

Bug description:

With Python 3.12, this code generates a PY_UNWIND event; with 3.13 and later, it no longer does.

import sys

def gen():
    yield 1
    yield 2

def callback(code, off, exc):
    if code is gen.__code__:
        print("got unwind!")

ID=3
sysmon = sys.monitoring
sysmon.use_tool_id(ID, "test")
sysmon.register_callback(ID, sysmon.events.PY_UNWIND, callback)
sysmon.set_events(ID, sysmon.events.PY_UNWIND)

g = gen()
next(g)
del g  # or g.close()

CPython versions tested on:

3.13, 3.14

Operating systems tested on:

macOS, Linux

Linked PRs

• gh-140373: Correctly emit PY_UNWIND event when generator is closed #140767
• [3.14] gh-140373: Correctly emit PY_UNWIND event when generator is closed (GH-140767) #140816
• [3.13] gh-140373: Correctly emit PY_UNWIND event when generator is closed (GH-140767) #140821

[efimov-mikhail]

Thanks for the report, confirmed on 3.13, 3.14 and main.

[Zheaoli]

Seems related with https://github.com/python/cpython/pull/111069/files

[Zheaoli]

https://github.com/python/cpython/blob/main/Objects/genobject.c#L413-L420

Seems we add a short path to just return instead of raise an exception when there is only one level generator

[markshannon]

What happening is that in #111069 we skipped the expensive raise then catch if the generator is not in a try block, as there would (or so we thought) be no observable effect.
But, if there is no exception raised, there is no unwinding, and noPY_UNWIND event.

This is fine for finalization, as the language is quite clear that finalization may never happen,
but the docs explicitly state that an exception will be raised when generator.close() is called.

If we are to fix this for generator.close() we might as well do it for finalization for consistency.

My intended purpose of PY_UNWIND is that profilers and other tools can balance entries and exits to functions: for every PY_START or PY_RESUME there should be a PY_RETURN, PY_YIELD or PY_UNWIND, but it does seem that you ought to be able to use the event to detect generator close/finalization.

So, do we add the additional check for instrumentation in generator.close() or do we update the docs to say that the exception may not be raised if the generator is not currently executing in a try block?

@jaltmayerpizzorno how did you notice this, and is it breaking anything?

[jaltmayerpizzorno]

@jaltmayerpizzorno how did you notice this, and is it breaking anything?

Hi @markshannon! RightTyper uses PY_START and PY_UNWIND; the PY_UNWIND is useful for detecting (in the absence of PY_RETURN) when a function doesn't return a value, or when a stack frame is done, so that it doesn't possibly take it for another call that happened to reuse the frame.

To observe types, RightTyper executes the target program (using runpy). Once that execution ends, generators (but apparently not async generators or coroutines) may remain running. RightTyper used to close() them to have PY_UNWIND process normally. I've now changed that to a throw(GeneratorExit), which works around the issue for us... but I thought I'd raise the issue anyway (pun partly intended, sorry).

I think generally it's useful to be able to tell when a call is done (by getting either PY_RETURN and PY_UNWIND), but you'd know best what it costs to support it.

[markshannon]

It sounds like it is being relied on, and it is in the spec. So, we'd better fix it.
It's not a complicated fix and I expect the performance impact will not be measurable, and definitely not significant.

[efimov-mikhail]

I'd like to prepare a fix for this.

[jaltmayerpizzorno]

It sounds like it is being relied on, and it is in the spec. So, we'd better fix it.

In short: yes; thank you!