`STOP_ITERATION` monitoring event is not triggered by unspecialized bytecode

[swtaarrs]

Bug report

Bug description:

Repro: Check out this commit, build, and run python -mtest test_monitoring -mtest_implicit_stop_iteration.

Context: The unspecialized implementation of FOR_ITER has some built-in opcode fusion, in that it skips the following END_FOR and POP_TOP when tp_iternext returns NULL. When the iterator is a generator, as it is in ExceptionMonitoringTest.test_implicit_stop_iteration(), that means we skip the monitor_stop_iteration() call in INSTRUMENTED_END_FOR.

test_implicit_stop_iteration() passes as written, though, because the FOR_ITER in the helper function has been specialized to FOR_ITER_GEN by the time the generator finishes, and FOR_ITER_GEN's implementation doesn't perform the same fusion as FOR_ITER.

This bug can be exposed by making this change to test_implicit_stop_iteration(), so the FOR_ITER stays unspecialized while iterating over the generator in question.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• GH-116090: Fix test and clarify behavior for exception events when exhausting a generator. #120697
• GH-116090: Fire RAISE events from _FOR_ITER_TIER_TWO #122413
• [3.13] GH-116090: Fire RAISE events from _FOR_ITER_TIER_TWO (GH-122413) #122419

[swtaarrs]

It should be easy to remove the opcode fusion from FOR_ITER, which would let the END_FOR happen naturally. That could hurt performance a little, but maybe it's OK since common cases should be specialized anyway?

[gvanrossum]

For @markshannon.

[markshannon]

FTR, this is expected behavior. You either get a STOP_ITERATION event or a RAISE event with a StopIteration exception (and corresponding EXCEPTION_HANDLED event).

The reason for this slightly odd behavior is as follows:

• Most tools aren't interested in StopIteration being raised as it isn't a "real" exception.
• For performance reasons, we really don't want to raise StopIteration every time a generator exhausts.
• If tools are interested in StopIteration events that would otherwise be optimized away, we need a way to provide those events without impacting performance for everyone else. That's the STOP_ITERATION event.

[markshannon]

It should be easy to remove the opcode fusion from FOR_ITER, which would let the END_FOR happen naturally.

That would result in both the RAISE event and the STOP_ITERATION event for unoptimized exits from generators.

[markshannon]

In hindsight, we should probably tried to have done the following:

• Not allow STOP_ITERATION to be disabled (for consistency with RAISE events)
• Generate a STOP_ITERATION instead of a RAISE event with StopIteration when a generator or coroutine is exhausted.
• Do not generate EXCEPTION_HANDLED events for those RAISE events that are converted to STOP_ITERATION events.

Maybe we could do this for 3.14