With asynchronous exception raised, thread.is_alive() unexpectedly blocks in Python 3.11 with coverage

[andersk]

Describe the bug

This program (minimized from Zulip’s TimeoutTestCase.test_timeout_warn) succeeds in Python 3.10 without coverage, 3.10 with coverage, and 3.11 without coverage, but it fails in 3.11 with coverage.

import ctypes
import threading
import time

thread = threading.Thread(target=lambda: time.sleep(1))
thread.daemon = True
thread.start()
time.sleep(0.1)
ctypes.pythonapi.PyThreadState_SetAsyncExc(
    ctypes.c_ulong(thread.ident), ctypes.py_object(TimeoutError)
)
time.sleep(0.1)
thread.is_alive()
time.sleep(0.1)
assert thread.is_alive()

The program spawns sleep(1) in a thread, then uses the Python C API to raise an asynchronous exception within it, which has no immediate effect since it’s still sleeping. Both thread.is_alive() calls should immediately return True. But in Python 3.11 with coverage, it seems the first thread.is_alive() call unexpectedly blocks until the thread’s sleep(1) finishes before returning, so the second thread.is_alive() call returns False.

A git bisect of Python shows that this started failing with

• bpo-46409: Make generators in bytecode python/cpython#30633

To Reproduce

Run python3.11 -m coverage run test.py. I’m using the latest coverage==7.2.5 on Linux x86-64 (NixOS 23.05).

[nedbat]

I have no idea what that change in CPython did, or how coverage affects it. If you have any more details about why this might be happening, I'd appreciate them.

I also don't understand why ctypes is needed to demonstrate the problem.

[andersk]

I’m not sure whether ctypes is necessary to demonstrate this problem, but I can say that ctypes is necessary to call PyThreadState_SetAsyncExc (“To prevent naive misuse, you must write your own C extension to call this.”), which is necessary to implement a timeout mechanism for an arbitrary function (see the func_timeout package, for example).

I don’t have more information about where the problem might be yet, but I’ll let you know if I find something.