Threads started in exit handler are still running after their thread states are destroyed

[chgnrdv]

Reproduced on main (663c049), bisected to 283ab0e.

import atexit
import threading

def t0():
    pass

def t1():
    threading.Thread(target=t0).start()

def f():
    threading.Thread(target=t1).start()

atexit.register(f)
exit()

Output (can also pass without crash or segv depending on at which point t0 tries to access its tstate):

python: Python/pystate.c:244: bind_tstate: Assertion `tstate_is_alive(tstate) && !tstate->_status.bound' failed.
Aborted (core dumped)

Stack traces:

(gdb) b Python/pylifecycle.c:1810
Breakpoint 1 at 0x372529: file Python/pylifecycle.c, line 1828.
(gdb) r repro.py 
Starting program: /home/chgnrdv/cpython/python repro.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff7621700 (LWP 1539803)]
[New Thread 0x7ffff6e20700 (LWP 1539804)]
python: Python/pystate.c:244: bind_tstate: Assertion `tstate_is_alive(tstate) && !tstate->_status.bound' failed.

Thread 3 "python" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff6e20700 (LWP 1539804)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff7c8f537 in __GI_abort () at abort.c:79
#2  0x00007ffff7c8f40f in __assert_fail_base (fmt=0x7ffff7e076a8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=0x555555a1e830 "tstate_is_alive(tstate) && !tstate->_status.bound", file=0x555555a1df0f "Python/pystate.c", line=244, function=<optimized out>)
    at assert.c:92
#3  0x00007ffff7c9e662 in __GI___assert_fail (assertion=assertion@entry=0x555555a1e830 "tstate_is_alive(tstate) && !tstate->_status.bound", 
    file=file@entry=0x555555a1df0f "Python/pystate.c", line=line@entry=244, function=function@entry=0x555555a1f678 <__PRETTY_FUNCTION__.52> "bind_tstate")
    at assert.c:101
#4  0x00005555558c8213 in bind_tstate (tstate=tstate@entry=0x7ffff0000e40) at Python/pystate.c:244
#5  0x00005555558c9d9e in _PyThreadState_Bind (tstate=tstate@entry=0x7ffff0000e40) at Python/pystate.c:1929
#6  0x000055555596afac in thread_run (boot_raw=boot_raw@entry=0x7ffff77e80e0) at ./Modules/_threadmodule.c:1077
#7  0x00005555558e5ce7 in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:233
#8  0x00007ffff7f98ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#9  0x00007ffff7d69a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) frame level 4
#4  0x00005555558c8213 in bind_tstate (tstate=tstate@entry=0x7ffff0000e40) at Python/pystate.c:244
244	    assert(tstate_is_alive(tstate) && !tstate->_status.bound);
(gdb) print *tstate
$1 = {prev = 0xdddddddddddddddd, next = 0xdddddddddddddddd, interp = 0xdddddddddddddddd, _status = {initialized = 1, bound = 0, unbound = 1, bound_gilstate = 1, 
    active = 1, finalizing = 0, cleared = 1, finalized = 1}, py_recursion_remaining = -572662307, py_recursion_limit = -572662307, c_recursion_remaining = -572662307, 
  recursion_headroom = -572662307, tracing = -572662307, what_event = -572662307, cframe = 0xdddddddddddddddd, c_profilefunc = 0xdddddddddddddddd, 
  c_tracefunc = 0xdddddddddddddddd, c_profileobj = 0xdddddddddddddddd, c_traceobj = 0xdddddddddddddddd, current_exception = 0xdddddddddddddddd, 
  exc_info = 0xdddddddddddddddd, dict = 0xdddddddddddddddd, gilstate_counter = -572662307, async_exc = 0xdddddddddddddddd, thread_id = 15987178197214944733, 
  native_thread_id = 15987178197214944733, trash = {delete_nesting = -572662307, delete_later = 0xdddddddddddddddd}, on_delete = 0xdddddddddddddddd, 
  on_delete_data = 0xdddddddddddddddd, coroutine_origin_tracking_depth = -572662307, async_gen_firstiter = 0xdddddddddddddddd, 
  async_gen_finalizer = 0xdddddddddddddddd, context = 0xdddddddddddddddd, context_ver = 15987178197214944733, id = 15987178197214944733, 
  datastack_chunk = 0xdddddddddddddddd, datastack_top = 0xdddddddddddddddd, datastack_limit = 0xdddddddddddddddd, exc_state = {exc_value = 0xdddddddddddddddd, 
    previous_item = 0xdddddddddddddddd}, root_cframe = {current_frame = 0xdddddddddddddddd, previous = 0xdddddddddddddddd}}
(gdb) info threads
  Id   Target Id                                    Frame 
  1    Thread 0x7ffff7c6c280 (LWP 1539799) "python" Py_FinalizeEx () at Python/pylifecycle.c:1828
  2    Thread 0x7ffff7621700 (LWP 1539803) "python" futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffff00010b0)
    at ../sysdeps/nptl/futex-internal.h:323
* 3    Thread 0x7ffff6e20700 (LWP 1539804) "python" __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff7621700 (LWP 1539803))]
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffff00010b0) at ../sysdeps/nptl/futex-internal.h:323
323	../sysdeps/nptl/futex-internal.h: No such file or directory.
(gdb) bt
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffff00010b0) at ../sysdeps/nptl/futex-internal.h:323
#1  do_futex_wait (sem=sem@entry=0x7ffff00010b0, abstime=0x0, clockid=0) at sem_waitcommon.c:112
#2  0x00007ffff7fa2278 in __new_sem_wait_slow (sem=sem@entry=0x7ffff00010b0, abstime=0x0, clockid=0) at sem_waitcommon.c:184
#3  0x00007ffff7fa22f1 in __new_sem_wait (sem=sem@entry=0x7ffff00010b0) at sem_wait.c:42
#4  0x00005555558e6120 in PyThread_acquire_lock_timed (lock=lock@entry=0x7ffff00010b0, microseconds=microseconds@entry=-1000000, intr_flag=intr_flag@entry=1)
    at Python/thread_pthread.h:478
#5  0x000055555596a3be in acquire_timed (lock=0x7ffff00010b0, timeout=-1000000000) at ./Modules/_threadmodule.c:98
#6  0x000055555596a50d in lock_PyThread_acquire_lock (self=0x7ffff77d9c70, args=<optimized out>, kwds=<optimized out>) at ./Modules/_threadmodule.c:179
#7  0x000055555570defe in method_vectorcall_VARARGS_KEYWORDS (func=0x7ffff792ef90, args=0x7ffff7fbd378, nargsf=<optimized out>, kwnames=<optimized out>)
    at Objects/descrobject.c:365
#8  0x00005555556fc9ca in _PyObject_VectorcallTstate (tstate=0x555555d24ac0, callable=0x7ffff792ef90, args=0x7ffff7fbd378, nargsf=9223372036854775809, kwnames=0x0)
    at ./Include/internal/pycore_call.h:92
#9  0x00005555556fcae5 in PyObject_Vectorcall (callable=callable@entry=0x7ffff792ef90, args=args@entry=0x7ffff7fbd378, nargsf=<optimized out>, 
    kwnames=kwnames@entry=0x0) at Objects/call.c:325
#10 0x0000555555856917 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x555555d24ac0, frame=0x7ffff7fbd300, throwflag=throwflag@entry=0) at Python/bytecodes.c:2643
#11 0x000055555585e296 in _PyEval_EvalFrame (throwflag=0, frame=<optimized out>, tstate=0x555555d24ac0) at ./Include/internal/pycore_ceval.h:87
#12 _PyEval_Vector (tstate=0x555555d24ac0, func=0x7ffff7649fd0, locals=locals@entry=0x0, args=0x7ffff7620db8, argcount=1, kwnames=0x0) at Python/ceval.c:1610
#13 0x00005555556fc45b in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:419
#14 0x0000555555700b6f in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0x7ffff7620db8, callable=0x7ffff7649fd0, tstate=0x555555d24ac0)
    at ./Include/internal/pycore_call.h:92
#15 method_vectorcall (method=<optimized out>, args=0x555555ca9190 <_PyRuntime+91760>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/classobject.c:67
#16 0x00005555556fecfe in _PyVectorcall_Call (tstate=tstate@entry=0x555555d24ac0, func=0x5555557008f9 <method_vectorcall>, callable=callable@entry=0x7ffff77b03b0, 
    tuple=tuple@entry=0x555555ca9178 <_PyRuntime+91736>, kwargs=kwargs@entry=0x0) at Objects/call.c:271
#17 0x00005555556ff0aa in _PyObject_Call (tstate=0x555555d24ac0, callable=0x7ffff77b03b0, args=0x555555ca9178 <_PyRuntime+91736>, kwargs=0x0) at Objects/call.c:354
#18 0x00005555556ff103 in PyObject_Call (callable=<optimized out>, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:379
#19 0x000055555596afdc in thread_run (boot_raw=boot_raw@entry=0x7ffff77ada80) at ./Modules/_threadmodule.c:1081
#20 0x00005555558e5ce7 in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:233
#21 0x00007ffff7f98ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#22 0x00007ffff7d69a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) frame level 19
#19 0x000055555596afdc in thread_run (boot_raw=boot_raw@entry=0x7ffff77ada80) at ./Modules/_threadmodule.c:1081
1081	    PyObject *res = PyObject_Call(boot->func, boot->args, boot->kwargs);
(gdb) print *tstate
$2 = {prev = 0xdddddddddddddddd, next = 0xdddddddddddddddd, interp = 0xdddddddddddddddd, _status = {initialized = 1, bound = 0, unbound = 1, bound_gilstate = 1, 
    active = 1, finalizing = 0, cleared = 1, finalized = 1}, py_recursion_remaining = -572662307, py_recursion_limit = -572662307, c_recursion_remaining = -572662307, 
  recursion_headroom = -572662307, tracing = -572662307, what_event = -572662307, cframe = 0xdddddddddddddddd, c_profilefunc = 0xdddddddddddddddd, 
  c_tracefunc = 0xdddddddddddddddd, c_profileobj = 0xdddddddddddddddd, c_traceobj = 0xdddddddddddddddd, current_exception = 0xdddddddddddddddd, 
  exc_info = 0xdddddddddddddddd, dict = 0xdddddddddddddddd, gilstate_counter = -572662307, async_exc = 0xdddddddddddddddd, thread_id = 15987178197214944733, 
  native_thread_id = 15987178197214944733, trash = {delete_nesting = -572662307, delete_later = 0xdddddddddddddddd}, on_delete = 0xdddddddddddddddd, 
  on_delete_data = 0xdddddddddddddddd, coroutine_origin_tracking_depth = -572662307, async_gen_firstiter = 0xdddddddddddddddd, 
  async_gen_finalizer = 0xdddddddddddddddd, context = 0xdddddddddddddddd, context_ver = 15987178197214944733, id = 15987178197214944733, 
  datastack_chunk = 0xdddddddddddddddd, datastack_top = 0xdddddddddddddddd, datastack_limit = 0xdddddddddddddddd, exc_state = {exc_value = 0xdddddddddddddddd, 
    previous_item = 0xdddddddddddddddd}, root_cframe = {current_frame = 0xdddddddddddddddd, previous = 0xdddddddddddddddd}}
(gdb) thread 1
[Switching to thread 1 (Thread 0x7ffff7c6c280 (LWP 1539799))]
#0  Py_FinalizeEx () at Python/pylifecycle.c:1828
1828	    if (flush_std_files() < 0) {
(gdb) bt
#0  Py_FinalizeEx () at Python/pylifecycle.c:1828
#1  0x00005555558c6dd4 in Py_Exit (sts=0) at Python/pylifecycle.c:3002
#2  0x00005555558cf99e in handle_system_exit () at Python/pythonrun.c:756
#3  0x00005555558cfcea in _PyErr_PrintEx (tstate=0x555555d07728 <_PyRuntime+478216>, set_sys_last_vars=set_sys_last_vars@entry=1) at Python/pythonrun.c:765
#4  0x00005555558d00e3 in PyErr_PrintEx (set_sys_last_vars=set_sys_last_vars@entry=1) at Python/pythonrun.c:845
#5  0x00005555558d00f3 in PyErr_Print () at Python/pythonrun.c:851
#6  0x00005555558d06bb in _PyRun_SimpleFileObject (fp=fp@entry=0x555555d35500, filename=filename@entry=0x7ffff775ac70, closeit=closeit@entry=1, 
    flags=flags@entry=0x7fffffffdfa8) at Python/pythonrun.c:439
#7  0x00005555558d08bd in _PyRun_AnyFileObject (fp=fp@entry=0x555555d35500, filename=filename@entry=0x7ffff775ac70, closeit=closeit@entry=1, 
    flags=flags@entry=0x7fffffffdfa8) at Python/pythonrun.c:78
#8  0x00005555558f80d7 in pymain_run_file_obj (program_name=program_name@entry=0x7ffff77a95b0, filename=filename@entry=0x7ffff775ac70, skip_source_first_line=0)
    at Modules/main.c:360
#9  0x00005555558f83b9 in pymain_run_file (config=config@entry=0x555555ce9a60 <_PyRuntime+356160>) at Modules/main.c:379
#10 0x00005555558f9480 in pymain_run_python (exitcode=exitcode@entry=0x7fffffffe11c) at Modules/main.c:610
#11 0x00005555558f94d8 in Py_RunMain () at Modules/main.c:689
#12 0x00005555558f952c in pymain_main (args=args@entry=0x7fffffffe160) at Modules/main.c:719
#13 0x00005555558f95a1 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:743
#14 0x000055555565273e in main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15

Looks like some race condition between main thread and threads started and still running at finalization, whose tstate main thread is trying to delete, although I'm not an expert and didn't make further debugging.

Platform: Linux 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux

Linked PRs

• gh-104690 Disallow thread creation and fork at interpreter finalization #104826
• [3.12] gh-104690 Disallow thread creation and fork at interpreter finalization (GH-104826) #105277
• gh-104690: thread_run() checks for tstate dangling pointer #109056
• [3.12] gh-104690: thread_run() checks for tstate dangling pointer (#109056) #109133
• [3.11] gh-104690: thread_run() checks for tstate dangling pointer (#109056) #109134

[chgnrdv]

cc @brandtbucher

[sunmy2019]

I am not sure if this is a bug. The thread cleanup happens before atexit calls. (ignore the outdated comments about threading in the _PyAtExit_Call, now the thread join happen inside wait_for_thread_shutdown)

cpython/Python/pylifecycle.c

Lines 1761 to 1777 in d1732fe

	// Wrap up existing "threading"-module-created, non-daemon threads.
	wait_for_thread_shutdown(tstate);
	// Make any remaining pending calls.
	_Py_FinishPendingCalls(tstate);
	/* The interpreter is still entirely intact at this point, and the
	* exit funcs may be relying on that. In particular, if some thread
	* or exit func is still waiting to do an import, the import machinery
	* expects Py_IsInitialized() to return true. So don't say the
	* runtime is uninitialized until after the exit funcs have run.
	* Note that Threading.py uses an exit func to do a join on all the
	* threads created thru it, so this also protects pending imports in
	* the threads created via Threading.
	*/
	_PyAtExit_Call(tstate->interp);

So you need to clean up those Thread objects on your own. For this specific problem, you can do the following.

import atexit
import sys
import threading
import gc

def t0():
    pass

def t1():
    t = threading.Thread(target=t0)
    t.start()
    t.join()

def f():
    t = threading.Thread(target=t1)
    t.start()
    t.join()

atexit.register(f)

exit()

[sunmy2019]

CC @vstinner, who might be an expert here

[chgnrdv]

Surely, it's an edge case, and in real Python code these threads (whose creation at interpreter exit also looks strange to me) should be manually joined. But I found this worth reporting anyway :)

[chgnrdv]

Looks like the same issue was spotted earlier in #103824 as reproducible with bug_15 script, but I still can't reproduce it on ASAN build without Py_DEBUG defined, and on debug build it appears occasionally, so I might have missed it back then.

[vstinner]

_thread.start_new_thread() must fail if Python is being finalized: if tstate->interp->finalizing is non-zero.

See Python/pylifecycle.c:

    // Block some operations.
    tstate->interp->finalizing = 1;

    // Wrap up existing "threading"-module-created, non-daemon threads.
    wait_for_thread_shutdown(tstate);

    // Make any remaining pending calls.
    _Py_FinishPendingCalls(tstate);

    /* The interpreter is still entirely intact at this point, and the
     * exit funcs may be relying on that.  In particular, if some thread
     * or exit func is still waiting to do an import, the import machinery
     * expects Py_IsInitialized() to return true.  So don't say the
     * runtime is uninitialized until after the exit funcs have run.
     * Note that Threading.py uses an exit func to do a join on all the
     * threads created thru it, so this also protects pending imports in
     * the threads created via Threading.
     */

    _PyAtExit_Call(tstate->interp);

It's too late to spawn new threads. Currently, there is a race condition is there is a pending pthread_create() call while _PyThreadState_DeleteExcept() and/or PyInterpreterState_Delete(interp) are being called.

[vstinner]

IMO we should also disallow spawning new processes at this point: os.fork(), subprocess.Popen, etc.

[gpshead]

IMO we should also disallow spawning new processes at this point: os.fork(), subprocess.Popen, etc.

I'd exclude subprocess.Popen from that - its child process isn't using the Python interpreter anymore. (barring preexec_fn= but we already tell people that preexec_fn= is a foot-gun not to be used so... raising an error if that is set when called during finalization would be fine)