3.12 regression: "can't create new thread at interpreter shutdown" from non-daemon threads or atexit handlers

[zenbones]

Bug report

Bug description:

I have a single threading.Timer held on a class as self.timer...

    def refresh_timer(self):
        if self.timer is not None:
            self.timer.cancel()
        self.timer = threading.Timer(self.worker_context.timeout_minutes * 60, self.stop_cycle)
        self.timer.start()

As recently as 3.10 this code worked. As of 3.12 it errors with "can't create new thread at interpreter shutdown". The code has not changed, and the interpreter is not shutting down at the time this error is thrown, as in the process is live and code is executing. There are no code changes from where this is working in 3.10 to where it is not in 3.12.1 (upgraded from 3.12 to 3.12.1 and tsted again to see if this had been fixed).

CPython versions tested on:

3.12

Operating systems tested on:

Windows

Linked PRs

• gh-113964: Don't prevent new threads until all non-daemon threads exit #116677
• gh-113964: Restore the ability to start threads from atexit handler functions. #116982
• [3.12] gh-113964: Don't prevent new threads until all non-daemon threads exit (GH-116677) #117029

[zware]

Can you please provide a full, self-contained reproducer? The provided snippet does not appear to be complete.

[zenbones]

Not easily. The use case is complex, but the error is real, and was not there in 3.10. Maybe you can help me define a case I can get a reproducer for, or help me debug why this is happening now. The process entry point uses importlib.import_module() to load 'external code', starts a thread...

class WorkerThread(threading.Thread):
    def __init__(self, worker):
        threading.Thread.__init__(self)

        self.setDaemon(False)
        self.stopped = False
        self.worker = worker

self.worker_thread = WorkerThread(self)
self.worker_thread.start()

...and starts the timer (which is another thread I guess)...

self.timer = None
self.refresh_timer()

...and then, I don't know... exits, but doesn't because the WorkerThread is not a daemon. The WorkerThread uses the the passed in 'worker' to handle all the business logic. Periodically, as in often, the refresh_timer() method will be called from the WorkerThread instance, via the self.worker reference.

The initial timer setup is fine in the 'main' thread, but after that thread finishes (but doe not exit due to the non-daemon thread), the refresh_timer() is called again and fails with "can't create new thread at interpreter shutdown". Maybe this is related to work on multiple interpreters in 3.12?

Explaining this this way has helped in any case, and maybe I can get these classes setup. I think the module load, which is hard to replicate, may have nothing to do with this and I can try with a main thread,, a worker thread, and a timer thread. However, if the answer is obvious from my use case don't hesitate to save me some work and tell me what's up.

[zenbones]

Voila...

import threading
import time

def main():
	
	worker = Worker();

class Worker:
    def __init__(self):
        self.worker_thread = WorkerThread(self)
        self.worker_thread.start()

        self.timer = None
        self.refresh_timer()
        print("worker init complete")

    def refresh_timer(self):
        if self.timer is not None:
            self.timer.cancel()
        self.timer = threading.Timer(30 * 60, self.stop_cycle)
        self.timer.start()

    def stop_cycle(self):
 	    print("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")

class WorkerThread(threading.Thread):
    def __init__(self, worker):
        threading.Thread.__init__(self)

        self.setDaemon(False)
        self.stopped = False
        self.worker = worker

    def run(self):
        while True:
        	print("running...")
        	self.worker.refresh_timer()
        	time.sleep(3)

main()

C:\Users\david\Documents\testme.py:30: DeprecationWarning: setDaemon() is deprecated, set the daemon attribute instead
  self.setDaemon(False)
running...
worker init complete
running...
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Program Files\Python312\Lib\threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "C:\Users\david\Documents\testme.py", line 37, in run
    self.worker.refresh_timer()
  File "C:\Users\david\Documents\testme.py", line 21, in refresh_timer
    self.timer.start()
  File "C:\Program Files\Python312\Lib\threading.py", line 992, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't create new thread at interpreter shutdown

[zenbones]

If I keep the main entry point running, as with timer.sleep(a_long_time), then I don't get the thread creation error. However, that would force me to throw in some waiting loop on a lock to keep the main thread alive until the non-daemon thread terminates, being very careful to trap any error and release the main thread (try/catch/finally). But none of that makes any difference to me. The whole point of designating the second thread as non-daemon is so the main thread does not exit until all the non-daemon threads terminate, so the system is responsible for all that.

My guess is that whatever throws "can't create new thread at interpreter shutdown" is checking some attribute of the main thread, but failing to check for any running non-daemon threads. Or, whatever code makes up the main thread is responding that yes, it has terminated, despite the fact that non-demon child threads are still running. So I think this is a python bug.

[ronaldoussoren]

On a first glance this behaviour does seem correct:

main() creates a new non-daemon thread that does the actual work, and then returns. That mains the main thread immediately tries to exit the script and is therefore finalising the interpreter, waiting for the WorkerThread to stop before actually doing most of the cleanup and exiting the interpreter.

If you call worker.worker_thread.join() in main() the script will continue to run as long as the worker thread is.

I'm not an expert on the finer details of interpreter shutdown though.

[zenbones]

The problem is that I want the function to return. I'll agree that a join is better than an event or a barrier, but anything like those will prevent the init function return, which is ugly, But more than that, it's wrong. The first problem with the argument that the main thread should exit the interpreter, but not 'finish' until the non-daemon thread exits, is that this is a significant change in behavior. This works correctly in 3.10 and throws this error in 3.12. That alone is dangerous. The second problem is that it makes no sense. A thread which is 'finalising' in the interpreter has exited as far as any client usage, and the docs on daemon threads say...

A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left.

So the 'program' can not exit due to the non-daemon thread, but the main thread kind of can? Even if that wasn't vague enough, if the program has not exited, why can't the live thread create a timer, or start another thread? The program has not exited. There is code running in the interpreter. But it's now blocked from creating a new thread?

The docs should be amended to say...

A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left. However, a program running only child non-daemon threads will not be able to create new threads, such as with timer.start().

Does that really make sense?

[zenbones]

by the way, def _shutdown() in threading .py, contains this code...

    # Join all non-deamon threads
    while True:
        with _shutdown_locks_lock:
            locks = list(_shutdown_locks)
            _shutdown_locks.clear()

        if not locks:
            break

        for lock in locks:
            # mimic Thread.join()
            lock.acquire()
            lock.release()

        # new threads can be spawned while we were waiting for the other
        # threads to complete

Note the comments. I really think the problem is the check in timer.start() which s not properly deciding to check for active non-daemon threads. I'm trying to find that code. Any pointers to it would be helpful.

[ronaldoussoren]

There should IMHO at least be documentation about this behaviour. I'm therefore reopening the issue.

I've done some research. Py_FinalizeEx first sets the finalizing flag in the interpreter state, the waits for threads to exit:

cpython/Python/pylifecycle.c

Lines 1823 to 1842 in ac92527

	int
	Py_FinalizeEx(void)
	{
	int status = 0;
	_PyRuntimeState *runtime = &_PyRuntime;
	if (!runtime->initialized) {
	return status;
	}
	/* Get current thread state and interpreter pointer */
	PyThreadState *tstate = _PyThreadState_GET();
	// XXX assert(_Py_IsMainInterpreter(tstate->interp));
	// XXX assert(_Py_IsMainThread());
	// Block some operations.
	tstate->interp->finalizing = 1;
	// Wrap up existing "threading"-module-created, non-daemon threads.
	wait_for_thread_shutdown(tstate);

Starting a new thread ends up in _threading.start_joinable_thead, which in the end calls do_start_new_thread which raises an exception when the finalizing flag in the interpreter state is set:

cpython/Modules/_threadmodule.c

Lines 1249 to 1265 in ac92527

	static int
	do_start_new_thread(thread_module_state* state,
	PyObject *func, PyObject* args, PyObject* kwargs,
	int joinable,
	PyThread_ident_t* ident, PyThread_handle_t* handle)
	{
	PyInterpreterState *interp = _PyInterpreterState_GET();
	if (!_PyInterpreterState_HasFeature(interp, Py_RTFLAGS_THREADS)) {
	PyErr_SetString(PyExc_RuntimeError,
	"thread is not supported for isolated subinterpreters");
	return -1;
	}
	if (interp->finalizing) {
	PyErr_SetString(PyExc_RuntimeError,
	"can't create new thread at interpreter shutdown");
	return -1;
	}

[zenbones]

This is unfortunately deep as Timer sub-classes Thread, and the error gets thrown in do_start_new_thread() of _threadmodule.c...

    if (interp->finalizing) {
        PyErr_SetString(PyExc_RuntimeError,
                        "can't create new thread at interpreter shutdown");
        return -1;
    }

So the interpreter is responding that it is finalizing, so I think my contention is either that...

1. The interpreter should be blocked from entering the finalizing state until all non-daemon threads have exited. I think that makes sense if there's a single interpreter because there's code still running. My guess is this bug is tied up with the work on allowing multiple interpreters, which makes sense version wise, and the main thread's interpreter is finalizing and there are now other interpreters for the non-daemon threads, but the thread start only checks the main interpreter?

or

2. That both the main thread and all non-daemon thread interpreters need to be checked before throwing that error?

There is a specific test for this error being thrown in test_threading.py, but no test that this error is not thrown if there's a non-daemon thread still running.

[zenbones]

And yes, what you saw above. Thank you.