Skip to content

test_concurrent_futures.test_shutdown: test_interpreter_shutdown() fails randomly (race condition) #109047

Closed
@vstinner

Description

@vstinner

The test_interpreter_shutdown() test of test_concurrent_futures.test_shutdown has a race condition. On purpose, the test doesn't wait until the executor completes (!). Moreover, it expects the executor to always be able to submit its job, and the job to complete successfully! It's a very optimistic bet.

See also issue #107219: test_concurrent_futures: test_crash_big_data() hangs randomly on Windows.


When Python is shutting down, Py_Finalize() quickly blocks the creation of new threads in _thread.start_new_thread():

if (interp->finalizing) {
PyErr_SetString(PyExc_RuntimeError,
"can't create new thread at interpreter shutdown");
return NULL;
}

This exception was added recently (last June) by commit ce558e6: see issue gh-104690 for the rationale.

The multiprocessing executor spawns _ExecutorManagerThread thread which runs its "main loop" in its run() method:

def run(self):
# Main loop for the executor manager thread.
while True:
self.add_call_item_to_queue()
result_item, is_broken, cause = self.wait_result_broken_or_wakeup()
if is_broken:
self.terminate_broken(cause)
return
if result_item is not None:
self.process_result_item(result_item)
process_exited = result_item.exit_pid is not None
if process_exited:
p = self.processes.pop(result_item.exit_pid)
p.join()
# Delete reference to result_item to avoid keeping references
# while waiting on new results.
del result_item
if executor := self.executor_reference():
if process_exited:
with self.shutdown_lock:
executor._adjust_process_count()
else:
executor._idle_worker_semaphore.release()
del executor
if self.is_shutting_down():
self.flag_executor_shutting_down()
# When only canceled futures remain in pending_work_items, our
# next call to wait_result_broken_or_wakeup would hang forever.
# This makes sure we have some running futures or none at all.
self.add_call_item_to_queue()
# Since no new work items can be added, it is safe to shutdown
# this thread if there are no pending work items.
if not self.pending_work_items:
self.join_executor_internals()
return

It tries to submit new jobs to the worker process through a queue, but oops, the Python main thread is finalizing (called Py_Finalizing())! There is not notification system to notify threads that Python is being finalized.


Moreover, there are 3 "finalization" states:

  • interp->finalizing -- used by _thread.start_new_thread() to block thread creation during Python finazlization
  • runtime->_finalizing -- used by sys.is_finalizing(), Py_IsFinalizing() and _PyRuntimeState_GetFinalizing(runtime)
  • interp->_finalizing -- used by ceval.c to decide if a Python thread "must exit" or not, as soon as it's set, all Python threads must exit as soon as they attempt to acquire the GIL

These 3 states at not set at the same time.

  1. Calling Py_Finalize() sets interp->finalizing to 1 as soon as possible: so spawning new threads is immediately blocked (which is a good thing to get a reliable finalization!)
  2. Py_Finalize() calls threading._shutdown() which blocks until all non-daemon threads completes
  3. Py_Finalize() calls atexit callbacks
  4. And only then, Py_Finalize() sets runtime->_finalizing and interp->_finalizing to the Python thread state (tstate) which calls Py_Finalize()

The delay between (1) and (4) can be quite long, a thread can take several milliseconds, if not seconds, to complete.


Can multiprocessing or concurrent.futures check if Python is finalizing or be notified? Well, did you hear about Time-of-check to time-of-use race conditions? Even if it would be possible, I don't think that we can "check" if it's safe to spawn a thread just before spawning a thread, since the main thread can decide to finalize Python "at any time". It will become even more tricky with Python nogil ;-)

So what's left? Well, multiprocessing and concurrent.futures should be optimistic, call Python functions and only then check for exceptions. Depending on the exceptions, they can decide how to handle it. I would suggest to exit as soon as possible, and try to cleanup resources if possible.

Another option would be to make multiprocessing and concurrent.futures more determistic. Rather than spawning threads and processes in the background "on demand" and hope that everything will be fine, add more synchronization to "wait" until everything is ready to submit jobs. I think that I already tried this approach in the past, but @pitrou didn't like it since it made some workloads slower. You may not always need to actually submits jobs. You may not always need all threads and processes.

Well, I don't know even these complex modules to tell which option is the least bad :-)

Finally, as usually, I beg you to make these APIs less magical, and enforce more explicit resources management! It shouldn't even be possible to not wait until an executor complete. It should be enforced by emitting loudly ResourceWarning warnings :-) Well, that's my opinion. I know that it's not shared by @pitrou :-)

Linked PRs

Metadata

Metadata

Assignees

No one assigned

    Labels

    testsTests in the Lib/test dir

    Projects

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions