Description
Bug report
Bug description:
from concurrent.futures import ProcessPoolExecutor as Pool
def nope():
pass
def mk_done(pool):
def done(_fut):
pool.shutdown(wait=True)
return done
if __name__ == "__main__":
pool = Pool()
future = pool.submit(nope)
future.add_done_callback(mk_done(pool))
ProcessPoolExecutor
invokes future done callbacks from its manager thread. When one of those future callbacks invokes ProcessPoolExecutor.shutdown()
, shut down fails with a RuntimeError: cannot join current thread
because the implementation joins the manager thread without protecting against self-joins.
The relevant lines Lib/concurrent/futures/process.py:845-846
currently read:
if self._executor_manager_thread is not None and wait:
self._executor_manager_thread.join()
but should probably be updated to:
t = self._executor_manager_thread
if t is not None and wait and threading.current_thread() != t:
self._executor_manager_thread.join()
Invoking shutdown(False)
avoids the problem but also makes it impossible to wait for pool completion. A more general solution would be to deprecate the wait
parameter for shutdown
, never waiting in that method, and add a separate method, say wait_for_shutdown
, that uses a threading.Event
to wake any waiting threads. Alas, correctly setting that Event
is going to be a bit involved.
While I tested on 3.12, the bug is also present in 3.13 and current.
CPython versions tested on:
3.12
Operating systems tested on:
macOS