concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used

[yilei]

BPO	46464
Nosy	@gpshead, @pitrou, @methane, @yilei
PRs	gh-90622: Prevent concurrent.futures.process deadlock due to fork() after spawning a thread. #30847

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = None
created_at = <Date 2022-01-21.23:32:53.148>
labels = ['3.11', 'library', '3.9', '3.10', 'performance']
title = 'concurrent.futures.ProcessPoolExecutor can deadlock when tcmalloc is used'
updated_at = <Date 2022-01-25.11:06:46.406>
user = 'https://github.com/yilei'

bugs.python.org fields:

activity = <Date 2022-01-25.11:06:46.406>
actor = 'pitrou'
assignee = 'gregory.p.smith'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2022-01-21.23:32:53.148>
creator = 'yilei'
dependencies = []
files = []
hgrepos = []
issue_num = 46464
keywords = ['patch', '3.9regression']
message_count = 7.0
messages = ['411208', '411213', '411451', '411543', '411544', '411545', '411574']
nosy_count = 4.0
nosy_names = ['gregory.p.smith', 'pitrou', 'methane', 'yilei']
pr_nums = ['30847']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue46464'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[yilei]

When Python is built and linked with tcmalloc, using ProcessPoolExecutor may deadlock. Here is a reproducible example:

$ cat t.py
from concurrent import futures
import sys
def work(iteration, item):
  sys.stdout.write(f'working: iteration={iteration}, item={item}\n')
  sys.stdout.flush()
for i in range(0, 10000):
    with futures.ProcessPoolExecutor(max_workers=2) as executor:
        executor.submit(work, i, 1)
        executor.submit(work, i, 2)

$ python t.py
working: iteration=0, item=1
working: iteration=0, item=2
working: iteration=1, item=1
working: iteration=1, item=2
...
working: iteration=3631, item=1
working: iteration=3631, item=2
<hang here>

The child process fails to finish. It's more likely to reproduce when the system is busy.

With some bisect search internally, this commit 1ac6e37 "triggered" the deadlock threshold with tcmalloc.

[gpshead]

For context, the fundamental problem is that os.fork() is unsafe to use in any process with threads. concurrent/futures/process.py violates this. So long as its design involves a thread, it can never be guaranteed not to deadlock.

POSIX forbids execution of most code after fork() has been called. Returning to the CPython interpreter after os.fork() in the child process is never safe unless the parent process had no threads at fork time.

The change that triggered the issue moved from doing all of the os.fork() calls for the workers up front *before the thread was spawned* to doing them on demand after the thread was running.

The bugfix for this is simply a rollback to revert the bpo-39207 change. It was a performance enhancement, but it causes a new deadlock in code that didn't already have one when thread tuned malloc implementations are used.

The only way to safely launch worker processes on demand is to spawn a worker launcher process spawned prior to any thread creation that remains idle, with a sole job of spawn new worker processes for us. That sounds complicated. That'd be a feature. Lets go with the bugfix first.

[gpshead]

https://bugs.python.org/issue44733 for 3.11 attempts to build upon the dynamic spawning ability and will need reverting unless a non-thread+fork implementation is provided.