Unable to use concurrent.futures in atexit hook

[jd]

BPO	42647
Nosy	@pitrou, @vstinner, @jd, @aeros, @iritkatriel

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-12-15.13:59:10.547>
labels = ['3.9', 'type-crash']
title = 'Unable to use concurrent.futures in atexit hook'
updated_at = <Date 2021-02-01.13:41:43.911>
user = 'https://github.com/jd'

bugs.python.org fields:

activity = <Date 2021-02-01.13:41:43.911>
actor = 'corona10'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2020-12-15.13:59:10.547>
creator = 'jd'
dependencies = []
files = []
hgrepos = []
issue_num = 42647
keywords = []
message_count = 4.0
messages = ['383058', '385584', '385972', '386067']
nosy_count = 5.0
nosy_names = ['pitrou', 'vstinner', 'jd', 'aeros', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue42647'
versions = ['Python 3.9']

[jd]

Python 3.9 introduced a regression with concurrent.futures. The following program works fine on Python < 3.8 but raises an error on 3.9:

import atexit
import concurrent.futures

def spawn():
    with concurrent.futures.ThreadPoolExecutor() as t:
        pass

atexit.register(spawn)

$ python3.9 rep.py
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 37, in <module>
    threading._register_atexit(_python_exit)
  File "/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 1370, in _register_atexit
    raise RuntimeError("can't register atexit after shutdown")
RuntimeError: can't register atexit after shutdown

[iritkatriel]

b61b818 is the first bad commit
commit b61b818
Author: Kyle Stanley <aeros167@gmail.com>
Date: Fri Mar 27 15:31:22 2020 -0400

bpo-39812: Remove daemon threads in concurrent.futures (GH-19149)

Remove daemon threads from :mod:`concurrent.futures` by adding
an internal `threading._register_atexit()`, which calls registered functions
prior to joining all non-daemon threads. This allows for compatibility
with subinterpreters, which don't support daemon threads.

[aeros]

Thanks for bringing attention to this, Julien.

While the regression is definitely unfortunate, I'm uncertain about whether the behavior is *correct* to allow in the first place. Specifically, allowing the registration of an atexit hook which uses a ThreadPoolExecutor within it means that the finalization of the executor will be done *after* thread finalization occurs, leaving dangling threads which will have to be abruptly killed upon interpreter exit instead of being safely joined. From my perspective at least, this doesn't seem like something to be encouraged.

Is there a real-world situation where it's specifically necessary or even beneficial to utilize ThreadPoolExecutor at this point after thread finalization rather than earlier in the program? Not that it doesn't exist, but to me it intuitively seems very odd to utilize an executor within an atexit hook, which are intended to just be resource finalization/cleanup functions called at interpreter shutdown. Assuming there is a genuine use case I'm not seeing, it may be worth weighing against the decision to convert the executors to not use daemon threads, as I presently don't think there's a way to (safely) allow that behavior without reverting back to using daemon threads.

That said, I'll admit that I'm more than a bit biased as the author of the commit which introduced the regression, so I'll CC Antoine Pitrou (active expert for threading and concurrent.futures) to help make the final decision.

[jd]

Is there a real-world situation where it's specifically necessary or even beneficial to utilize ThreadPoolExecutor at this point after thread finalization rather than earlier in the program? Not that it doesn't exist, but to me it intuitively seems very odd to utilize an executor within an atexit hook, which are intended to just be resource finalization/cleanup functions called at interpreter shutdown. Assuming there is a genuine use case I'm not seeing, it may be worth weighing against the decision to convert the executors to not use daemon threads, as I presently don't think there's a way to (safely) allow that behavior without reverting back to using daemon threads.

To put that in perspective, here is the original issue that trigged this bug for me:

Traceback (most recent call last):
  File "/root/project/ddtrace/profiling/scheduler.py", line 50, in flush
    exp.export(events, start, self._last_export)
  File "/root/project/ddtrace/profiling/exporter/http.py", line 186, in export
    self._upload(client, self.endpoint_path, body, headers)
  File "/root/project/ddtrace/profiling/exporter/http.py", line 189, in _upload
    self._retry_upload(self._upload_once, client, path, body, headers)
  File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__
    do = self.iter(retry_state=retry_state)
  File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/tenacity/__init__.py", line 360, in iter
    return fut.result()
  File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/_base.py", line 433, in result
    return self.__get_result()
  File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/tenacity/__init__.py", line 426, in __call__
    result = fn(*args, **kwargs)
  File "/root/project/ddtrace/profiling/exporter/http.py", line 193, in _upload_once
    client.request("POST", path, body=body, headers=headers)
  File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 921, in connect
    self.sock = self._create_connection(
  File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/socket.py", line 88, in create_connection
    addrs = list(getaddrinfo(host, port, 0, SOCK_STREAM))
  File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_socketcommon.py", line 247, in getaddrinfo
    addrlist = get_hub().resolver.getaddrinfo(host, port, family, type, proto, flags)
  File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/hub.py", line 841, in _get_resolver
    self._resolver = self.resolver_class(hub=self) # pylint:disable=not-callable
  File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/resolver/thread.py", line 39, in __init__
    self.pool = hub.threadpool
  File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/hub.py", line 865, in _get_threadpool
    self._threadpool = self.threadpool_class(self.threadpool_size, hub=self)
  File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/hub.py", line 860, in threadpool_class
    return GEVENT_CONFIG.threadpool
  File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 50, in getter
    return self.settings[setting_name].get()
  File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 146, in get
    self.value = self.validate(self._default())
  File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 248, in validate
    return self._import_one_of([self.shortname_map.get(x, x) for x in value])
  File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 223, in _import_one_of
    return self._import_one(candidates[-1])
  File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 237, in _import_one
    module = importlib.import_module(module)
  File "/root/.pyenv/versions/3.9.0/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/threadpool.py", line 748, in <module>
    class ThreadPoolExecutor(concurrent.futures.ThreadPoolExecutor):
  File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/__init__.py", line 49, in __getattr__
    from .thread import ThreadPoolExecutor as te
  File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/thread.py", line 37, in <module>
    threading._register_atexit(_python_exit)
  File "/root/.pyenv/versions/3.9.0/lib/python3.9/threading.py", line 1370, in _register_atexit
    raise RuntimeError("can't register atexit after shutdown")
RuntimeError: can't register atexit after shutdown

What's happening is that the ddtrace library registers an atexit hook that does an HTTP call. As the application runs using gevent, some gevent modules that were not loaded are loaded during the atexit() hook and the loading of concurrent.futures.thread is done very late, at the point where the interpreter is shutting down.

I'm totally fine blaming gevent here if you prefer. The problem is that there's nothing preventing any library call to be made in an atexit() hook, and any library could decide to use concurent.futures without the library user being able to do anything about it, except maybe, making sure concurent.futures is loaded very early in the program. However, having to load this library even if you don't use it to be sure it does not break would break the separation of principles.

At at least, at this stage, it might be the responsibility of Python to make sure all threading._register_at_exit calls are done whatever if the library is used or not (i.e. that'd mean loading concurent.futures with threading unconditionally).

[jamesra]

I'm running into the same issue after attempting to migrate my code to Python 3.10. My use case is there is a temporary folder full of files. I have an atexit hook which deletes the temporary folder recursively in parallel. The temporary folder is over a network mount and shutil.rmtree runs too slowly. I'm far from a python expert but it seems very odd to remove API functionality simply because a notification has been triggered. I'm not really sure how to work around this other than registering a separate "atexit safe" rmtree implementation. How is a user supposed to know which calls are safe to use in atexit and which are not?

In case it is informative this is the implementation for my use-case. I register this function to be called by atexit. (Credit goes to a stackoverflow post I read sometime trying to figure out why shutil.rmtree was so slow)

atexit.register(rmtree, _sharedTempRoot)

def rmtree(directory):  
    with concurrent.futures.ThreadPoolExecutor() as executor:
        for root, dirs, files in os.walk(directory, topdown=False):
            try: 
                files_futures = executor.map(os.remove, [os.path.join(root, f) for f in files])
                t = list(files_futures) #Force the map operation to complete
            except Exception as e:
                prettyoutput.error(f'{e}')
                
            try:
                dir_futures = executor.map(os.rmdir, [os.path.join(root, d) for d in dirs])
                d = list(dir_futures) #Force the map operation to complete
            except Exception as e:
                prettyoutput.error(f'{e}')
            
    os.rmdir(directory)
            
    return