starting a thread in __del__ hangs at interpreter shutdown

[sylikc]

BPO	43784
Nosy	@rhettinger, @pfmoore, @pitrou, @tjguk, @zware, @eryksun, @zooba, @sylikc
Files	win_subprocess_hang.py: Sample code that hangs

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-04-09.05:25:49.909>
labels = ['3.8', 'type-bug', 'library', '3.9', '3.10']
title = 'starting a thread in __del__ hangs at interpreter shutdown'
updated_at = <Date 2021-04-10.06:05:35.590>
user = 'https://github.com/sylikc'

bugs.python.org fields:

activity = <Date 2021-04-10.06:05:35.590>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-04-09.05:25:49.909>
creator = 'sylikc'
dependencies = []
files = ['49947']
hgrepos = []
issue_num = 43784
keywords = []
message_count = 4.0
messages = ['390587', '390593', '390658', '390695']
nosy_count = 8.0
nosy_names = ['rhettinger', 'paul.moore', 'pitrou', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'sylikc']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43784'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[sylikc]

I've noticed an issue (or user error) in which Python a call that otherwise usually works in the __del__ step of a class will freeze when the Python interpreter is exiting.

I've attached sample code that I've ran against Python 3.9.1 on Windows 10.

The code below runs a process and communicates via the pipe.

class SubprocTest(object):
	def run(self):
		print("run")
		proc_args = ["cmd.exe"]
		self._process = subprocess.Popen(proc_args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
	
	def __del__(self):
		print("__del__")
		if self._process is not None:
			self.terminate()
	
	def terminate(self):
		print("terminate")
		self._process.communicate(input=b"exit\n", timeout=1)
		print("kill")
		self._process.kill()
		self._process = None

if __name__ == "__main__":
	s = SubprocTest()
	s.run()
	del s
	print("s done")
	
	t = SubprocTest()
	t.run()
	print("t done")

Current output:
run
__del__
terminate
kill
s done
run
t done
__del__
terminate
<<<<<< hangs indefinitely here, even though timeout=1

Expected output:
run
__del__
terminate
kill
s done
run
t done
__del__
terminate
kill

In normal circumstances, when you del the object and force a run of __del__(), the process ends properly and the terminate() method completes.

When the Python interpreter exits, Python calls the __del__() method of the class. In this case, the terminate() never completes and the script freezes indefinitely on the communicate()

[eryksun]

It's not a subprocess bug, per se. It's due to creating the stdout/stderr worker threads from the __del__ finalizer while the interpreter is shutting down. Minimal reproducer, confirmed in both Linux and Windows:

    import threading

    class C:
        def __del__(self):
            t = threading.Thread(target=print, args=('spam',), daemon=True)
            t.start()

    c = C()
    #del c # uncomment to prevent hanging