Repository navigation
SharedMemory.unlink() can fail if called during interpreter shutdown #91577
Description
SharedMemory.unlink() contains a import which can fail if called during interpreter shutdown.
For example when working with SharedMemory as the buffer for a numpy array it is useful to create a wrapper object to ensure that clean up is done with the object is deleted.
#import numpy as np
from multiprocessing import shared_memory
class ShArray:
def __init__(self):
self.shared_memory = shared_memory.SharedMemory(create=True, size=400)
#self.array = np.ndarray((10,10), dtype="i4", buffer=self.shared_memory.buf)
def __del__(self):
self.shared_memory.close()
self.shared_memory.unlink()
a1 = ShArray()(Commented lines show my typical usage, but are not need to demonstrate the issue)
Exception ignored in: <function ShArray.__del__ at 0x7f36eee510d0>
Traceback (most recent call last):
File "/home/sam/git/cpython/test_shared_mem2.py", line 13, in __del__
File "/home/sam/git/cpython/Lib/multiprocessing/shared_memory.py", line 240, in unlink
ImportError: sys.meta_path is None, Python is likely shutting down
/home/sam/git/cpython/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
This happens on 3.8 and current main, both on Linux x86-64.
Patch to follow
Activity
pythongh-91577: SharedMemory move imports out of methods
gh-91577: SharedMemory move imports out of methods (#91579)
pythongh-91577: SharedMemory move imports out of methods (pythonGH-91579
pythongh-91577: SharedMemory move imports out of methods (pythonGH-91579
vstinner commentedon Jun 16, 2022
I'm not a supporter of multiprocessing programs relying on the GC to call
__del__()finalizers. You should close resources explicitly, usingwith:statement if possible. But well, the change seems acceptable to me, so I merged it.cc @pitrou
gh-91577: SharedMemory move imports out of methods (GH-91579)
gh-91577: SharedMemory move imports out of methods (GH-91579)
pitrou commentedon Jun 16, 2022
Seems fine to me.
withstatements are not always usable.vstinner commentedon Jun 16, 2022
So we should make them usable in more cases :-)
Even without
with:, the idea is to explicitly release resources, not rely on the GC for that. The objects are not finalized in a specific order by the GC. The exact order is really complicated and depends on too many things. So it's likely that your code will fall into many corner cases.ImportError: sys.meta_path is None, Python is likely shutting downis just one example of such issue that you can get if you rely on the GC.In PyPy, it expect that it will be even worse.
pitrou commentedon Jun 16, 2022
I mean that
withstatements only work if your lifetime is lexically-defined. It's not the case for more sophisticated use cases unfortunately.Revert 30935 defer bpo45162 to 312 (#94285)