Inconsistent behaviour of multiprocessing.shared_memory

[Souradeep1995]

Bug report

Bug description:

An example code already exists in the documentation (see from here: The following example demonstrates a practical use of the SharedMemory class with NumPy arrays, accessing the same numpy.ndarray from two distinct Python shells).

The code executes fine as expected and mentioned there until one closes the terminals which report warnings.

When the second terminal is closed: /home/souradeep/miniconda/envs/py39/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '

and when the first terminal is closed: /home/souradeep/miniconda/envs/py39/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/trial': [Errno 2] No such file or directory: '/trial'

This can have implications when one tries to use the feature somewhere in an application (see an example).

CPython versions tested on:

3.9

Operating systems tested on:

Linux

Linked PRs

• gh-116849: restrict shared memory registration on creation #119126

[Souradeep1995]

Any news on this?

[zetadin]

I just ran into a similar issue with sharing memory with forked process under WSL(ubuntu) with python 3.8.
It also appears in this discussion.

After some debugging, here is what I found out:

Whenever SharedMemory.init() is called, it registers the SharedMemory._name with the resource_tracker:
resource_tracker.register(self._name, "shared_memory")

This happens twice for the same name, once during memory creation in the first terminal and once during attaching of existing memory in the second terminal. Resource_tracker does not check for duplicates and both times the same name is added to the cache!

When we unlink() the memory, it removes the mmap and removes one instance of the name from the resource_tracker's cache. The other instance remains in place.

When each terminal process ends, resource_tracker tries to clean up any remaining registered named resources. So if the second terminal ends first, the resource_tracker warns about the still registered resource and removes its mmap. When the first terminal is about to end, unlink() is called on the memory from there, but the corresponding mmap is already gone.

Suggested permanent fix:
Only register SharedMemory with the resource_tracker when it is created, not when attaching existing memory.

            if self._track and create:
                resource_tracker.register(self._name, "shared_memory")

Current workaround:
Manually unregister SharedMemory in processes where it will not be unlinked.

# /trial created in another process
existing_shm = shared_memory.SharedMemory(name='trial')
...
existing_shm.close()
from multiprocessing.resource_tracker import unregister
unregister(existing_shm._name, "shared_memory")
# /trial unlinked in another process

Full working equivalent of the example in the docs.