Calling `ShareableList.count` in threads aborts: `Assertion 'self->exports == 0' failed`

[devdanzin]

Crash report

What happened?

It's possible to abort the interpreter by calling multiprocessing.shared_memory.ShareableList.count in threads with PYTHON_GIL=0 in a debug build:

import gc
import multiprocessing.shared_memory
from threading import Thread

obj = multiprocessing.shared_memory.ShareableList("Uq..SeDAmB+EBrkLl.SG.Z+Z.ZdsV..wT+zLxKwdN\b")

for x in range(10):
    Thread(target=obj.count, args=(1,)).start()

del obj
gc.collect()

Result:

Exception ignored in: <function SharedMemory.__del__ at 0x200006bbfb0>
Traceback (most recent call last):
  File "/home/danzin/projects/mycpython/Lib/multiprocessing/shared_memory.py", line 189, in __del__
    self.close()
  File "/home/danzin/projects/mycpython/Lib/multiprocessing/shared_memory.py", line 229, in close
    self._buf.release()
BufferError: memoryview has 2 exported buffers
python: Objects/memoryobject.c:1143: memory_dealloc: Assertion `self->exports == 0' failed.
Aborted

Found using fusil by @vstinner.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.14.0a2+ experimental free-threading build (heads/main-dirty:c9b399fbdb0, Nov 19 2024, 20:12:48) [GCC 11.4.0]

Linked PRs

• gh-127085: Fix memoryview->exports race condition. #127412
• gh-127085: Add a test skip if multiprocessing isn't available #128019

[LindaSummer]

Hi @devdanzin ,

I'm a newbie of CPython and want to have a try on this issue.

Could it be assigned to me?😊

Best Regards,
Edward

[JelleZijlstra]

@LindaSummer you don't need to be assigned an issue to work on it. If you're interested in this issue, I suggest you read through the code where the crash happens to understand it. If you feel you found something useful, you can post on this issue with your findings. If you find a possible solution, you can open a PR with your fix.

[LindaSummer]

@LindaSummer you don't need to be assigned an issue to work on it. If you're interested in this issue, I suggest you read through the code where the crash happens to understand it. If you feel you found something useful, you can post on this issue with your findings. If you find a possible solution, you can open a PR with your fix.

Hi @JelleZijlstra ,

Got it! Thanks very much for your warm guidance! 😊

I will try to go through the code and search for the root cause.

Best Regards,
Edward

[LindaSummer]

Hi @JelleZijlstra ,

Here are some updates during these days.

1. I constructed a new case to reproduce this issue. This issue should have no relation to gc.

import multiprocessing.shared_memory
from threading import Thread

def test_memoryview_buffer():
    import struct
    class TA:
        data = bytearray("Uq..SeDAmB+EBrkLl.SG.Z+Z.ZdsV..wT+zLxKwdN\b", encoding="utf8")
        _fmt = "b"*len(data)
        shm = multiprocessing.shared_memory.SharedMemory(create=True, size=struct.calcsize(_fmt))
        def __init__(self):
            struct.pack_into(self._fmt, self.shm.buf, 0, *self.data)
        def count(self, t):
            res = 0
            for i in range(len(self.data)):
                (v,) = struct.unpack_from('b', self.shm.buf, i)
                if i == v:
                    res += 1
            return res
    obj = TA()
    for x in range(2000):
        Thread(target=obj.count, args=(1,)).start()
    del obj

test_memoryview_buffer()

2. I print the ignored exception details and find below.

Exception ignored in: <function SharedMemory.__del__ at 0x20000c0efd0>
Traceback (most recent call last):
  File "/workspaces/cpython/Lib/multiprocessing/shared_memory.py", line 189, in __del__
    self.close()
  File "/workspaces/cpython/Lib/multiprocessing/shared_memory.py", line 230, in close
    self._buf.release()
BufferError: memoryview:  negative export count 
python.exe: Objects/memoryobject.c:1150: void memory_dealloc(PyObject *): Assertion `self->exports == 0' failed.

This should be inside the SharedMemory release function.

3. I use an bytearray memoryview and this issue doesn't happen again. I think it should be related to SharedMemory accessors.

4. I searched for the exports references and find it is incremented and decremented in file memory_getbuf and memory_releasebuf.

I think it should be a race condition issue on the shm.buf memoryview object.

I will try to learn more about the non-gil synchronization logic and try to find the root cause.

Please correct me if my analysis has any problems. 😊

Best Regards,
Edward