Crash after error in PickleBuffer

[serhiy-storchaka]

Crash report

The following script crashes at finalization stage:

import unittest
import pickle

class Test(unittest.TestCase):
    def test_non_continuous_buffer(self):
        pb = pickle.PickleBuffer(memoryview(b"foobar")[::2])
        pb.raw()

unittest.main()

Output:

E
======================================================================
ERROR: test_non_continuous_buffer (__main__.Test.test_non_continuous_buffer)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/test_non_continuous_buffer.py", line 7, in test_non_continuous_buffer
    pb.raw()
    ~~~~~~^^
BufferError: cannot extract raw buffer from non-contiguous buffer

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (errors=1)
python: Objects/memoryobject.c:125: mbuf_dealloc: Assertion `self->exports == 0' failed.
Aborted (core dumped)

I cannot yet reproduce the crash without unittest, perhaps it is related to some reference loop.

The crash is only happens in 3.12 an newer. In 3.8-3.11 it just raises an exception (PickleBuffer was added in 3.8).

cc @pitrou

Linked PRs

• gh-122306: Fix circular reference issue between MemoryView and PickleBuffer during garbage collection #123400

[devdanzin]

Try:

import pickle
import atexit

m = memoryview(b"foobar")[::2]
pb = pickle.PickleBuffer(m)
def p(): pb.raw()
atexit.register(m.bit_count)

[devdanzin]

Better:

import pickle

m = memoryview(b"foobar")[::2]
pb = pickle.PickleBuffer(m)
def p(): m.bit_length()
m.invalid

[marko1616]

The following analysis report will explain the bug by inserting comments into the log.

Main Causes

1. Creating a circular reference through a constructed exception, preventing garbage collection of PickleBuffer and memory objects
2. Calling gc.collect()
3. During the collection of the memory object, PyMemoryViewObject->mbuf is cleared for the first time (mbuf is set to a null pointer)
4. When collecting PickleBuffer, memory_dealloc is called again through picklebuf_clear, at which point an exception occurs because mbuf was set to a null pointer in step 3

Next

I'm trying to find a method fix this this days.

Bug reproduction(tested on mainline)

import gc, pickle

mv = memoryview(b"test")
error = None

def func():
    try:
        pb = pickle.PickleBuffer(mv)
        raise Exception("test")
    except Exception as e:
        # Ensure `error` references the exception
        error = e
    finally:
        print(f"pb:{pb}\nmv:{mv}")

# Cyclic References Explanation:
# An exception creates a traceback object referencing the frame.
# The frame references local variables (`error`, `pb`).
# The exception object references the traceback.
# The `error` variable references the exception.

func()
del mv
gc.collect() # Crash here

GDB Debugging Details

╭─xxx at xxx in ~/path/to/repos/cpython on main✔
╰─± gdb ./python 
GNU gdb (Debian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./python...
warning: File "/home/marko1616/Documents/repos/cpython/python-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
        add-auto-load-safe-path /home/marko1616/Documents/repos/cpython/python-gdb.py
line to your configuration file "/home/marko1616/.config/gdb/gdbinit".
To completely disable this security protection add
        set auto-load safe-path /
line to your configuration file "/home/marko1616/.config/gdb/gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
        info "(gdb)Auto-loading safe path"
(gdb) b _memory_release
Breakpoint 1 at 0x14a97e: _memory_release. (3 locations)
(gdb) disable 1
(gdb) run
Starting program: /home/marko1616/Documents/repos/cpython/python 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Python 3.14.0a0 (heads/main:54a05a46002, Aug 27 2024, 11:19:04) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
[Detaching after vfork from child process 302825]
>>> import gc, pickle
... 
... mv = memoryview(b"test")
... error = None
... 
... def func():
...     try:
...         pb = pickle.PickleBuffer(mv)
...         raise Exception("test")
...     except Exception as e:
...         # Ensure `error` references the exception
...         error = e
...     finally:
...         print(f"pb:{pb}\nmv:{mv}")
... 
... # Cyclic References Explanation:
... # An exception creates a traceback object referencing the frame.
... # The frame references local variables (`error`, `pb`).
... # The exception object references the traceback.
... # The `error` variable references the exception.
... 
... func()
... input(f"del mv {mv}")
... del mv
... input("before gc")
... gc.collect() # Crash here
... 
pb:<pickle.PickleBuffer object at 0x7ffff744f740>
mv:<memory at 0x7ffff76cbf40>
del mv <memory at 0x7ffff76cbf40>
before gc
         Program received signal SIGINT, Interrupt.
                                                   0x00007ffff7de419d in __GI___libc_read (fd=fd@entry=3, buf=buf@entry=0x555555de78e0, nbytes=nbytes@entry=10000) at ../sysdeps/unix/sysv/linux/read.c:26
26      ../sysdeps/unix/sysv/linux/read.c: No such file or directory.
(gdb) watch (*(PyMemoryViewObject *)0x7ffff76cbf40)->mbuf
Hardware watchpoint 2: (*(PyMemoryViewObject *)0x7ffff76cbf40)->mbuf
(gdb) c
Continuing.


Hardware watchpoint 2: (*(PyMemoryViewObject *)0x7ffff76cbf40)->mbuf

Old value = (_PyManagedBufferObject *) 0x7ffff744e3c0
New value = (_PyManagedBufferObject *) 0x0
memory_clear (_self=0x7ffff76cbf40) at Objects/memoryobject.c:1168
1168        Py_CLEAR(self->mbuf);
(gdb) bt
#0  memory_clear (_self=0x7ffff76cbf40) at Objects/memoryobject.c:1168
#1  0x00005555557c5ec4 in delete_garbage (old=0x555555ae2930 <_PyRuntime+183280>, collectable=0x7fffffffd740, gcstate=0x555555ae28e8 <_PyRuntime+183208>, 
    tstate=0x555555b17858 <_PyRuntime+400152>) at Python/gc.c:1119
...
--Type <RET> for more, q to quit, c to continue without paging--q
Quit
(gdb) c
Continuing.
Exception ignored in tp_clear of memoryview:
Traceback (most recent call last):
  File "<python-input-0>", line 26, in <module>
BufferError: memoryview has 1 exported buffer

Program received signal SIGSEGV, Segmentation fault.
0x000055555569e9ad in _memory_release (self=0x7ffff76cbf40) at Objects/memoryobject.c:1111
1111            if (--self->mbuf->exports == 0)
(gdb) bt
#0  0x000055555569e9ad in _memory_release (self=0x7ffff76cbf40) at Objects/memoryobject.c:1111
#1  memory_dealloc (_self=0x7ffff76cbf40) at Objects/memoryobject.c:1148
#2  0x00005555556c999d in picklebuf_clear (self=<optimized out>) at Objects/picklebufobject.c:103
#3  0x00005555557c5ec4 in delete_garbage (old=0x555555ae2930 <_PyRuntime+183280>, collectable=0x7fffffffd740, gcstate=0x555555ae28e8 <_PyRuntime+183208>, 
...
(gdb) 

[marko1616]

cc @serhiy-storchaka

[serhiy-storchaka]

This looks like an example of the hypothetical issue discussed in #77894. The solution was proposed there, but we did not have any reliable way to test it. Now we can, using pickle.PickleBuffer. And other way -- using memoryview.__getbuffer__(). Both ways were not available when the original issue was opened.

I'm closing this issue as a duplicate of #77894.