Python pickler unable to pickle object the native pickler can

[apmorton]

Bug report

Bug description:

import io
import pickle


class ZeroCopyByteArray(bytearray):
    def __reduce_ex__(self, protocol):
        return type(self), (pickle.PickleBuffer(self),), None


data = [
    ZeroCopyByteArray(),
    ZeroCopyByteArray(),
]

# this works
f = io.BytesIO()
pickler = pickle.Pickler(f, protocol=5)
pickler.dump(data)

# this hits memo assertion
f = io.BytesIO()
pickler = pickle._Pickler(f, protocol=5)
pickler.dump(data)

Traceback (most recent call last):
  File "bug.py", line 24, in <module>
    pickler.dump(data)
  File "/pickle.py", line 487, in dump
    self.save(obj)
  File "/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/pickle.py", line 932, in save_list
    self._batch_appends(obj)
  File "/pickle.py", line 956, in _batch_appends
    save(x)
  File "/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/pickle.py", line 692, in save_reduce
    save(args)
  File "/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/pickle.py", line 887, in save_tuple
    save(element)
  File "/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/pickle.py", line 842, in save_picklebuffer
    self.save_bytearray(m.tobytes())
  File "/pickle.py", line 821, in save_bytearray
    self.memoize(obj)
  File "/pickle.py", line 508, in memoize
    assert id(obj) not in self.memo
           ^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

CPython versions tested on:

3.11

Operating systems tested on:

Linux

Linked PRs

• gh-120380: fix Python implementation of pickle.Pickler for bytes and bytearray objects in protocol version 5. #120422
• [3.13] gh-120380: fix Python implementation of pickle.Pickler for bytes and bytearray objects in protocol version 5. (GH-120422) #120832
• [3.12] gh-120380: fix Python implementation of pickle.Pickler for bytes and bytearray objects in protocol version 5. (GH-120422) #120833

[picnixz]

I'd say it's not a bug because pickle._Pickler is private and should not be used directly. Why do you need the private one?

[aisk]

I'd say it's not a bug because pickle._Pickler is private and should not be used directly. Why do you need the private one?

pickle.Pickler may be provided by pickle._Pickler if the native extension _pickle is not presented:

cpython/Lib/pickle.py

Lines 1770 to 1784 in 42b25dd

	try:
	from _pickle import (
	PickleError,
	PicklingError,
	UnpicklingError,
	Pickler,
	Unpickler,
	dump,
	dumps,
	load,
	loads
	)
	except ImportError:
	Pickler, Unpickler = _Pickler, _Unpickler
	dump, dumps, load, loads = _dump, _dumps, _load, _loads

[picnixz]

Oh! yes, my bad! I'll hide my comment to reduce the noise!

[picnixz]

The ZeroCopyBytearray buffer appears to need the following implementation (it's actually in Lib/test/pickletester) but I confirmed that the issue remains but only if the buffer is empty (otherwise it's fine). The implementation is roughly

class ZeroCopyBytearray(bytearray):
    def __reduce_ex__(self, protocol):
        if protocol >= 5:
            return type(self)._reconstruct, (pickle.PickleBuffer(self),), None
        else:
            return type(self)._reconstruct, (bytes(self),)

    @classmethod
    def _reconstruct(cls, obj):
        with memoryview(obj) as m:
            obj = m.obj
            if type(obj) is cls:
                # Zero-copy
                return obj
            else:
                return cls(obj)

So, the __reduce_ex__ method makes more things than what you provided in your example, but it does not fix the issue at all.

EDIT: I'm investigating the issue in picnixz@a8c1464

[apmorton]

Correct, I minimized the example to only the relevant parts for reproduction.

The issue only happens if you use protocol 5 and end up with two objects having different IDs produce a PickleBuffer with the same underlying buffer ID.

[picnixz]

Yes, the issue is that this specific test case is not covered (I've added one and I'll try to fix it), which is why we did not see the issue (by specific case, I mean the two different instances that are empty, not that it still works if I use the same non-empty buffer values).