[Python] Clean up ExtensionType.__reduce__

[AlenkaF]

Describe the enhancement requested

Currently __reduce__ method is defined on the ExtensionType class so that each subclass can be pickled and doesn't need a separate __reduce__ method (see #36170).

But this can be cleaned up. Instead of defining __reduce__ for ExtensionType we can define __reduce__ on BaseExtensionType and make use of the C++-level (de)serialization. See #35933 (comment).

Component(s)

Python

[jorisvandenbossche]

Quickly looking into this, but now I am unsure whether it's actually an easy change as I mentioned at #35933 (comment) to use self.ext_type.Deserialize(..).

To be able to call the C++ Deserialize method, we first need to instantiate the relevant C++ ExtensionType in the cython unpickle method. But how do you do that? When you have a python instance, you can get that with self.ext_type. But when unpickling, you don't have an instance of the specific extension type class yet, because that's exactly what you are unpickling.

For pyarrow.ExtensionType, __arrow_ext_deserialize__ is also a classmethod (i.e. doesn't require an instance), but the implementation here is in python and custom done by the user and can call the class constructor of itself as it sees fit. However, if we want to put similar generic logic on the BaseExtensionType class, there is no __init__ that we can call.

But maybe I am overlooking something cc @pitrou

[pitrou]

The way we would do it is like in IPC: serialize the extension name and the extension serialized form.
Then on deserialization (in C++):

1. first get the extension type from its name using GetExtensionType
2. then deserialize the actual instance using ExtensionType::Deserialize

[jorisvandenbossche]

Ah, yes of course, just serialize the name and get the instance from the registry based on the name

[jorisvandenbossche]

However, that then only works for registered extension types. I don't know if we are OK with that restriction. In a Python session you could also be working with an ExtensionType that hasn't been registered.

[pitrou]

That's a good point, I'm not sure which is better here.