Variable sized base types

[mbway]

This PR introduces the capability to inherit from classes where the exact layout/size is not known using the mechanism described in PEP 697. This allows inheriting from type in order to create metaclasses. These metaclasses cannot yet be used with pyo3 classes (i.e. #[pyclass(metaclass=Foo)]) but this should be a possibility in future. It may also be possible in future for pyo3 classes to extend builtin classes using the limited API and to potentially extend imported classes (eg enum.EnumType).

This PR supersedes my first attempt at creating metaclasses: #4621. Previously I was using the normal class layout which relies on the exact layout of every base class to be known. For type the base structure is ffi::PyHeapTypeObject. The contents of this structure are intended to be private (even for the 'unlimited' API) and so an alternative mechanism is required (PEP 697).

Instead of nesting each base class at the beginning of the derived class in memory, objects created with PEP 697 only require knowledge of how much additional memory the derived class requires and the address of this section of memory is accessed via PyObject_GetTypeData. The PEP 697 mechanism could potentially be used to inherit from other builtins when only the 'limited' API is available however many of these builtins also store items (eg list, set) which introduces more complications, so this PR focuses on inheriting from PyType only.

This PR is a proof of concept but I am sure that some of the decisions I made aren't the best so feedback is welcome. The current design is as follows:

Before this PR the only possible layout was the 'static' layout where the size of the base class is known. This layout was described by PyClassObject<T> (now renamed to PyStaticClassObject). This PR introduces hides the details of the static layout behind a trait InternalPyClassObjectLayout so that other layouts are possible.

Before this PR: PyClassObject<T> was used to obtain the exact layout of T: PyClassImpl assuming that layout is the 'static' layout. This PR introduces PyClassImpl::Layout so that a pyclass declares its layout. The pyclass uses PyTypeInfo::Layout to set T::Layout. Each base type declares either PyTypeInfo::Layout = PyStaticClassObject or PyTypeInfo::Layout = PyVariableClassObject.

Potential Improvements

Some questions still remain:

• how best to handle initialisation (see below)
  • using the __init__ approach, how to initialise super classes?
  • using the __init__ approach, how to restrict return value to () or PyResult<()>?
• how to handle subclassing variable sized pyclasses
  • I tried writing some unit tests that inherited from type -> MyMetaclass -> MySubMetaclass but I think there are some outstanding problems. I also am not sure how to prevent users from doing this so I have left it for now.
    • problems encountered: setattr works without using #[pyclass(dict)] but the set values aren't accessible in the subclass
    • values are not as expected after initialisation, perhaps only the subclass data is getting initialised?
• for some reason using setattr on a class extending type does not fail, unlike on a class extending object. I'm not sure if this means type has __dict__ already?

Initialisation

I'm not sure what the best way is to handle initialisation. tp_new cannot be used with metaclasses (source) however the best semantics of __init__ aren't clear. For now I am requiring that metaclasses implement Default which allows the struct to be initialised before __init__ is called (but this is a hack and may have issues properly initialising superclasses). There are runtime checks when a class is constructed that metaclasses do not define #[new] and do define __init__. This should be sufficient to ensure that users cannot create a struct with uninitialized memory.

An alternative to using __init__ could be to repurpose #[new] to wrap it and assign it to tp_init instead in the case of a metaclass but that would be more complex to implement.

[mbway]

I didn't realise I was going to hit the issue that the minimum supported rust version of pyo3 (1.63) doesn't support generic associated types which I'm using for type Layout<T: PyClassImpl>: InternalPyClassObjectLayout<T>;.

Rust 1.65 (the first with GAT support) was released on 2022-11-03 and python 3.12 (the first version to allow extending variable sized base classes) was released on 2023-10-02.

I'm going to bump the MSRV to 1.65 but I'm open to other suggestions that avoid requiring GATs. I checked the MSRV requirements according to Contributing.md and from what I can see Debian stable ships 1.63 so this is going to be a problem.

Debian bookworm is supported until June 2026 which seems like a long time to wait...

[mejrs]

I'm going to bump the MSRV to 1.65

You can't do that unfortunately. Our msrv policy is at https://github.com/PyO3/pyo3/blob/main/Contributing.md#python-and-rust-version-support-policy

What you can do is to feature gate this and only enable the functionality on Rust 1.65 and up, or figure out a way to do it without gats.

[mbway]

What you can do is to feature gate this and only enable the functionality on Rust 1.65 and up

That sounds good to me. Do you mean introduce a feature like variable_base_types that has a different MSRV? How would I go about doing that?

Would it be sufficient to introduce the feature and have it disabled by default?

[mejrs]

A feature is one way to do it, a cfg that is automatically enabled when the build script detects it's on rust 1.65+ is another method. Both have pros and cons as far as user experience goes, so it's best to have a solution that doesn't need them. The layout code is also reasonably complex, so I'd rather not mix conditional compilation into it if possible.

[mbway]

The design currently revolves around the #[pyclass] macro being able to fill out the Layout entry like so

type Layout = <#cls as #pyo3_path::PyTypeInfo>::Layout<Self>;

This way the layout is set generically in the base types like PyAny and PyType and inherited from there.

The ways I can currently see around this are:

1. continue using GAT and feature gate the functionality
   • when disabled, hard code to the static layout and catch any discrepancy at construction? Better designs are probably possible but this would be simple to implement.
2. use a mechanism like PyClass::Frozen and instead of T::Layout have PyClassLayout<T> be a single struct containing the functionality that selects between implementations at runtime by observing T::VariableSized::VALUE. This of course has some runtime overhead.
3. use some trait magic to have a marker trait like IsStaticType that is inherited from the base type and again there is a single PyClassLayout<T> that implements the functionality but the functionality is selected at compile time using the marker traits. Unfortunately I'm not sure how to achieve this. Below is my attempt
   • related to this approach, have a single marker IsVariableType and an implementation with and without this marker. Unfortunately this requires specialisation which isn't stable yet

I'm going to implement (1) for now because the steps seem clearest, so this PR can be in a state where it could hypothetically be merged, then I'm open to suggestions if someone finds a way around the limitations I'm running into.

Attempt 1

The problem here is that the static and variable markers aren't known to be disjoint, so they conflict.

trait IsStaticType {}
trait IsVariableType {}

struct PyAny;
impl IsStaticType for PyAny {}
struct PyType;
impl IsVariableType for PyType {}

struct Layout<T> { _data: PhantomData<T> }

trait LayoutMethods {
    fn access_data();
}

impl<T: IsStaticType> LayoutMethods for Layout<T> {
    fn access_data() {
        println!("hi from static");
    }
}

impl<T: IsVariableType> LayoutMethods for Layout<T> {  // ERROR: conflicting implementations
    fn access_data() {
        println!("hi from variable");
    }
}

Attempt 2

The problem here is that <MyClass as LayoutMethods>::access_data() panics, and while <MyClass as StaticLayout>::access_data() compiles and <MyClass as VariableLayout>::access_data() does not, it doesn't solve the problem of being able to use the layout without specifying which it is.

trait LayoutMethods {
    fn access_data() {
        panic!("unknown layout")
    }
}

impl<T: PyClassImpl> LayoutMethods for T {}

trait StaticLayout: Sized + 'static + LayoutMethods {
    fn access_data() {
        println!("hi from static");
    }
}
impl<T, B> StaticLayout for T
where
    T: PyClassImpl<BaseType = B> + 'static,
    B: StaticLayout,
{ }

trait VariableLayout: Sized + 'static + LayoutMethods {
    fn access_data() {
        println!("hi from variable");
    }
}
impl<T, B> VariableLayout for T
where
    T: PyClassImpl<BaseType = B> + 'static,
    B: VariableLayout,
{ }

trait PyClassImpl { type BaseType; }

struct PyAny;
impl LayoutMethods for PyAny {}
impl StaticLayout for PyAny {}

struct PyType;
impl LayoutMethods for PyType {}
impl VariableLayout for PyType {}

struct MyClass {}
impl PyClassImpl for MyClass {
    type BaseType = PyAny;
}

[mejrs]

I wonder what kind of overhead 2 has. If it's barely noticeable I think that would be my preference. We can switch to gats the next time we bump msrv.

[mbway]

Ok. I wasn't sure what kind of trade-offs the project is looking for. I imagine querying the layout is a 'hot' part of the code as it's needed whenever converting to and from python objects. I've almost finished the feature gating implementation so I could post that as a separate branch for comparison. In my opinion the feature gating doesn't add too much complexity to the internals over what is there now but it would be potentially more annoying for users.

[mbway]

here is the branch with the feature gated implementation: https://github.com/mbway/pyo3/tree/variable_sized_base_types_feature_gated

the feature gating is implemented here: e591314

I will implement the (2) approach here hopefully some time next week unless it is decided that (1) is preferred.

[mejrs]

It would be nice to get @davidhewitt's thoughts about all of this, as he's much more experienced with this part of the code base than I am.

As for 3), perhaps take a look at #4254. It uses those kind of tricks to either compute offsets at compile time or at runtime.

[davidhewitt]

Hey, thanks for this PR and I'm very pumped; I've been waiting for something like this for long time, never quite got around to doing it myself.

Agree we can't use GATs yet, I think it might be another year before we bump MSRV (waiting on the next Debian stable release).

If 2 is switching on a const bool it feels like it should be possible for the optimizer to reduce it down, so I'd suggest proceeding that way for now if it feels simple enough.

As for a deeper review - I'm trying to merge the last couple pieces for a very overdue 0.23 release, so I think this might take me a couple days before I can read in depth. And then I'm very keen to help out here to get this PR into 0.24!

[mbway]

good to hear you are enthusiastic about this feature 🙂

please hold off reading in its current state as I work to rewrite avoiding GATs. I think it could end up being simpler than my current implementation but may take some time to get there.