Sharing pyclasses between multiple Rust packages

[davidhewitt]

I've seen several questions about re-using #[pyclass] types between multiple Rust packages, for example on gitter, stack overflow and most recently in #1418.

This issue is indended to be a place for the discussion to centralize regarding understanding the difficulties in this and to brainstorm solutions / API designs which we can add to PyO3.

Below is my write-up of the issue. Note that I haven't tried to any of this myself; this is just where I think the difficulties are. I could be totally wrong - please use this thread to share evidence, ask questions, and test ideas.

---

Let's call package A the "original" package, which defines a #[pyclass] MyClass. Package B is a child crate which makes use of use a::MyClass in Rust code, to re-use the pyclass.

The key issue is that #[pyclass] stores the pyclass type object in static storage. This means that (if Rust's usual rlib linkage is used) packages A and B will have their own copies of the MyClass type object, and Python will think that they're actually different types coming from the two packages.

I think the core of the solution to this is to make B link against the package A C shared library - the same .so file which Python usess to load package A. Then B should be able to re-use the pyclass from A.

This comes with a few steps:

• A will need to export additional symbols which B can use. This probably includes at a minimum the MyClass static type object, and probably functions for converting PyAny <-> MyClass.
• B will need to use A as a "C" dependency which it links to using an extern "C" block, where it imports symbols from A.
• I'm unsure if B will be able to use a::MyClass; at all - it may need to have a duplicate definition which uses the exported symbols internally.

Looking at that, it might mean that we'd eventually have pyclass_export! and pyclass_import! macros in PyO3 to help with this.

The above is all a poorly-worded brain dump of what might be a solution. I'm really not sure - anyone who is trying to do this, please comment, ask questions, provide error logs, and we can figure this out together.

[milesgranger]

After getting great help from you in #1535 I've hit this issue. The additional feature I want to add would still make use of multiple objects defined in the parent package. But alas, as you say, Python thinks they aren't the same object and fails to convert the object into an object it actually already is. 😅

[davidhewitt]

Yeah; unfortunately unlike #1535 this one's hard and I haven't thought more than the words above as to how it might work. I'm also not planning to design a solution any time soon as there's a number of big pieces of work in the pyO3 core which I think are higher priority.

If this is something you need sooner rather than later, I suggest having a go at this yourself based on my sketch above and asking questions liberally on this issue for anything which you need help understanding. It should be possible to cobble something together for this without needing to change the current pyO3 crate itself. No guarantees it won't be painful though!

[milesgranger]

Thanks, it is helpful all the same. I just might take you up on that offer to cobble something together. Albeit, I've started paternity leave so won't have anything up for some time most likely. We'll see how things go. Thanks again. 👍

[davidhewitt]

Albeit, I've started paternity leave so won't have anything up for some time most likely. We'll see how things go.

No rush & congratulations!

[milesgranger]

While dabbling with this earlier today, I hit #1193; thinking entering pyclasses into a capsule could be a good avenue for sharing structs/functions between crates. Is it not an absurd approach for when that issue is solved?

[davidhewitt]

Possibly - why were you thinking of using capsules rather than using Python APIs to import the Python objects directly?

Or were you thinking of putting function pointers to supporting Rust functions in the capsules? I believe numpy might do something like that.

[milesgranger]

I was indeed taking inspiration from pyo3 numpy and this description about using capsules seemed to reinforce my plan.

... using Python APIs to import the Python objects directly?

Is there any benefit/loss from doing it this way vs capsules? Naively, it seemed like there would be some overhead going through Python to get an object/pyclass vs the capsule workflow.

[davidhewitt]

Yeah, especially if you're just exporting Rust functions which you don't want to wrap in Python at all, then capsules are very much a suitable tool.