RFC: Implement Multi-Phase Module Initialization as per PEP 489

[Aequitosh]

RFC: Multi-Phase Module Initialization as per PEP 489

This PR implements multi-phase initialization in PyO3 as a drop-in replacement for single phase initialization. While it still needs some polish here and there, the implementation is mostly complete.

What's still left to do:

• Figure out how to add submodules again - needs some kind of support for creating m_slots or perhaps an API around creating PyModuleDefs
  • This works now, thanks to a mechanism that makes it possible to (de-facto) statically allocate m_slots at runtime.
• Reliably get a Py<PyModule> from a *mut ffi::PyModuleDef
  • Is this even possible? If not, we'll require some API breaking, I fear
• Improve error handling - currently panics if calling the module setup function (#ident(#(#module_args),*) in proc macros) fails
• Implement multi-phase initialization for PyModule::new_bound
  • I wonder if this is actually necessary, though.
• Address remaining TODOs
• Rephrase commit messages / perhaps split up commits
  • The messages could IMO be somewhat more detailed, elaborating on the reasoning behind certain changes / additions

Please let me know what you think! I'm grateful for any feedback! :)

Specifically, I'd be very happy to receive insight / comments on the TODOs I've added here and there.

[Aequitosh]

Upgraded this PR from a draft to an RFC. The description was also updated to reflect that.

[a-reich]

Just curious, any progress getting this reviewed?

[Aequitosh]

Just curious, any progress getting this reviewed?

I'm planning to rebase this on main as soon as I got time; perhaps then someone's gonna have a look.

[davidhewitt]

Yes, I had a lot to think about with freethreading and also our changes for IntoPyObject. I want to come back to this after releasing 0.23.

[Aequitosh]

Yes, I had a lot to think about with freethreading and also our changes for IntoPyObject. I want to come back to this after releasing 0.23.

Sounds good! I've been quite busy myself, so it's all good; seems like the timing is just right.

[Sunnatillo]

Hi @Aequitosh. Thank you for working on this issue.

How can I help to get this forward?

[epuertat]

@Aequitosh I'm just curious, as a workaround, would cleaning-up/freeing a PyO3 module (m_clear, m_free callbacks) would resolve this issue?

Something like?:

from cryptography.fernet import Fernet   # a version that relies on this version of PyO3

key = Fernet.generate_key()

del cryptography
del sys.modules['cryptography']
gc.collect()

If that fully removes all module state, we could have a context manager like this in sub-interpreters:

with _import('cryptography.fernet') as Fernet:
    key = Fernet.generate_key()

What do you think?

[Aequitosh]

Hi @Aequitosh. Thank you for working on this issue.

How can I help to get this forward?

Hey @Sunnatillo, thank you for offering your help! I'm actually intending on rebasing my branch on main today and then make a little write-up / plan on what's left to do. There you'll then find everything I'd need help with.

I really wanna get this ball rolling again, so I'm grateful for any help.

I'm just curious, as a workaround, would cleaning-up/freeing a PyO3 module (m_clear, m_free callbacks) would resolve this issue?

Hi @epuertat! This unfortunately won't resolve things, because PyO3 modules are (as of time of writing) loaded using single-phase initialisation. The callbacks you mentioned are currently not used; there's no actual module state being managed at the moment, because native modules based on PyO3 are "global" (due to the fact of them using the older single-phase init mechanism). Glossing over some details here, but that's essentially what the situation is right now.

This PR aims to replace the single-phase init with the newer multi-phase initialisation, which would then (almost) automatically make PyO3-based Python modules sound when used with sub-interpreters. The remaining static data that PyO3 stores would also need to be moved to the per-module memory location in PyModuleDef.

So, unfortunately, your solution most likely won't work (or rather, it won't be safe).

Relevant reads, if you're curious:

• Python C-API - Module Objects
• PEP 489

---

With all that being said, expect some updates / changes to trickle in (hopefully) soon.

[epuertat]

Thanks a lot @Aequitosh for your quick & thorough reply!

[ngoldbaum]

I'm very interested in this because NumPy eventually needs to make a similar transition to multi-phase init and the whole thing is very confusing to me. I'd also like to update the CPython docs to use multi-phase init in the C extension tutorial: https://docs.python.org/3/extending/extending.html

[Sunnatillo]

@Aequitosh Are you still interested to continue this work?

[davidhewitt]

Inspired by this PR (and several before it) I have implemented the first step of multi-phase-init in #5142

This PR goes further in implementing module state traversal etc, so there are probably pieces remaining here to still explore if #5142 merges.