RFC: Implement Multi-Phase Module Initialization as per PEP 489 #4162
+561
−74
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5 tasks
epuertat
added a commit
to rhcs-dashboard/ceph
that referenced
this pull request
Jun 7, 2024
The latest versions of pyO3 (a Python binding for Rust) explicitly added a check to detect multiple imports. In subinterpreter environments, this leads to an ImportError: "PyO3 modules may only be initialized once per interpreter process" (it has been recenlty replaced with a more specific: "PyO3 modules may only be initialized once per interpreter process". This is only a workaround while the root cause is fixed (see PyO3/pyo3#4162). Fixes: https://tracker.ceph.com/issues/64213 Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
14 tasks
Aequitosh
force-pushed
the
pyo3-pep-489-multi-phase-init
branch
from
02eb845
to
909b49f
Compare
This module contains `ModuleState`, a struct used to represent state that lies on the per-module memory region of a Python module, as well as various exported functions that are used to integrate `ModuleState` with the Python interpreter. This commit does not implement any further functionality (such as actually representing any kind of state) and is instead more of a blank slate for further additions. Signed-off-by: Max R. Carrara <aequitosh@gmail.com>
These helpers and wrapper structs ensure that any slots and function pointers used during multi-phase initialization stay allocated throughout the remaining lifetime of the program. Signed-off-by: Max R. Carrara <aequitosh@gmail.com>
This commit adds experimental but fully functional support for multi-phase module initialization as specified in PEP 489. Note that this commit serves as a demonstration & basis for further improvements only; many tests have therefore not been adapted correspondingly yet. Signed-off-by: Max R. Carrara <aequitosh@gmail.com>
Signed-off-by: Max R. Carrara <aequitosh@gmail.com>
Aequitosh
force-pushed
the
pyo3-pep-489-multi-phase-init
branch
from
909b49f
to
a1beab4
Compare
Aequitosh
changed the title
|
Upgraded this PR from a draft to an RFC. The description was also updated to reflect that. |
|
Just curious, any progress getting this reviewed? |
I'm planning to rebase this on main as soon as I got time; perhaps then someone's gonna have a look. |
|
Yes, I had a lot to think about with freethreading and also our changes for |
Sounds good! I've been quite busy myself, so it's all good; seems like the timing is just right. |
pecastro
pushed a commit
to pecastro/ceph
that referenced
this pull request
Nov 19, 2024
The latest versions of pyO3 (a Python binding for Rust) explicitly added a check to detect multiple imports. In subinterpreter environments, this leads to an ImportError: "PyO3 modules may only be initialized once per interpreter process" (it has been recenlty replaced with a more specific: "PyO3 modules may only be initialized once per interpreter process". This is only a workaround while the root cause is fixed (see PyO3/pyo3#4162). Fixes: https://tracker.ceph.com/issues/64213 Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
|
Hi @Aequitosh. Thank you for working on this issue. How can I help to get this forward? |
|
@Aequitosh I'm just curious, as a workaround, would cleaning-up/freeing a PyO3 module ( Something like?: from cryptography.fernet import Fernet # a version that relies on this version of PyO3
key = Fernet.generate_key()
del cryptography
del sys.modules['cryptography']
gc.collect()If that fully removes all module state, we could have a context manager like this in sub-interpreters: with _import('cryptography.fernet') as Fernet:
key = Fernet.generate_key()What do you think? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
RFC: Multi-Phase Module Initialization as per PEP 489
This PR implements multi-phase initialization in PyO3 as a drop-in replacement for single phase initialization. While it still needs some polish here and there, the implementation is mostly complete.
What's still left to do:
m_slotsor perhaps an API around creatingPyModuleDefsm_slotsat runtime.Py<PyModule>from a*mut ffi::PyModuleDef#ident(#(#module_args),*)in proc macros) failsPyModule::new_boundPlease let me know what you think! I'm grateful for any feedback! :)
Specifically, I'd be very happy to receive insight / comments on the TODOs I've added here and there.