Isolate Stdlib Extension Modules

[ericsnowcurrently]

See PEP 687.

Currently most stdlib extension have been ported to multi-phase init. There are still a number of them to be ported, almost entirely non-builtin modules. Also, some that have already been ported still have global state that needs to be fixed.

(This is part of the effort to finish isolating multiple interpreters from each other. See gh-100227.)

High-Level Info

How to isolate modules: https://docs.python.org/3/howto/isolating-extensions.html (AKA PEP 630).

The full list of modules that need porting can be found with: ...

The full list of remaining (unsupported) global variables is:

• builtin extensions: https://github.com/python/cpython/blob/main/Tools/c-analyzer/cpython/globals-to-fix.tsv#L308-L335
• non-builtin extensions: https://github.com/python/cpython/blob/main/Tools/c-analyzer/cpython/globals-to-fix.tsv#L338-L551

A full analysis of the modules may be found at the bottom of this post.

(other info)

Previous Work

• Convert static types to heap types: use PyType_FromSpec() #84258
• _xxsubinterpreters [issue] [PR]
• _testinternalcapi [issue] [PR]
• ...

Related Links

• [C API] Convert a few stdlib extensions to the limited C API (PEP 384) #85283
• https://discuss.python.org/t/a-new-c-api-for-extensions-that-need-runtime-global-locks/20668
• https://discuss.python.org/t/how-to-share-module-state-among-multiple-instances-of-an-extension-module/20663

TODO

Here is the list of modules that need attention, in a rough, best-effort priority order. Additional details (e.g. if there is an issue and/or PR) is found in the analysis table at the bottom.

• builtins (high priority)
  • Isolate the _io extension module #101819
  • isolate _collections
  • Isolate the tracemalloc Module Between Interpreters #101520
  • Isolate the faulthandler Module Between Interpreters #101509
• essential (higher priority)
  • port _socket
  • port readline (leave as is: only available in main interp)
  • isolate _ssl
  • port _pickle
  • Isolate the _datetime extension module #117398
  • isolate _asyncio
  • Isolate the _decimal extension module #106078
  • isolate array
  • port _ctypes
    • Convert _ctypes static types (and the "meta class") to heap types #114314
    • Convert _ctypes extension module to multi-phase init #117142
  • port _tkinter (leave as is: only available in main interp)
  • Isolate _multibytecodec #103583
  • Isolate the _curses extension module #101714
  • isolate _curses_panel (leave as is: only available in main interp)
  • isolate _elementtree
  • isolate pyexpat
  • port winreg (Windows)
  • port msvcrt (Windows)
• non-essential (lower priority)
  • port winsound (Windows)
  • isolate _lsprof

The above does not include test modules. They don't need to be ported/isolated (except for a few which already have been).

---

Modules Analysis

module	builtin	Windows	PEP 594	issue	PR	ported	# static types	# other global objects	# other globals
_asyncio						yes		2
_collections	X			(???)	(branch)	yes	7
_ctypes						**NO**	37	6	4
_curses						**NO**	1	2	4
_curses_panel						yes			1
_datetime				gh-71587	gh-102995	**NO**	7	10	1
_decimal						**NO**	4	10	6
_elementtree						yes			1
_io	X			gh-101819	gh-101520	**NO**	5
_lsprof						yes			2
_multibytecodec						yes			23
_pickle				(???)	(yes)	**NO**	5
_socket						**NO**	1	2	3
_ssl				(???)	(branch)	yes			1
_tkinter						**NO**		8	9
_tracemalloc	X			gh-101520		**NO**		6	7
array						yes		1
faulthandler	X			gh-101509		yes		3+	~22+
msvcrt		Y				**NO**	???	???	???
pyexpat						yes			1
readline						**NO**			9
winreg		Y				**NO**	???	???	???
winsound		Y				**NO**	???	???	???

test/example modules

These can be ported/isolated but don't have to be. They are the lowest priority.

module	issue	PR	ported	# static types	# other global objects	# other globals
xxmodule				3	1
xxsubtype				2
xxlimited_35					2
...
...

Linked PRs

• gh-103092: Isolate _collections #103093
• gh-103092: Isolate _pickle #102982
• gh-103092: Isolate socket #103094
• gh-103092: Isolate _tkinter (WIP) #103108
• gh-103092: Isolate msvcrt #103248
• gh-103092: Isolate winsound #103249
• gh-103092: Isolate winreg #103250
• gh-103092: Isolate _decimal #103381
• gh-103583: Add codecs and maps to _codecs_* module state #103540
• gh-103092: Isolate ctypes #103612
• gh-103092: Isolate _ctypes, part 1 #103893
• gh-103092: Adapt _ctypes's PyCStgDict type to heap type #103932
• gh-103092: Port _ctypes.COMError to heap type #104020
• GH-103092: port _asyncio freelist to module state #104196
• GH-103092: isolate pyexpat  #104506
• GH-103092: isolate _elementtree #104561
• GH-103092: isolate _ssl #104725
• gh-103092: Make _elementtree module importable in sub-interpreters #113434
• gh-103092: Make pyexpat module importable in sub-interpreters #113555
• gh-103092: Convert some _ctypes metatypes to heap types #113620
• gh-103092: Port some _ctypes data types to heap types #113630
• gh-103092: Test _ctypes type hierarchy and features #113727
• gh-103092: Port some _ctypes data types to heap types (alt) #113774
• gh-103092: Ensure _ctypes.c static types are accessed via global state #113857
• gh-103092: isolate _lsprof #115130
• gh-103092: Adapt _lsprof methods to Argument Clinic #115242
• gh-103092: Add a mutex to make the random state of rotatingtree concurrent-safe #115301
• gh-103092: Prep curses module for multi-phase init #23091
• gh-103092: Support subinterpreters in _zstd #133674
• [3.14] gh-103092: Support subinterpreters in _zstd (GH-133674) #133695

[ericsnowcurrently]

Ideally we'll get most of this done for 3.12. FWIW, there isn't enough time if it's just me, so consider this a call out to whoever can help. 😄 Feel free to adjust the TODO list and (hidden) "analysis" table above. Also feel free to reach out to any others that might be interested in pitching in here. Thanks!!!

@erlend-aasland, @corona10, @kumaraditya303, etc.

[ericsnowcurrently]

@terryjreedy (for _tkinter)

[erlend-aasland]

I've got WIP branches for datetime, io, pickle, collections, and ssl. The three former are up as draft PRs.

(FYI, I have quite a bit of CPython time scheduled for next week.)

[ericsnowcurrently]

@markshannon

[erlend-aasland]

Regarding the freelists in _asyncio, Kumar and Guido had some thoughts over at #91375 (comment).

[erlend-aasland]

Elementtree was isolated/ported in gh-92123.

[erlend-aasland]

_lsprof is also done, AFAICS. ... just noticed rotatingtree.c; sorry 'bout the noise.

[ericsnowcurrently]

Thanks for the updates!

[terryjreedy]

@serhiy-storchaka for _tkinter. I have nothing to do with it.