Improve Interpreter Isolation

[ericsnowcurrently]

In CPython, we store (persistent) runtime data in several locations. (See https://github.com/ericsnowcurrently/multi-core-python/wiki/3-CPython's-Runtime#cpythons-run-time-data.)

The state of each interpreter (PyInterpreterState) encapsulates the data which is unique to that interpreter and isolated from other interpreters. That isolation is important for the proper operation of multiple interpreters in a process, and will be critical for a per-interpreter GIL, AKA PEP 684 (assuming it gets approved). There are still a number of gaps in isolation which we will address here.

The deficiencies may be categorized by the following:

• shared runtime state (_PyRuntimeState) conceptually better suited to each interpreter
• shared runtime state currently protected by the GIL
• shared global variables that shouldn't be shared
• shared global variables currently protected by the GIL

Linked PRs

• gh-100227: Port dup3_works to the module state variable. #100262
• gh-100227: Isolate the Import State to Each Interpreter #101941
• gh-100227: Move the dtoa State to PyInterpreterState #102331
• gh-100227: Move _str_replace_inf to PyInterpreterState #102333
• gh-100227: Move func_state.next_version to PyInterpreterState #102334
• gh-100227: Move next_keys_version to PyInterpreterState #102335
• gh-100227: Move dict_state.global_version to PyInterpreterState #102338
• gh-100227: Move the Dict of Interned Strings to PyInterpreterState #102339
• GH-100227: cleanup initialization of global interned dict #102682
• gh-100227: Make the Global Interned Dict Safe for Isolated Interpreters #102858
• gh-100227: Make the Global Interned Dict Safe for Isolated Interpreters #102925
• gh-100227: Make the Global PyModuleDef Cache Safe for Isolated Interpreters #102938
• gh-100227: Revert "gh-100227: Make the Global Interned Dict Safe for Isolated Interpreters (gh-102925)" #103063
• gh-100227: Make the Global PyModuleDef Cache Safe for Isolated Interpreters #103084
• gh-100227: Fix Cleanup of the Extensions Cache #103150
• gh-100227: Use an Array for _PyRuntime's Set of Locks During Init #103315
• gh-100227: Add a Granular Lock for _PyRuntime.imports.extensions.dict #103460
• gh-100227: Only Use deepfreeze for the Main Interpreter #103794
• gh-100227: Add Missing Locks for Specific Global Runtime State #105470
• gh-100227: Lock Around Use of the Global "atexit" State #105514
• gh-100227: Lock Around Adding Global Audit Hooks #105515
• gh-100227: Lock Around Modification of the Global Allocators State #105516
• [3.12] gh-100227: Lock Around Use of the Global "atexit" State (gh-105514) #105517
• [3.12] gh-100227: Lock Around Adding Global Audit Hooks (gh-105515) #105525
• [3.12] gh-100227: Lock Around Modification of the Global Allocators State (gh-105516) #105532
• gh-105766: Add Locking Around Custom Allocators #105619

[ericsnowcurrently]

A related analysis of _PyRuntimeState and the remaining global variables is found at https://github.com/ericsnowcurrently/multi-core-python/wiki/3-CPython's-Runtime#isolating-interpreter-state.

In summary, the following state needs attention:

• isolate to PyInterpreterState
  • _PyRuntimeState.obmalloc  (issue)
  • _PyRuntimeState.signals.default_handler (move to module state)
  • _PyRuntimeState.signals.ignore_handler (move to module state)
  • _PyRuntimeState.imports.lock
  • _PyRuntimeState.imports.find_and_load
  • _PyRuntimeState.dtoa
  • _PyRuntimeState.dict_state.global_version
  • _PyRuntimeState.dict_state.next_keys_version
  • _PyRuntimeState.func_state.next_version
  • _PyRuntimeState.types.next_version_tag  (issue) (special-case for global/immortal types)
  • _PyRuntimeState.cached_objects.str_replace_inf
  • _PyRuntimeState.cached_objects.interned
• currently protected by the GIL (need granular locks for per-interpreter GIL)
  • _PyRuntimeState.exitfuncs (race in Py_AtExit())
  • _PyRuntimeState.nexitfuncs (race in Py_AtExit())
  • _PyRuntimeState.allocators (PyMem_SetAllocator() with "wrappers"; also when using mem/obj allocators)
  • _PyRuntimeState.audit_hook_head (race in PySys_AddAuditHook())
  • _PyRuntimeState.ceval.perf
  • Objects/longobject.c - (long_from_non_binary_base) log_base_BASE (lazy)
  • Objects/longobject.c - (long_from_non_binary_base) convwidth_base (lazy)
  • Objects/longobject.c - (long_from_non_binary_base) convmultmax_base (lazy)
• both (move some state and keep some state; copy)
  • Objects/object.c - _Py_RefTotal  (issue)

Moves to PyInterpreterState with the GIL (not part of this issue):

• _PyRuntimeState.gilstate.tstate_current

Also note:

• the GIL is currently fine as it is (part of _PyRuntimeState); moving it to PyInterpreterState is a separate matter
• we'll handle faulthandler and tracemalloc state separately

old list

The following state needs further investigation and maybe further attention:

• maybe isolate to PyInterpreterState
  • _PyRuntimeState.allocators.obj_arena (effectively coupled to _PyRuntime.obmalloc)
  • _PyRuntimeState.obmalloc.dump_debug_stats
  • ...
  • _PyRuntimeState.signals.wakeup
  • _PyRuntimeState.signals.is_tripped
  • _PyRuntimeState.signals.unhandled_keyboard_interrupt
  • ...
  • _PyRuntimeState.tracemalloc.traced_memory
  • _PyRuntimeState.tracemalloc.peak_traced_memory
  • _PyRuntimeState.tracemalloc.traces
  • _PyRuntimeState.tracemalloc.domains
  • ...
  • _PyRuntimeState.parser.memo_statistics
• maybe protected by the GIL
  • _PyRuntimeState.fileutils.force_ascii
  • Modules/posixmodule.c - (os_dup2_impl) dup3_works (lazy)
  • Modules/faulthandler.c - (faulthandler_dump_traceback) reentrant
  • Objects/unicodeobject.c - bloom_linebreak (lazy)
  • Python/pylifecycle.c - (_Py_FatalErrorFormat) reentrant
  • Python/pylifecycle.c - (fatal_error) reentrant
• currently protected by the GIL but maybe isolate to PyInterpreterState
  • _PyRuntimeState.imports.last_module_index
  • _PyRuntimeState.imports.extensions
  • _PyRuntimeState.imports.pkgcontext
  • ...
  • _PyRuntimeState.tracemalloc.filenames
  • _PyRuntimeState.tracemalloc.traceback
  • _PyRuntimeState.tracemalloc.tracebacks
• maybe either
  • _PyRuntimeState.open_code_hook
  • _PyRuntimeState.open_code_userdata
  • ...
  • _PyRuntimeState.faulthandler.fatal_error
  • _PyRuntimeState.faulthandler.thread
  • _PyRuntimeState.faulthandler.user_signals
  • _PyRuntimeState.faulthandler.stack
  • _PyRuntimeState.faulthandler.old_stack
  • ...
  • _PyRuntimeState.tracemalloc.config
  • _PyRuntimeState.tracemalloc.allocators
  • _PyRuntimeState.tracemalloc.tables_lock
  • ...
  • Parser/action_helpers.c - (_PyPegen_dummy_name) cache
• safe as-is but may be unnecessary (or need other solution)
  • _PyRuntimeState.gilstate.check_enabled
  • _PyRuntimeState.gilstate.autoInterpreterState

[corona10]

Objects/object.c - _Py_RefTotal

Hmm how about introducing a new debug API: _Py_Get_RefTotal()?
And then move _Py_RefTotal into per interpreter attribute(_ref_total) from the global.
_Py_Get_RefTotal() will return the aggregated value of each interpreter _ref_total value..
Current _Py_RefTotal will only return the main interpreter value and will be deprecated.

Ahh.. But it looks like stable API, I am not sure it's proper approach..

[encukou]

It's OK to deprecate it. It just can't be removed :)
If PEP 684 is accepted, _Py_RefTotal should be used for old-style interpreters, and interpreters with own_gil could get their own counter.

[markshannon]

Before proceeding with these changes, I think we need some up-front design.
(TBH I think we need more up-front design for a lot of things in CPython)

Memory allocation

We need some sort of global memory allocator. Is a per-interpreter ob_malloc, backed by a global allocator, presumably malloc the correct design? It might be, but we seem to be making an architectural decision by accident here.

An alternative design for allocation, that I would like to try, is per-interpreter size-based freelists backed by a global allocator.
This allows backwards compatibility with pluggable allocators and is fast. For more details faster-cpython/ideas#132

Tracemalloc

Tracemalloc data structures should be per-interpreter. Access to global values (like process-wide total memory traced) will need to be done through a function interface.

Efficiency

As more and more of the VM becomes per-interpreter, the performance impact of getting the interpreter with a C-API that was not designed with it in mind becomes larger.
@vstinner added a version of many C API functions that take the thread-state, but these aren't joined up, so there is a lot of PyThreadState_Get() calls scattered throughout the code.

We need to decide whether to pass the thread state or interpreter state.
IMO, we should follow the lead of HPy and pass the interpreter state.
We will need new API functions in many places, but many of those will be simple mechanical transformations.

Backwards compatibility

_Py_RefTotal is only defined if Py_REF_DEBUG is defined.
All updates to _Py_RefTotal can be made atomic (which will have terrible performance with multiple parallel interpreters) to preserve the API.

Ownership (runtime or per-interpreter) and constants

Making something const means it can be global (and shouldn't be in the runtime state, which is mutable).
A few of the values listed above are constants. E.g. log_base_BASE

Some values are not const but are constant once initialized. These can be shared without locks. These values should be grouped separately from values that are shared and mutable and will need locks.

Nothing should be in the "maybe both" category above.

[ericsnowcurrently]

FYI, I have a plan for _Py_RefTotal that should preserve stable ABI compatibility without breaking interpreter isolation (and without relying on granular locks). Basically:

• deprecate _Py_Reftotal but keep exporting the symbol (for stable ABI compatibility)
• add _PyRuntimeState.object_state.last_legacy_reftotal
• add _PyRuntimeState.object_state.reftotal
• update everywhere to use the reftotal field (for simplicity, add #define _Py_RefTotal _PyRuntime.object_state.reftotal)

We would update _Py_GetRefTotal() to something like this:

Py_ssize_t
_Py_GetRefTotal(void)
{
    // _Py_GetLegacyRefTotal() returns the value of the actual `_Py_RefTotal` global.
    Py_ssize_t legacy = _Py_GetLegacyRefTotal();
    _PyRuntimeState.object_state.reftotal += legacy - _PyRuntimeState.object_state.last_legacy_reftotal;
    _PyRuntimeState.object_state.last_legacy_reftotal = legacy;
    return _PyRuntimeState.object_state.reftotal;
}

All this assumes that folks are not using _Py_RefTotal directly.

---

For a per-interpreter GIL, I see two options:

1. add a granular lock around modifying _PyRuntimeState.object_state.reftotal, which would hurt the performance of incref/decref, but only on debug builds
2. switch to PyInterpreterState.object_state.reftotal and aggregate the values from all interpreters in _Py_GetRefTotal()

[ericsnowcurrently]

FYI, I have a plan for _Py_RefTotal that should preserve stable ABI compatibility without breaking interpreter isolation

FWIW, the 3.10 changes to Py_INCREF() and Py_DECREF() help here because _Py_RefTotal is no longer used by stable ABI extensions (other than ones compiled against 3.9 and earlier).

[ericsnowcurrently]

Before proceeding with these changes, I think we need some up-front design.

Yes, for any changes that don't make sense without a per-interpreter GIL, we would not merge them before PEP 684 is accepted.

All the points you have brought up are worth discussing. Note, though, that I consider the current per-interpreter GIL work to be a minimal approach to getting there, with at little disruption as possible while tackling a variety of improvements that make sense anyway, including improving the isolation of the existing interpreters. Much of what you've brought up are important points, but ones that will require much more discussion, take substantially more effort, and introduce more disruption. I don't think we should hold up the current project for a resolution of those broader matters.

That said, in the context of the above checklists, I agree that we must be very conservative in how we bake in any new design/architecture. I just consider isolated interpreters to be an old design (with a less than ideal API and an incomplete implementation) that we are carrying through. Improving the design is a separate project, which I wholly support!