Implement Python Critical Sections from PEP 703

[colesbury]

Feature or enhancement

PEP 703 introduces the concept of "Python Critical Sections", which are an abstraction to help replace the GIL with finer grained locking. The key ideas is to replace the GIL with per-object locks and implicitly release these locks in the same places where the GIL would have been released. The mechanism is:

1. Keep track of locked mutexes in a per-thread stack
2. When the thread "detaches" from the interpreter (i.e., _PyThreadState_Detach()), unlock all of the thread's locked mutexes
3. When the thread re-"attaches" to the interpreter, re-lock the top-most mutex or mutexes.

The "public" ¹ API consists of four macros:

Py_BEGIN_CRITICAL_SECTION(object);
Py_END_CRITICAL_SECTION;
Py_BEGIN_CRITICAL_SECTION2(object1, object2);
Py_END_CRITICAL_SECTION2;

These will be no-ops in the default build of CPython.

Note that if you need to operate on two objects at once, then you must use the Py_BEGIN_CRITICAL_SECTION2 macro. Nesting two calls to Py_BEGIN_CRITICAL_SECTION does not guarantee that both objects are locked because the inner calls may suspend the outer critical section.

Linked PRs

• gh-111569: Implement Python critical section API #111571
• gh-111569: Fix critical sections test on WebAssembly #111897

Footnotes

1. At least at the start, even these "public" macros will still be internal-only (i.e., in Include/internal). I expect that we will eventually want to make them public so that C-API extensions can make use of them. ↩

[brettcannon]

The latest PR related to this change broke the WebAssembly buildbots by causing them to time out: https://buildbot.python.org/all/#/builders/1046/builds/3456/steps/11/logs/stdio .