feat: scoped_critical_section

[henryiii]

Description

This adds a wrapper, py::scoped_critical_section. Simplifies the usage in our tests. I think this will likely allow us to fix our GIL tests, as long as we find the right place to include it. Updated the docs, which I believe were missing a .ptr() before.

Suggested changelog entry:

• Added py::scoped_critical_section for free-threaded use.

---

📚 Documentation preview 📚: https://pybind11--5684.org.readthedocs.build/

[rwgk]

Suggestion to move this to gil_simple.h, but with this name (scoped_critical_section).

Why: It will be usable without bringing in #include "detail/internals.h"

[henryiii]

Maybe we should just make a new file (edit: done). What's the status of gil_simple? Should we have changed the default? Don't remember if there were drawbacks. @tonybaloney was talking to me at PyCon about GIL management issues, I wonder if that would help.

[rwgk]

Maybe we should just make a new file (edit: done).

I really like that!

The code looks good to me as-is. I didn't look at the test failures, I guess you're it it?

Should we have changed the default?

That crossed my mind, too, but I came down on the side: it's best to not open that can of worms.

With v3.0.0, we can tell users to explicitly switch to gil_scoped_acquire_simple or gil_scoped_release_simple if they have a reason.

I.e.: We're not causing a stir by changing the default, and it's very easy to opt-in to the simple variants (simply append _simple, no other changes required, I think).

[Copilot]

Pull Request Overview

This PR introduces a new wrapper, py::scoped_critical_section, to simplify and improve thread synchronization in tests that require free‐threaded Python usage. Key changes include adding the new critical section header and its implementation, updating test files to use the new wrapper, and revising documentation and build configuration accordingly.

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/test_embed/test_interpreter.cpp	Updated thread test to use py::scoped_critical_section.
tests/extra_python_package/test_files.py	Added the new critical_section header to the file list.
include/pybind11/critical_section.h	New implementation of the scoped_critical_section class.
docs/advanced/misc.rst	Revised docs to reflect usage and emphasize correct lines.
CMakeLists.txt	Added critical_section.h to header installation list.

[Copilot]

In the conditional block for Py_GIL_DISABLED, the update to locals['count'] appears inside the lock for the older branch but is performed outside for the new branch. To ensure consistent thread safety, consider moving the update inside the critical section scope in all branches.

[rwgk]

The production code looks great, but I'm still worried about sending inexperienced users into deadlock pitfalls. See suggestions.

[rwgk]

While we're touching this code, I think it'll be prudent to warn readers about the classic pitfall:

                 py::gil_scoped_acquire gil{};
 #ifdef Py_GIL_DISABLED
 #    if PY_VERSION_HEX < 0x030E0000
+                // WARNING: This assumes that the Python C API call below does
+                // not release and reacquire the GIL. Guarding Python object
+                // access with a C++ mutex (while holding the GIL) can
+                // deadlock if that assumption is violated.
                 std::lock_guard<std::mutex> lock(mutex);
 #    else
+                // Safe: uses CPython's thread-safe API in no-GIL mode.
                 py::scoped_critical_section lock(locals);
 #    endif
 #endif

[rwgk]

My concern: I feel like this could send inexperienced users into the deadlock pitfall.

Please take a look here for background:

https://chatgpt.com/share/682a07ec-dd70-8008-92ea-e722ff92a055

Concrete suggestions from the chat:

Add a comment to clarify:

            py::scoped_critical_section guard(g); // Safe only if this is not nested and no GIL is held.

Suggested addition to the explanatory text below the code block:

---

⚠️ When using py::scoped_critical_section, make sure it is not nested and that no other synchronization primitives (such as the GIL or a std::mutex) are held that could lead to deadlocks. This RAII guard wraps Python’s PyCriticalSection_* APIs, which are only active in free-threaded builds (Py_GIL_DISABLED). In standard (GIL-enabled) Python builds, the guard is a no-op.

On Python 3.13, nested use of critical sections may re-lock underlying internal structures; this was optimized away in Python 3.14+. If needed, consider using a std::mutex to avoid such cases.

---

My own refinement of the last point: Depending on the situation, you may have to resort to a std::mutex.

[henryiii]

py::scoped_critical_section is safe WRT the GIL, because it is GIL aware, so "such as the GIL" doesn't apply to it. But std::mutex could deadlock.

[rwgk]

Oh, merged already? I hope you can still consider the doc and comment suggestions.

[henryiii]

Ahh, sorry, I thought you said it looked good above. I can make a new PR with suggestions.