Merge branch 'main' into re_improvements

Doc/c-api/init.rst

@@ -430,7 +430,11 @@ Initializing and finalizing the interpreter
    Some memory allocated by extension modules may not be freed.  Some extensions may not
    work properly if their initialization routine is called more than once; this can
    happen if an application calls :c:func:`Py_Initialize` and :c:func:`Py_FinalizeEx`
-   more than once.
+   more than once.  :c:func:`Py_FinalizeEx` must not be called recursively from
+   within itself.  Therefore, it must not be called by any code that may be run
+   as part of the interpreter shutdown process, such as :py:mod:`atexit`
+   handlers, object finalizers, or any code that may be run while flushing the
+   stdout and stderr files.
 
    .. audit-event:: cpython._PySys_ClearAuditHooks "" c.Py_FinalizeEx
 
@@ -960,6 +964,37 @@ thread, where the CPython global runtime was originally initialized.
 The only exception is if :c:func:`exec` will be called immediately
 after.
 
+.. _cautions-regarding-runtime-finalization:
+
+Cautions regarding runtime finalization
+---------------------------------------
+
+In the late stage of :term:`interpreter shutdown`, after attempting to wait for
+non-daemon threads to exit (though this can be interrupted by
+:class:`KeyboardInterrupt`) and running the :mod:`atexit` functions, the runtime
+is marked as *finalizing*: :c:func:`_Py_IsFinalizing` and
+:func:`sys.is_finalizing` return true.  At this point, only the *finalization
+thread* that initiated finalization (typically the main thread) is allowed to
+acquire the :term:`GIL`.
+
+If any thread, other than the finalization thread, attempts to acquire the GIL
+during finalization, either explicitly via :c:func:`PyGILState_Ensure`,
+:c:macro:`Py_END_ALLOW_THREADS`, :c:func:`PyEval_AcquireThread`, or
+:c:func:`PyEval_AcquireLock`, or implicitly when the interpreter attempts to
+reacquire it after having yielded it, the thread enters **a permanently blocked
+state** where it remains until the program exits.  In most cases this is
+harmless, but this can result in deadlock if a later stage of finalization
+attempts to acquire a lock owned by the blocked thread, or otherwise waits on
+the blocked thread.
+
+Gross? Yes. This prevents random crashes and/or unexpectedly skipped C++
+finalizations further up the call stack when such threads were forcibly exited
+here in CPython 3.13 and earlier. The CPython runtime GIL acquiring C APIs
+have never had any error reporting or handling expectations at GIL acquisition
+time that would've allowed for graceful exit from this situation. Changing that
+would require new stable C APIs and rewriting the majority of C code in the
+CPython ecosystem to use those with error handling.
+
 
 High-level API
 --------------
@@ -1033,11 +1068,14 @@ code, or when embedding the Python interpreter:
    ensues.
 
    .. note::
-      Calling this function from a thread when the runtime is finalizing
-      will terminate the thread, even if the thread was not created by Python.
-      You can use :c:func:`Py_IsFinalizing` or :func:`sys.is_finalizing` to
-      check if the interpreter is in process of being finalized before calling
-      this function to avoid unwanted termination.
+      Calling this function from a thread when the runtime is finalizing will
+      hang the thread until the program exits, even if the thread was not
+      created by Python.  Refer to
+      :ref:`cautions-regarding-runtime-finalization` for more details.
+
+   .. versionchanged:: next
+      Hangs the current thread, rather than terminating it, if called while the
+      interpreter is finalizing.
 
 .. c:function:: PyThreadState* PyThreadState_Get()
 
@@ -1092,11 +1130,14 @@ with sub-interpreters:
    to call arbitrary Python code.  Failure is a fatal error.
 
    .. note::
-      Calling this function from a thread when the runtime is finalizing
-      will terminate the thread, even if the thread was not created by Python.
-      You can use :c:func:`Py_IsFinalizing` or :func:`sys.is_finalizing` to
-      check if the interpreter is in process of being finalized before calling
-      this function to avoid unwanted termination.
+      Calling this function from a thread when the runtime is finalizing will
+      hang the thread until the program exits, even if the thread was not
+      created by Python.  Refer to
+      :ref:`cautions-regarding-runtime-finalization` for more details.
+
+   .. versionchanged:: next
+      Hangs the current thread, rather than terminating it, if called while the
+      interpreter is finalizing.
 
 .. c:function:: void PyGILState_Release(PyGILState_STATE)
 
@@ -1224,7 +1265,7 @@ All of the following functions must be called after :c:func:`Py_Initialize`.
 .. c:function:: void PyThreadState_DeleteCurrent(void)
 
    Destroy the current thread state and release the global interpreter lock.
-   Like :c:func:`PyThreadState_Delete`, the global interpreter lock need not
+   Like :c:func:`PyThreadState_Delete`, the global interpreter lock must
    be held. The thread state must have been reset with a previous call
    to :c:func:`PyThreadState_Clear`.
 
@@ -1374,17 +1415,20 @@ All of the following functions must be called after :c:func:`Py_Initialize`.
    If this thread already has the lock, deadlock ensues.
 
    .. note::
-      Calling this function from a thread when the runtime is finalizing
-      will terminate the thread, even if the thread was not created by Python.
-      You can use :c:func:`Py_IsFinalizing` or :func:`sys.is_finalizing` to
-      check if the interpreter is in process of being finalized before calling
-      this function to avoid unwanted termination.
+      Calling this function from a thread when the runtime is finalizing will
+      hang the thread until the program exits, even if the thread was not
+      created by Python.  Refer to
+      :ref:`cautions-regarding-runtime-finalization` for more details.
 
    .. versionchanged:: 3.8
       Updated to be consistent with :c:func:`PyEval_RestoreThread`,
       :c:func:`Py_END_ALLOW_THREADS`, and :c:func:`PyGILState_Ensure`,
       and terminate the current thread if called while the interpreter is finalizing.
 
+   .. versionchanged:: next
+      Hangs the current thread, rather than terminating it, if called while the
+      interpreter is finalizing.
+
    :c:func:`PyEval_RestoreThread` is a higher-level function which is always
    available (even when threads have not been initialized).
 

Doc/library/functools.rst

@@ -347,8 +347,7 @@ The :mod:`functools` module defines the following functions:
 
       def partial(func, /, *args, **keywords):
           def newfunc(*more_args, **more_keywords):
-              keywords_union = {**keywords, **more_keywords}
-              return func(*args, *more_args, **keywords_union)
+              return func(*args, *more_args, **(keywords | more_keywords))
           newfunc.func = func
           newfunc.args = args
           newfunc.keywords = keywords

Doc/whatsnew/3.13.rst

@@ -498,30 +498,6 @@ are not tier 3 supported platforms, but will have best-effort support.
 .. seealso:: :pep:`730`, :pep:`738`
 
 
-.. _whatsnew313-incremental-gc:
-
-Incremental garbage collection
-------------------------------
-
-The cycle garbage collector is now incremental.
-This means that maximum pause times are reduced
-by an order of magnitude or more for larger heaps.
-
-There are now only two generations: young and old.
-When :func:`gc.collect` is not called directly, the
-GC is invoked a little less frequently. When invoked, it
-collects the young generation and an increment of the
-old generation, instead of collecting one or more generations.
-
-The behavior of :func:`!gc.collect` changes slightly:
-
-* ``gc.collect(1)``: Performs an increment of garbage collection,
-  rather than collecting generation 1.
-* Other calls to :func:`!gc.collect` are unchanged.
-
-(Contributed by Mark Shannon in :gh:`108362`.)
-
-
 Other Language Changes
 ======================
 
@@ -918,36 +894,6 @@ fractions
   (Contributed by Mark Dickinson in :gh:`111320`.)
 
 
-gc
---
-
-The cyclic garbage collector is now incremental,
-which changes the meaning of the results of
-:meth:`~gc.get_threshold` and :meth:`~gc.set_threshold`
-as well as :meth:`~gc.get_count` and :meth:`~gc.get_stats`.
-
-* For backwards compatibility, :meth:`~gc.get_threshold` continues to return
-  a three-item tuple.
-  The first value is the threshold for young collections, as before;
-  the second value determines the rate at which the old collection is scanned
-  (the default is 10, and higher values mean that the old collection
-  is scanned more slowly).
-  The third value is meaningless and is always zero.
-
-* :meth:`~gc.set_threshold` ignores any items after the second.
-
-* :meth:`~gc.get_count` and :meth:`~gc.get_stats` continue to return
-  the same format of results.
-  The only difference is that instead of the results referring to
-  the young, aging and old generations,
-  the results refer to the young generation
-  and the aging and collecting spaces of the old generation.
-
-In summary, code that attempted to manipulate the behavior of the cycle GC
-may not work exactly as intended, but it is very unlikely to be harmful.
-All other code will work just fine.
-
-
 glob
 ----
 
@@ -1512,11 +1458,6 @@ zipimport
 Optimizations
 =============
 
-* The new :ref:`incremental garbage collector <whatsnew313-incremental-gc>`
-  means that maximum pause times are reduced
-  by an order of magnitude or more for larger heaps.
-  (Contributed by Mark Shannon in :gh:`108362`.)
-
 * Several standard library modules have had
   their import times significantly improved.
   For example, the import time of the :mod:`typing` module
@@ -2629,13 +2570,6 @@ Changes in the Python API
   Wrap it in :func:`staticmethod` if you want to preserve the old behavior.
   (Contributed by Serhiy Storchaka in :gh:`121027`.)
 
-* The :ref:`garbage collector is now incremental <whatsnew313-incremental-gc>`,
-  which means that the behavior of :func:`gc.collect` changes slightly:
-
-  * ``gc.collect(1)``: Performs an increment of garbage collection,
-    rather than collecting generation 1.
-  * Other calls to :func:`!gc.collect` are unchanged.
-
 * An :exc:`OSError` is now raised by :func:`getpass.getuser`
   for any failure to retrieve a username,
   instead of :exc:`ImportError` on non-Unix platforms

Include/cpython/longobject.h

@@ -2,6 +2,9 @@
 #  error "this header file must not be included directly"
 #endif
 
+#define _PyLong_CAST(op) \
+    (assert(PyLong_Check(op)), _Py_CAST(PyLongObject*, (op)))
+
 PyAPI_FUNC(PyObject*) PyLong_FromUnicodeObject(PyObject *u, int base);
 
 #define Py_ASNATIVEBYTES_DEFAULTS -1

Include/cpython/weakrefobject.h

@@ -42,13 +42,13 @@ struct _PyWeakReference {
 
 PyAPI_FUNC(void) _PyWeakref_ClearRef(PyWeakReference *self);
 
+#define _PyWeakref_CAST(op) \
+    (assert(PyWeakref_Check(op)), _Py_CAST(PyWeakReference*, (op)))
+
 Py_DEPRECATED(3.13) static inline PyObject* PyWeakref_GET_OBJECT(PyObject *ref_obj)
 {
-    PyWeakReference *ref;
-    PyObject *obj;
-    assert(PyWeakref_Check(ref_obj));
-    ref = _Py_CAST(PyWeakReference*, ref_obj);
-    obj = ref->wr_object;
+    PyWeakReference *ref = _PyWeakref_CAST(ref_obj);
+    PyObject *obj = ref->wr_object;
     // Explanation for the Py_REFCNT() check: when a weakref's target is part
     // of a long chain of deallocations which triggers the trashcan mechanism,
     // clearing the weakrefs can be delayed long after the target's refcount

Include/internal/pycore_moduleobject.h

@@ -46,7 +46,7 @@ static inline PyObject* _PyModule_GetDict(PyObject *mod) {
 }
 
 PyObject* _Py_module_getattro_impl(PyModuleObject *m, PyObject *name, int suppress);
-PyObject* _Py_module_getattro(PyModuleObject *m, PyObject *name);
+PyObject* _Py_module_getattro(PyObject *m, PyObject *name);
 
 #ifdef __cplusplus
 }

Include/internal/pycore_pythread.h

@@ -152,6 +152,19 @@ PyAPI_FUNC(int) PyThread_join_thread(PyThread_handle_t);
  * a non-zero value on failure.
  */
 PyAPI_FUNC(int) PyThread_detach_thread(PyThread_handle_t);
+/*
+ * Hangs the thread indefinitely without exiting it.
+ *
+ * gh-87135: There is no safe way to exit a thread other than returning
+ * normally from its start function.  This is used during finalization in lieu
+ * of actually exiting the thread.  Since the program is expected to terminate
+ * soon anyway, it does not matter if the thread stack stays around until then.
+ *
+ * This is unfortunate for embedders who may not be terminating their process
+ * when they're done with the interpreter, but our C API design does not allow
+ * for safely exiting threads attempting to re-enter Python post finalization.
+ */
+void _Py_NO_RETURN PyThread_hang_thread(void);
 
 #ifdef __cplusplus
 }

Include/pythread.h

@@ -17,7 +17,26 @@ typedef enum PyLockStatus {
 
 PyAPI_FUNC(void) PyThread_init_thread(void);
 PyAPI_FUNC(unsigned long) PyThread_start_new_thread(void (*)(void *), void *);
-PyAPI_FUNC(void) _Py_NO_RETURN PyThread_exit_thread(void);
+/* Terminates the current thread. Considered unsafe.
+ *
+ * WARNING: This function is only safe to call if all functions in the full call
+ * stack are written to safely allow it.  Additionally, the behavior is
+ * platform-dependent.  This function should be avoided, and is no longer called
+ * by Python itself.  It is retained only for compatibility with existing C
+ * extension code.
+ *
+ * With pthreads, calls `pthread_exit` causes some libcs (glibc?) to attempt to
+ * unwind the stack and call C++ destructors; if a `noexcept` function is
+ * reached, they may terminate the process. Others (macOS) do unwinding.
+ *
+ * On Windows, calls `_endthreadex` which kills the thread without calling C++
+ * destructors.
+ *
+ * In either case there is a risk of invalid references remaining to data on the
+ * thread stack.
+ */
+Py_DEPRECATED(3.14) PyAPI_FUNC(void) _Py_NO_RETURN PyThread_exit_thread(void);
+
 PyAPI_FUNC(unsigned long) PyThread_get_thread_ident(void);
 
 #if (defined(__APPLE__) || defined(__linux__) || defined(_WIN32) \

Lib/argparse.py

@@ -527,8 +527,7 @@ def _format_action(self, action):
     def _format_action_invocation(self, action):
         if not action.option_strings:
             default = self._get_default_metavar_for_positional(action)
-            metavar, = self._metavar_formatter(action, default)(1)
-            return metavar
+            return ' '.join(self._metavar_formatter(action, default)(1))
 
         else:
 
@@ -703,7 +702,15 @@ def _get_action_name(argument):
     elif argument.option_strings:
         return '/'.join(argument.option_strings)
     elif argument.metavar not in (None, SUPPRESS):
-        return argument.metavar
+        metavar = argument.metavar
+        if not isinstance(metavar, tuple):
+            return metavar
+        if argument.nargs == ZERO_OR_MORE and len(metavar) == 2:
+            return '%s[, %s]' % metavar
+        elif argument.nargs == ONE_OR_MORE:
+            return '%s[, %s]' % metavar
+        else:
+            return ', '.join(metavar)
     elif argument.dest not in (None, SUPPRESS):
         return argument.dest
     elif argument.choices:

Lib/asyncio/base_events.py

@@ -17,7 +17,6 @@
 import collections.abc
 import concurrent.futures
 import errno
-import functools
 import heapq
 import itertools
 import os
@@ -1140,11 +1139,18 @@ async def create_connection(
                     except OSError:
                         continue
             else:  # using happy eyeballs
-                sock, _, _ = await staggered.staggered_race(
-                    (functools.partial(self._connect_sock,
-                                       exceptions, addrinfo, laddr_infos)
-                     for addrinfo in infos),
-                    happy_eyeballs_delay, loop=self)
+                sock = (await staggered.staggered_race(
+                    (
+                        # can't use functools.partial as it keeps a reference
+                        # to exceptions
+                        lambda addrinfo=addrinfo: self._connect_sock(
+                            exceptions, addrinfo, laddr_infos
+                        )
+                        for addrinfo in infos
+                    ),
+                    happy_eyeballs_delay,
+                    loop=self,
+                ))[0]  # can't use sock, _, _ as it keeks a reference to exceptions
 
             if sock is None:
                 exceptions = [exc for sub in exceptions for exc in sub]

Lib/asyncio/staggered.py

@@ -133,6 +133,7 @@ async def run_one_coro(previous_failed) -> None:
                         raise d.exception()
         return winner_result, winner_index, exceptions
     finally:
+        del exceptions
         # Make sure no tasks are left running if we leave this function
         for t in running_tasks:
             t.cancel()