Merge remote-tracking branch 'upstream/main' into nested-async-comp

Author: pablogsal

.github/workflows/doc.yml

@@ -40,7 +40,7 @@ jobs:
     - name: 'Build documentation'
       run: xvfb-run make -C Doc/ PYTHON=../python SPHINXOPTS="-q -W --keep-going -j4" doctest html
     - name: 'Upload'
-      uses: actions/upload-artifact@v2.2.3
+      uses: actions/upload-artifact@v2.2.4
       with:
         name: doc-html
         path: Doc/build/html

Doc/c-api/code.rst

@@ -59,3 +59,11 @@ bound into a function.
 
     For efficiently iterating over the line numbers in a code object, use `the API described in PEP 626
     <https://www.python.org/dev/peps/pep-0626/#out-of-process-debuggers-and-profilers>`_.
+
+.. c:function:: int PyCode_Addr2Location(PyObject *co, int byte_offset, int *start_line, int *start_column, int *end_line, int *end_column)
+
+   Sets the passed ``int`` pointers to the source code line and column numbers
+   for the instruction at ``byte_offset``. Sets the value to ``0`` when
+   information is not available for any particular element.
+
+   Returns ``1`` if the function succeeds and 0 otherwise.

Doc/library/argparse.rst

@@ -853,6 +853,8 @@ is available in ``argparse`` and adds support for boolean actions such as
     >>> parser.parse_args(['--no-foo'])
     Namespace(foo=False)
 
+.. versionadded:: 3.9
+
 The recommended way to create a custom action is to extend :class:`Action`,
 overriding the ``__call__`` method and optionally the ``__init__`` and
 ``format_usage`` methods.

Doc/library/bz2.rst

@@ -325,3 +325,8 @@ Writing and reading a bzip2-compressed file in binary mode:
     ...     content = f.read()
     >>> content == data  # Check equality to original object after round-trip
     True
+
+.. testcleanup::
+
+   import os
+   os.remove("myfile.bz2")

Doc/library/configparser.rst

@@ -46,6 +46,11 @@ can be customized by end users easily.
 
    import configparser
 
+.. testcleanup::
+
+   import os
+   os.remove("example.ini")
+
 
 Quick Start
 -----------

Doc/library/importlib.metadata.rst

@@ -8,13 +8,11 @@
    :synopsis: The implementation of the importlib metadata.
 
 .. versionadded:: 3.8
+.. versionchanged:: 3.10
+   ``importlib.metadata`` is no longer provisional.
 
 **Source code:** :source:`Lib/importlib/metadata.py`
 
-.. note::
-   This functionality is provisional and may deviate from the usual
-   version semantics of the standard library.
-
 ``importlib.metadata`` is a library that provides for access to installed
 package metadata.  Built in part on Python's import system, this library
 intends to replace similar functionality in the `entry point

Doc/library/io.rst

@@ -391,6 +391,9 @@ I/O Base Classes
       to control the number of lines read: no more lines will be read if the
       total size (in bytes/characters) of all lines so far exceeds *hint*.
 
+      *hint* values of ``0`` or less, as well as ``None``, are treated as no
+      hint.
+
       Note that it's already possible to iterate on file objects using ``for
       line in file: ...`` without calling ``file.readlines()``.
 

Doc/library/os.path.rst

@@ -340,6 +340,14 @@ the :mod:`glob` module.)
    that contains symbolic links.  On Windows, it converts forward slashes to
    backward slashes. To normalize case, use :func:`normcase`.
 
+  .. note::
+      On POSIX systems, in accordance with `IEEE Std 1003.1 2013 Edition; 4.13
+      Pathname Resolution <http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_13>`_,
+      if a pathname begins with exactly two slashes, the first component
+      following the leading characters may be interpreted in an implementation-defined
+      manner, although more than two leading characters shall be treated as a
+      single character.
+
    .. versionchanged:: 3.6
       Accepts a :term:`path-like object`.
 

Doc/library/traceback.rst

@@ -473,7 +473,7 @@ The output for the example would look similar to this:
    ['Traceback (most recent call last):\n',
     '  File "<doctest default[0]>", line 10, in <module>\n    lumberjack()\n    ^^^^^^^^^^^^\n',
     '  File "<doctest default[0]>", line 4, in lumberjack\n    bright_side_of_death()\n    ^^^^^^^^^^^^^^^^^^^^^^\n',
-    '  File "<doctest default[0]>", line 7, in bright_side_of_death\n    return tuple()[0]\n           ^^^^^^^^^^\n',
+    '  File "<doctest default[0]>", line 7, in bright_side_of_death\n    return tuple()[0]\n           ~~~~~~~^^^\n',
     'IndexError: tuple index out of range\n']
    *** extract_tb:
    [<FrameSummary file <doctest...>, line 10 in <module>>,
@@ -482,7 +482,7 @@ The output for the example would look similar to this:
    *** format_tb:
    ['  File "<doctest default[0]>", line 10, in <module>\n    lumberjack()\n    ^^^^^^^^^^^^\n',
     '  File "<doctest default[0]>", line 4, in lumberjack\n    bright_side_of_death()\n    ^^^^^^^^^^^^^^^^^^^^^^\n',
-    '  File "<doctest default[0]>", line 7, in bright_side_of_death\n    return tuple()[0]\n           ^^^^^^^^^^\n']
+    '  File "<doctest default[0]>", line 7, in bright_side_of_death\n    return tuple()[0]\n           ~~~~~~~^^^\n']
    *** tb_lineno: 10
 
 

Doc/reference/datamodel.rst

@@ -1015,6 +1015,39 @@ Internal types
       If a code object represents a function, the first item in :attr:`co_consts` is
       the documentation string of the function, or ``None`` if undefined.
 
+      .. method:: codeobject.co_positions()
+
+         Returns an iterable over the source code positions of each bytecode
+         instruction in the code object.
+
+         The iterator returns tuples containing the ``(start_line, end_line,
+         start_column, end_column)``. The *i-th* tuple corresponds to the
+         position of the source code that compiled to the *i-th* instruction.
+         Column information is 0-indexed utf-8 byte offsets on the given source
+         line.
+
+         This positional information can be missing. A non-exhaustive lists of
+         cases where this may happen:
+
+         - Running the interpreter with :option:`-X` ``no_debug_ranges``.
+         - Loading a pyc file compiled while using :option:`-X` ``no_debug_ranges``.
+         - Position tuples corresponding to artificial instructions.
+         - Line and column numbers that can't be represented due to
+           implementation specific limitations.
+
+         When this occurs, some or all of the tuple elements can be
+         :const:`None`.
+
+         .. versionadded:: 3.11
+
+         .. note::
+            This feature requires storing column positions in code objects which may
+            result in a small increase of disk usage of compiled Python files or
+            interpreter memory usage. To avoid storing the extra information and/or
+            deactivate printing the extra traceback information, the
+            :option:`-X` ``no_debug_ranges`` command line flag or the :envvar:`PYTHONNODEBUGRANGES`
+            environment variable can be used.
+
    .. _frame-objects:
 
    Frame objects

Doc/whatsnew/3.11.rst

@@ -70,19 +70,97 @@ Summary -- Release highlights
 New Features
 ============
 
-* Asynchronous comprehensions are now allowed inside comprehensions in
-  asynchronous functions. Outer comprehensions implicitly become
-  asynchronous. (Contributed by Serhiy Storchaka in :issue:`33346`.)
+.. _whatsnew311-pep657:
+
+Enhanced error locations in tracebacks
+--------------------------------------
+
+When printing tracebacks, the interpreter will now point to the exact expression
+that caused the error instead of just the line. For example:
+
+.. code-block:: python
+
+    Traceback (most recent call last):
+      File "distance.py", line 11, in <module>
+        print(manhattan_distance(p1, p2))
+              ^^^^^^^^^^^^^^^^^^^^^^^^^^
+      File "distance.py", line 6, in manhattan_distance
+        return abs(point_1.x - point_2.x) + abs(point_1.y - point_2.y)
+                               ^^^^^^^^^
+    AttributeError: 'NoneType' object has no attribute 'x'
+
+Previous versions of the interpreter would point to just the line making it
+ambiguous which object was ``None``. These enhanced errors can also be helpful
+when dealing with deeply nested dictionary objects and multiple function calls,
+
+.. code-block:: python
+
+    Traceback (most recent call last):
+      File "query.py", line 37, in <module>
+        magic_arithmetic('foo')
+        ^^^^^^^^^^^^^^^^^^^^^^^
+      File "query.py", line 18, in magic_arithmetic
+        return add_counts(x) / 25
+               ^^^^^^^^^^^^^
+      File "query.py", line 24, in add_counts
+        return 25 + query_user(user1) + query_user(user2)
+                    ^^^^^^^^^^^^^^^^^
+      File "query.py", line 32, in query_user
+        return 1 + query_count(db, response['a']['b']['c']['user'], retry=True)
+                                   ~~~~~~~~~~~~~~~~~~^^^^^
+    TypeError: 'NoneType' object is not subscriptable
+
+as well as complex arithmetic expressions:
+
+.. code-block:: python
+
+    Traceback (most recent call last):
+      File "calculation.py", line 54, in <module>
+        result = (x / y / z) * (a / b / c)
+                  ~~~~~~^~~
+    ZeroDivisionError: division by zero
+
+See :pep:`657` for more details. (Contributed by Pablo Galindo, Batuhan Taskaya
+and Ammar Askar in :issue:`43950`.)
+
+.. note::
+   This feature requires storing column positions in code objects which may
+   result in a small increase of disk usage of compiled Python files or
+   interpreter memory usage. To avoid storing the extra information and/or
+   deactivate printing the extra traceback information, the
+   :option:`-X` ``no_debug_ranges`` command line flag or the :envvar:`PYTHONNODEBUGRANGES`
+   environment variable can be used.
+
+Column information for code objects
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The information used by the enhanced traceback feature is made available as a
+general API that can be used to correlate bytecode instructions with source
+code. This information can be retrieved using:
+
+- The :meth:`codeobject.co_positions` method in Python.
+- The :c:func:`PyCode_Addr2Location` function in the C-API.
+
+The :option:`-X` ``no_debug_ranges`` option and the environment variable
+:envvar:`PYTHONNODEBUGRANGES` can be used to disable this feature.
+
+See :pep:`657` for more details. (Contributed by Pablo Galindo, Batuhan Taskaya
+and Ammar Askar in :issue:`43950`.)
+
 
 Other Language Changes
 ======================
 
-A :exc:`TypeError` is now raised instead of an :exc:`AttributeError` in
-:meth:`contextlib.ExitStack.enter_context` and
-:meth:`contextlib.AsyncExitStack.enter_async_context` for objects which do not
-support the :term:`context manager` or :term:`asynchronous context manager`
-protocols correspondingly.
-(Contributed by Serhiy Storchaka in :issue:`44471`.)
+* Asynchronous comprehensions are now allowed inside comprehensions in
+  asynchronous functions. Outer comprehensions implicitly become
+  asynchronous. (Contributed by Serhiy Storchaka in :issue:`33346`.)
+
+* A :exc:`TypeError` is now raised instead of an :exc:`AttributeError` in
+  :meth:`contextlib.ExitStack.enter_context` and
+  :meth:`contextlib.AsyncExitStack.enter_async_context` for objects which do not
+  support the :term:`context manager` or :term:`asynchronous context manager`
+  protocols correspondingly.
+  (Contributed by Serhiy Storchaka in :issue:`44471`.)
 
 * A :exc:`TypeError` is now raised instead of an :exc:`AttributeError` in
   :keyword:`with` and :keyword:`async with` statements for objects which do not

Include/funcobject.h

@@ -42,6 +42,14 @@ typedef struct {
     PyObject *func_module;      /* The __module__ attribute, can be anything */
     PyObject *func_annotations; /* Annotations, a dict or NULL */
     vectorcallfunc vectorcall;
+    /* Version number for use by specializer.
+     * Can set to non-zero when we want to specialize.
+     * Will be set to zero if any of these change:
+     *     defaults
+     *     kwdefaults (only if the object changes, not the contents of the dict)
+     *     code
+     *     annotations */
+    uint32_t func_version;
 
     /* Invariant:
      *     func_closure contains the bindings for func_code->co_freevars, so
@@ -74,6 +82,8 @@ PyAPI_FUNC(PyObject *) _PyFunction_Vectorcall(
     PyObject *const *stack,
     size_t nargsf,
     PyObject *kwnames);
+
+uint32_t _PyFunction_GetVersionForCurrentState(PyFunctionObject *func);
 #endif
 
 /* Macros for direct access to these values. Type checks are *not*

Lib/configparser.py

@@ -563,7 +563,7 @@ class RawConfigParser(MutableMapping):
     # Regular expressions for parsing section headers and options
     _SECT_TMPL = r"""
         \[                                 # [
-        (?P<header>[^]]+)                  # very permissive!
+        (?P<header>.+)                     # very permissive!
         \]                                 # ]
         """
     _OPT_TMPL = r"""

Lib/ctypes/test/test_bitfields.py

@@ -233,69 +233,6 @@ class X(Structure):
         else:
             self.assertEqual(sizeof(X), sizeof(c_int) * 2)
 
-    @unittest.skipIf(os.name == 'nt', reason='Posix only')
-    def test_packed_posix(self):
-        test_cases = {
-            (
-                ("a", c_uint8, 4),
-                ("b", c_uint8, 4),
-            ): 1,
-            (
-                ("a", c_uint8, 1),
-                ("b", c_uint16, 1),
-                ("c", c_uint32, 1),
-                ("d", c_uint64, 1),
-            ): 1,
-            (
-                ("a", c_uint8, 8),
-                ("b", c_uint16, 1),
-                ("c", c_uint32, 1),
-                ("d", c_uint64, 1),
-            ): 2,
-            (
-                ("a", c_uint32, 9),
-                ("b", c_uint16, 10),
-                ("c", c_uint32, 25),
-                ("d", c_uint64, 1),
-            ): 6,
-            (
-                ("a", c_uint32, 9),
-                ("b", c_uint16, 10),
-                ("c", c_uint32, 25),
-                ("d", c_uint64, 5),
-            ): 7,
-            (
-                ("a", c_uint16),
-                ("b", c_uint16, 9),
-                ("c", c_uint16, 1),
-                ("d", c_uint16, 1),
-                ("e", c_uint16, 1),
-                ("f", c_uint16, 1),
-                ("g", c_uint16, 3),
-                ("h", c_uint32, 10),
-                ("i", c_uint32, 20),
-                ("j", c_uint32, 2),
-            ): 8,
-            (
-                ("a", c_uint16, 9),
-                ("b", c_uint16, 10),
-                ("d", c_uint16),
-                ("c", c_uint8, 8),
-            ): 6,
-            (
-                ("a", c_uint32, 9),
-                ("b", c_uint32),
-                ("c", c_uint32, 8),
-            ): 7,
-        }
-
-        for fields, size in test_cases.items():
-            with self.subTest(fields=fields):
-                class X(Structure):
-                    _pack_ = 1
-                    _fields_ = list(fields)
-                self.assertEqual(sizeof(X), size)
-
     def test_anon_bitfields(self):
         # anonymous bit-fields gave a strange error message
         class X(Structure):

Lib/inspect.py

@@ -2096,10 +2096,6 @@ def _signature_fromstr(cls, obj, s, skip_bound_arg=True):
     """Private helper to parse content of '__text_signature__'
     and return a Signature based on it.
     """
-    # Lazy import ast because it's relatively heavy and
-    # it's not used for other than this function.
-    import ast
-
     Parameter = cls._parameter_cls
 
     clean_signature, self_parameter, last_positional_only = \

Lib/lib2to3/btm_utils.py

@@ -220,7 +220,6 @@ def reduce_tree(node, parent=None):
             else:
                 #TODO: handle {min, max} repeaters
                 raise NotImplementedError
-                pass
 
         #add children
         if details_node and new_node is not None:

Lib/multiprocessing/managers.py

@@ -1337,7 +1337,6 @@ def __init__(self, *args, **kwargs):
 
         def __del__(self):
             util.debug(f"{self.__class__.__name__}.__del__ by pid {getpid()}")
-            pass
 
         def get_server(self):
             'Better than monkeypatching for now; merge into Server ultimately'

Lib/platform.py

@@ -280,6 +280,7 @@ def _syscmd_ver(system='', release='', version='',
     for cmd in ('ver', 'command /c ver', 'cmd /c ver'):
         try:
             info = subprocess.check_output(cmd,
+                                           stdin=subprocess.DEVNULL,
                                            stderr=subprocess.DEVNULL,
                                            text=True,
                                            shell=True)