diff --git a/Doc/c-api/call.rst b/Doc/c-api/call.rst index 13ef8b217a1622..11d5c33a2d1560 100644 --- a/Doc/c-api/call.rst +++ b/Doc/c-api/call.rst @@ -57,6 +57,15 @@ This bears repeating: A class supporting vectorcall **must** also implement :c:member:`~PyTypeObject.tp_call` with the same semantics. +.. versionchanged:: 3.12 + + The :const:`Py_TPFLAGS_HAVE_VECTORCALL` flag is now removed from a class + when the class's :py:meth:`~object.__call__` method is reassigned. + (This internally sets :c:member:`~PyTypeObject.tp_call` only, and thus + may make it behave differently than the vectorcall function.) + In earlier Python versions, vectorcall should only be used with + :const:`immutable ` or static types. + A class should not implement vectorcall if that would be slower than *tp_call*. For example, if the callee needs to convert the arguments to an args tuple and kwargs dict anyway, then there is no point diff --git a/Doc/c-api/object.rst b/Doc/c-api/object.rst index 07a625bac02fc4..fb03366056b0d2 100644 --- a/Doc/c-api/object.rst +++ b/Doc/c-api/object.rst @@ -126,6 +126,14 @@ Object Protocol A generic implementation for the getter of a ``__dict__`` descriptor. It creates the dictionary if necessary. + This function may also be called to get the :py:attr:`~object.__dict__` + of the object *o*. Pass ``NULL`` for *context* when calling it. + Since this function may need to allocate memory for the + dictionary, it may be more efficient to call :c:func:`PyObject_GetAttr` + when accessing an attribute on the object. + + On failure, returns ``NULL`` with an exception set. + .. versionadded:: 3.3 @@ -137,6 +145,16 @@ Object Protocol .. versionadded:: 3.3 +.. c:function:: PyObject** _PyObject_GetDictPtr(PyObject *obj) + + Return a pointer to :py:attr:`~object.__dict__` of the object *obj*. + If there is no ``__dict__``, return ``NULL`` without setting an exception. + + This function may need to allocate memory for the + dictionary, so it may be more efficient to call :c:func:`PyObject_GetAttr` + when accessing an attribute on the object. + + .. c:function:: PyObject* PyObject_RichCompare(PyObject *o1, PyObject *o2, int opid) Compare the values of *o1* and *o2* using the operation specified by *opid*, diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index 7514801f2d4d59..b8baa7c7dc39ee 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -135,7 +135,7 @@ Quick Reference +------------------------------------------------+-----------------------------------+-------------------+---+---+---+---+ | [:c:member:`~PyTypeObject.tp_cache`] | :c:type:`PyObject` * | | | | | +------------------------------------------------+-----------------------------------+-------------------+---+---+---+---+ - | [:c:member:`~PyTypeObject.tp_subclasses`] | :c:type:`PyObject` * | __subclasses__ | | | | + | [:c:member:`~PyTypeObject.tp_subclasses`] | void * | __subclasses__ | | | | +------------------------------------------------+-----------------------------------+-------------------+---+---+---+---+ | [:c:member:`~PyTypeObject.tp_weaklist`] | :c:type:`PyObject` * | | | | | +------------------------------------------------+-----------------------------------+-------------------+---+---+---+---+ @@ -720,29 +720,29 @@ and :c:type:`PyType_Type` effectively act as defaults.) with the *vectorcallfunc* function. This can be done by setting *tp_call* to :c:func:`PyVectorcall_Call`. - .. warning:: - - It is not recommended for :ref:`mutable heap types ` to implement - the vectorcall protocol. - When a user sets :attr:`__call__` in Python code, only *tp_call* is updated, - likely making it inconsistent with the vectorcall function. - .. versionchanged:: 3.8 Before version 3.8, this slot was named ``tp_print``. In Python 2.x, it was used for printing to a file. In Python 3.0 to 3.7, it was unused. + .. versionchanged:: 3.12 + + Before version 3.12, it was not recommended for + :ref:`mutable heap types ` to implement the vectorcall + protocol. + When a user sets :attr:`~type.__call__` in Python code, only *tp_call* is + updated, likely making it inconsistent with the vectorcall function. + Since 3.12, setting ``__call__`` will disable vectorcall optimization + by clearing the :const:`Py_TPFLAGS_HAVE_VECTORCALL` flag. + **Inheritance:** This field is always inherited. However, the :const:`Py_TPFLAGS_HAVE_VECTORCALL` flag is not - always inherited. If it's not, then the subclass won't use + always inherited. If it's not set, then the subclass won't use :ref:`vectorcall `, except when :c:func:`PyVectorcall_Call` is explicitly called. - This is in particular the case for types without the - :const:`Py_TPFLAGS_IMMUTABLETYPE` flag set (including subclasses defined in - Python). .. c:member:: getattrfunc PyTypeObject.tp_getattr @@ -1178,12 +1178,18 @@ and :c:type:`PyType_Type` effectively act as defaults.) **Inheritance:** - This bit is inherited for types with the - :const:`Py_TPFLAGS_IMMUTABLETYPE` flag set, if - :c:member:`~PyTypeObject.tp_call` is also inherited. + This bit is inherited if :c:member:`~PyTypeObject.tp_call` is also + inherited. .. versionadded:: 3.9 + .. versionchanged:: 3.12 + + This flag is now removed from a class when the class's + :py:meth:`~object.__call__` method is reassigned. + + This flag can now be inherited by mutable classes. + .. data:: Py_TPFLAGS_IMMUTABLETYPE This bit is set for type objects that are immutable: type attributes cannot be set nor deleted. @@ -1709,18 +1715,11 @@ and :c:type:`PyType_Type` effectively act as defaults.) :c:member:`~PyTypeObject.tp_dictoffset` should be set to ``-4`` to indicate that the dictionary is at the very end of the structure. - The real dictionary offset in an instance can be computed from a negative - :c:member:`~PyTypeObject.tp_dictoffset` as follows:: - - dictoffset = tp_basicsize + abs(ob_size)*tp_itemsize + tp_dictoffset - if dictoffset is not aligned on sizeof(void*): - round up to sizeof(void*) - - where :c:member:`~PyTypeObject.tp_basicsize`, :c:member:`~PyTypeObject.tp_itemsize` and :c:member:`~PyTypeObject.tp_dictoffset` are - taken from the type object, and :attr:`ob_size` is taken from the instance. The - absolute value is taken because ints use the sign of :attr:`ob_size` to - store the sign of the number. (There's never a need to do this calculation - yourself; it is done for you by :c:func:`_PyObject_GetDictPtr`.) + The :c:member:`~PyTypeObject.tp_dictoffset` should be regarded as write-only. + To get the pointer to the dictionary call :c:func:`PyObject_GenericGetDict`. + Calling :c:func:`PyObject_GenericGetDict` may need to allocate memory for the + dictionary, so it is may be more efficient to call :c:func:`PyObject_GetAttr` + when accessing an attribute on the object. **Inheritance:** @@ -1928,9 +1927,17 @@ and :c:type:`PyType_Type` effectively act as defaults.) This field is not inherited. -.. c:member:: PyObject* PyTypeObject.tp_subclasses +.. c:member:: void* PyTypeObject.tp_subclasses + + A collection of subclasses. Internal use only. May be an invalid pointer. + + To get a list of subclasses, call the Python method + :py:meth:`~class.__subclasses__`. + + .. versionchanged:: 3.12 - List of weak references to subclasses. Internal use only. + For some types, this field does not hold a valid :c:expr:`PyObject*`. + The type was changed to :c:expr:`void*` to indicate this. **Inheritance:** diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst index 339ee35c7aa474..99afebd762a456 100644 --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -477,9 +477,6 @@ APIs: | | | :c:func:`PyObject_Repr`. | +-------------------+---------------------+----------------------------------+ - An unrecognized format character causes all the rest of the format string to be - copied as-is to the result string, and any extra arguments discarded. - .. note:: The width formatter unit is number of characters rather than bytes. The precision formatter unit is number of bytes for ``"%s"`` and @@ -500,6 +497,11 @@ APIs: Support width and precision formatter for ``"%s"``, ``"%A"``, ``"%U"``, ``"%V"``, ``"%S"``, ``"%R"`` added. + .. versionchanged:: 3.12 + An unrecognized format character now sets a :exc:`SystemError`. + In previous versions it caused all the rest of the format string to be + copied as-is to the result string, and any extra arguments discarded. + .. c:function:: PyObject* PyUnicode_FromFormatV(const char *format, va_list vargs) diff --git a/Doc/data/stable_abi.dat b/Doc/data/stable_abi.dat index 82cd5796efd27d..fde62eacd00a7c 100644 --- a/Doc/data/stable_abi.dat +++ b/Doc/data/stable_abi.dat @@ -783,6 +783,8 @@ function,PyUnicode_WriteChar,3.7,, type,PyVarObject,3.2,,members member,PyVarObject.ob_base,3.2,, member,PyVarObject.ob_size,3.2,, +function,PyVectorcall_Call,3.12,, +function,PyVectorcall_NARGS,3.12,, type,PyWeakReference,3.2,,opaque function,PyWeakref_GetObject,3.2,, function,PyWeakref_NewProxy,3.2,, @@ -883,4 +885,5 @@ type,symtable,3.2,,opaque type,ternaryfunc,3.2,, type,traverseproc,3.2,, type,unaryfunc,3.2,, +type,vectorcallfunc,3.12,, type,visitproc,3.2,, diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index 4f0f8c06fee787..555a0f5cb2a72b 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -332,7 +332,7 @@ Creating Futures and Tasks .. method:: loop.create_task(coro, *, name=None, context=None) - Schedule the execution of a :ref:`coroutine`. + Schedule the execution of :ref:`coroutine ` *coro*. Return a :class:`Task` object. Third-party event loops can use their own subclass of :class:`Task` diff --git a/Doc/library/bisect.rst b/Doc/library/bisect.rst index 513675d3685a52..76045ea511a5e7 100644 --- a/Doc/library/bisect.rst +++ b/Doc/library/bisect.rst @@ -13,10 +13,16 @@ This module provides support for maintaining a list in sorted order without having to sort the list after each insertion. For long lists of items with -expensive comparison operations, this can be an improvement over the more common -approach. The module is called :mod:`bisect` because it uses a basic bisection -algorithm to do its work. The source code may be most useful as a working -example of the algorithm (the boundary conditions are already right!). +expensive comparison operations, this can be an improvement over +linear searches or frequent resorting. + +The module is called :mod:`bisect` because it uses a basic bisection +algorithm to do its work. Unlike other bisection tools that search for a +specific value, the functions in this module are designed to locate an +insertion point. Accordingly, the functions never call an :meth:`__eq__` +method to determine whether a value has been found. Instead, the +functions only call the :meth:`__lt__` method and will return an insertion +point between values in an array. The following functions are provided: @@ -30,16 +36,17 @@ The following functions are provided: any existing entries. The return value is suitable for use as the first parameter to ``list.insert()`` assuming that *a* is already sorted. - The returned insertion point *i* partitions the array *a* into two halves so - that ``all(val < x for val in a[lo : i])`` for the left side and - ``all(val >= x for val in a[i : hi])`` for the right side. + The returned insertion point *ip* partitions the array *a* into two + slices such that ``all(elem < x for elem in a[lo : ip])`` is true for the + left slice and ``all(elem >= x for elem in a[ip : hi])`` is true for the + right slice. *key* specifies a :term:`key function` of one argument that is used to extract a comparison key from each element in the array. To support searching complex records, the key function is not applied to the *x* value. - If *key* is ``None``, the elements are compared directly with no - intervening function call. + If *key* is ``None``, the elements are compared directly and + no key function is called. .. versionchanged:: 3.10 Added the *key* parameter. @@ -51,16 +58,9 @@ The following functions are provided: Similar to :func:`bisect_left`, but returns an insertion point which comes after (to the right of) any existing entries of *x* in *a*. - The returned insertion point *i* partitions the array *a* into two halves so - that ``all(val <= x for val in a[lo : i])`` for the left side and - ``all(val > x for val in a[i : hi])`` for the right side. - - *key* specifies a :term:`key function` of one argument that is used to - extract a comparison key from each element in the array. To support - searching complex records, the key function is not applied to the *x* value. - - If *key* is ``None``, the elements are compared directly with no - intervening function call. + The returned insertion point *ip* partitions the array *a* into two slices + such that ``all(elem <= x for elem in a[lo : ip])`` is true for the left slice and + ``all(elem > x for elem in a[ip : hi])`` is true for the right slice. .. versionchanged:: 3.10 Added the *key* parameter. diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index 68c3d1c0a4b24b..63b064e7b444ec 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -408,6 +408,24 @@ The Python compiler currently generates the following bytecode instructions. .. versionadded:: 3.11 +.. opcode:: CACHE + + Rather than being an actual instruction, this opcode is used to mark extra + space for the interpreter to cache useful data directly in the bytecode + itself. It is automatically hidden by all ``dis`` utilities, but can be + viewed with ``show_caches=True``. + + Logically, this space is part of the preceding instruction. Many opcodes + expect to be followed by an exact number of caches, and will instruct the + interpreter to skip over them at runtime. + + Populated caches can look like arbitrary instructions, so great care should + be taken when reading or modifying raw, adaptive bytecode containing + quickened data. + + .. versionadded:: 3.11 + + **Unary operations** Unary operations take the top of the stack, apply the operation, and push the diff --git a/Doc/library/email.rst b/Doc/library/email.rst index 5eebcd9e896d93..816fae991d24cb 100644 --- a/Doc/library/email.rst +++ b/Doc/library/email.rst @@ -147,6 +147,3 @@ Legacy API: Module :mod:`mailbox` Tools for creating, reading, and managing collections of messages on disk using a variety standard formats. - - Module :mod:`smtpd` - SMTP server framework (primarily useful for testing) diff --git a/Doc/library/functools.rst b/Doc/library/functools.rst index dd4d76ef670987..00aca09bc7af45 100644 --- a/Doc/library/functools.rst +++ b/Doc/library/functools.rst @@ -119,7 +119,7 @@ The :mod:`functools` module defines the following functions: tool for programs being converted from Python 2 which supported the use of comparison functions. - A comparison function is any callable that accept two arguments, compares them, + A comparison function is any callable that accepts two arguments, compares them, and returns a negative number for less-than, zero for equality, or a positive number for greater-than. A key function is a callable that accepts one argument and returns another value to be used as the sort key. diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst index e91ec40c9add0d..81e0182e10bef8 100644 --- a/Doc/library/idle.rst +++ b/Doc/library/idle.rst @@ -61,17 +61,17 @@ New File Open... Open an existing file with an Open dialog. -Recent Files - Open a list of recent files. Click one to open it. - Open Module... Open an existing module (searches sys.path). +Recent Files + Open a list of recent files. Click one to open it. + .. index:: - single: Class browser + single: Module browser single: Path browser -Class Browser +Module Browser Show functions, classes, and methods in the current Editor file in a tree structure. In the shell, open a module first. @@ -87,11 +87,14 @@ Save Save As... Save the current window with a Save As dialog. The file saved becomes the - new associated file for the window. + new associated file for the window. (If your file namager is set to hide + extensions, the current extension will be omitted in the file name box. + If the new filename has no '.', '.py' and '.txt' will be added for Python + and text files, except that on macOS Aqua,'.py' is added for all files.) Save Copy As... Save the current window to different file without changing the associated - file. + file. (See Save As note above about filename extensions.) Print Window Print the current window to the default printer. @@ -114,6 +117,9 @@ Undo Redo Redo the last undone change to the current window. +Select All + Select the entire contents of the current window. + Cut Copy selection into the system-wide clipboard; then delete the selection. @@ -125,9 +131,6 @@ Paste The clipboard functions are also available in context menus. -Select All - Select the entire contents of the current window. - Find... Open a search dialog with many options @@ -156,12 +159,12 @@ Expand Word Expand a prefix you have typed to match a full word in the same window; repeat to get a different expansion. -Show call tip +Show Call Tip After an unclosed parenthesis for a function, open a small window with function parameter hints. See :ref:`Calltips ` in the Editing and navigation section below. -Show surrounding parens +Show Surrounding Parens Highlight the surrounding parenthesis. .. _format-menu: @@ -169,6 +172,11 @@ Show surrounding parens Format menu (Editor window only) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Format Paragraph + Reformat the current blank-line-delimited paragraph in comment block or + multiline string or selected line in a string. All lines in the + paragraph will be formatted to less than N columns, where N defaults to 72. + Indent Region Shift selected lines right by the indent width (default 4 spaces). @@ -195,12 +203,7 @@ New Indent Width Open a dialog to change indent width. The accepted default by the Python community is 4 spaces. -Format Paragraph - Reformat the current blank-line-delimited paragraph in comment block or - multiline string or selected line in a string. All lines in the - paragraph will be formatted to less than N columns, where N defaults to 72. - -Strip trailing whitespace +Strip Trailing Chitespace Remove trailing space and other whitespace characters after the last non-whitespace character of a line by applying str.rstrip to each line, including lines within multiline strings. Except for Shell windows, @@ -471,6 +474,14 @@ are restricted to four spaces due to Tcl/Tk limitations. See also the indent/dedent region commands on the :ref:`Format menu `. +Search and Replace +^^^^^^^^^^^^^^^^^^ + +Any selection becomes a search target. However, only selections within +a line work because searches are only performed within lines with the +terminal newline removed. If ``[x] Regular expresion`` is checked, the +target is interpreted according to the Python re module. + .. _completions: Completions @@ -972,3 +983,20 @@ changed with the Extensions tab of the preferences dialog. See the beginning of config-extensions.def in the idlelib directory for further information. The only current default extension is zzdummy, an example also used for testing. + +idlelib +^^^^^^^ + +.. module:: idlelib + :synopsis: Implementation package for the IDLE shell/editor. + +**Source code:** :source:`Lib/idlelib` + +The Lib/idlelib package implements the IDLE application. See the top +of this file or content listing on the left for how to use IDLE. + +The files in idlelib are described in idlelib/README.txt. Access it +either in idlelib or click Help => About IDLE on the IDLE menu. This +file also maps IDLE menu items to the code that implements the item. +Except for files listed under 'Startup', the idlelib code is 'private' in +sense that feature changes can be backported (see :pep:`434`). diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index c29d69c143cfe6..0fd765f5985f7c 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -493,20 +493,6 @@ ABC hierarchy:: other responsibilities of :meth:`load_module` when :meth:`exec_module` is implemented. - .. method:: module_repr(module) - - A legacy method which when implemented calculates and returns the given - module's representation, as a string. The module type's default - :meth:`__repr__` will use the result of this method as appropriate. - - .. versionadded:: 3.3 - - .. versionchanged:: 3.4 - Made optional instead of an abstractmethod. - - .. deprecated:: 3.4 - The import machinery now takes care of this automatically. - .. class:: ResourceLoader diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst index 19944bd7bd0a89..f7d7745eef52e7 100644 --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -876,10 +876,15 @@ call fails (for example because the path doesn't exist). function checks whether *path*'s parent, :file:`path/..`, is on a different device than *path*, or whether :file:`path/..` and *path* point to the same i-node on the same device --- this should detect mount points for all Unix - and POSIX variants. Not implemented on Windows. + and POSIX variants. On Windows, a mount point is considered to be a drive + letter root (e.g. ``c:\``), a UNC share (e.g. ``\\server\share``), or a + mounted filesystem directory. .. versionadded:: 3.7 + .. versionchanged:: 3.12 + Windows support was added. + .. method:: Path.is_symlink() diff --git a/Doc/library/smtpd.rst b/Doc/library/smtpd.rst deleted file mode 100644 index ac0c9aeb236091..00000000000000 --- a/Doc/library/smtpd.rst +++ /dev/null @@ -1,268 +0,0 @@ -:mod:`smtpd` --- SMTP Server -============================ - -.. module:: smtpd - :synopsis: A SMTP server implementation in Python. - :deprecated: - -.. moduleauthor:: Barry Warsaw -.. sectionauthor:: Moshe Zadka - -**Source code:** :source:`Lib/smtpd.py` - --------------- - -This module offers several classes to implement SMTP (email) servers. - -.. deprecated-removed:: 3.6 3.12 - The :mod:`smtpd` module is deprecated - (see :pep:`PEP 594 <594#smtpd>` for details). - The `aiosmtpd `_ package is a recommended - replacement for this module. It is based on :mod:`asyncio` and provides a - more straightforward API. - -Several server implementations are present; one is a generic -do-nothing implementation, which can be overridden, while the other two offer -specific mail-sending strategies. - -Additionally the SMTPChannel may be extended to implement very specific -interaction behaviour with SMTP clients. - -The code supports :RFC:`5321`, plus the :rfc:`1870` SIZE and :rfc:`6531` -SMTPUTF8 extensions. - -.. include:: ../includes/wasm-notavail.rst - -SMTPServer Objects ------------------- - - -.. class:: SMTPServer(localaddr, remoteaddr, data_size_limit=33554432,\ - map=None, enable_SMTPUTF8=False, decode_data=False) - - Create a new :class:`SMTPServer` object, which binds to local address - *localaddr*. It will treat *remoteaddr* as an upstream SMTP relayer. Both - *localaddr* and *remoteaddr* should be a :ref:`(host, port) ` - tuple. The object inherits from :class:`asyncore.dispatcher`, and so will - insert itself into :mod:`asyncore`'s event loop on instantiation. - - *data_size_limit* specifies the maximum number of bytes that will be - accepted in a ``DATA`` command. A value of ``None`` or ``0`` means no - limit. - - *map* is the socket map to use for connections (an initially empty - dictionary is a suitable value). If not specified the :mod:`asyncore` - global socket map is used. - - *enable_SMTPUTF8* determines whether the ``SMTPUTF8`` extension (as defined - in :RFC:`6531`) should be enabled. The default is ``False``. - When ``True``, ``SMTPUTF8`` is accepted as a parameter to the ``MAIL`` - command and when present is passed to :meth:`process_message` in the - ``kwargs['mail_options']`` list. *decode_data* and *enable_SMTPUTF8* - cannot be set to ``True`` at the same time. - - *decode_data* specifies whether the data portion of the SMTP transaction - should be decoded using UTF-8. When *decode_data* is ``False`` (the - default), the server advertises the ``8BITMIME`` - extension (:rfc:`6152`), accepts the ``BODY=8BITMIME`` parameter to - the ``MAIL`` command, and when present passes it to :meth:`process_message` - in the ``kwargs['mail_options']`` list. *decode_data* and *enable_SMTPUTF8* - cannot be set to ``True`` at the same time. - - .. method:: process_message(peer, mailfrom, rcpttos, data, **kwargs) - - Raise a :exc:`NotImplementedError` exception. Override this in subclasses to - do something useful with this message. Whatever was passed in the - constructor as *remoteaddr* will be available as the :attr:`_remoteaddr` - attribute. *peer* is the remote host's address, *mailfrom* is the envelope - originator, *rcpttos* are the envelope recipients and *data* is a string - containing the contents of the e-mail (which should be in :rfc:`5321` - format). - - If the *decode_data* constructor keyword is set to ``True``, the *data* - argument will be a unicode string. If it is set to ``False``, it - will be a bytes object. - - *kwargs* is a dictionary containing additional information. It is empty - if ``decode_data=True`` was given as an init argument, otherwise - it contains the following keys: - - *mail_options*: - a list of all received parameters to the ``MAIL`` - command (the elements are uppercase strings; example: - ``['BODY=8BITMIME', 'SMTPUTF8']``). - - *rcpt_options*: - same as *mail_options* but for the ``RCPT`` command. - Currently no ``RCPT TO`` options are supported, so for now - this will always be an empty list. - - Implementations of ``process_message`` should use the ``**kwargs`` - signature to accept arbitrary keyword arguments, since future feature - enhancements may add keys to the kwargs dictionary. - - Return ``None`` to request a normal ``250 Ok`` response; otherwise - return the desired response string in :RFC:`5321` format. - - .. attribute:: channel_class - - Override this in subclasses to use a custom :class:`SMTPChannel` for - managing SMTP clients. - - .. versionadded:: 3.4 - The *map* constructor argument. - - .. versionchanged:: 3.5 - *localaddr* and *remoteaddr* may now contain IPv6 addresses. - - .. versionadded:: 3.5 - The *decode_data* and *enable_SMTPUTF8* constructor parameters, and the - *kwargs* parameter to :meth:`process_message` when *decode_data* is - ``False``. - - .. versionchanged:: 3.6 - *decode_data* is now ``False`` by default. - - -DebuggingServer Objects ------------------------ - - -.. class:: DebuggingServer(localaddr, remoteaddr) - - Create a new debugging server. Arguments are as per :class:`SMTPServer`. - Messages will be discarded, and printed on stdout. - - -PureProxy Objects ------------------ - - -.. class:: PureProxy(localaddr, remoteaddr) - - Create a new pure proxy server. Arguments are as per :class:`SMTPServer`. - Everything will be relayed to *remoteaddr*. Note that running this has a good - chance to make you into an open relay, so please be careful. - - -SMTPChannel Objects -------------------- - -.. class:: SMTPChannel(server, conn, addr, data_size_limit=33554432,\ - map=None, enable_SMTPUTF8=False, decode_data=False) - - Create a new :class:`SMTPChannel` object which manages the communication - between the server and a single SMTP client. - - *conn* and *addr* are as per the instance variables described below. - - *data_size_limit* specifies the maximum number of bytes that will be - accepted in a ``DATA`` command. A value of ``None`` or ``0`` means no - limit. - - *enable_SMTPUTF8* determines whether the ``SMTPUTF8`` extension (as defined - in :RFC:`6531`) should be enabled. The default is ``False``. - *decode_data* and *enable_SMTPUTF8* cannot be set to ``True`` at the same - time. - - A dictionary can be specified in *map* to avoid using a global socket map. - - *decode_data* specifies whether the data portion of the SMTP transaction - should be decoded using UTF-8. The default is ``False``. - *decode_data* and *enable_SMTPUTF8* cannot be set to ``True`` at the same - time. - - To use a custom SMTPChannel implementation you need to override the - :attr:`SMTPServer.channel_class` of your :class:`SMTPServer`. - - .. versionchanged:: 3.5 - The *decode_data* and *enable_SMTPUTF8* parameters were added. - - .. versionchanged:: 3.6 - *decode_data* is now ``False`` by default. - - The :class:`SMTPChannel` has the following instance variables: - - .. attribute:: smtp_server - - Holds the :class:`SMTPServer` that spawned this channel. - - .. attribute:: conn - - Holds the socket object connecting to the client. - - .. attribute:: addr - - Holds the address of the client, the second value returned by - :func:`socket.accept ` - - .. attribute:: received_lines - - Holds a list of the line strings (decoded using UTF-8) received from - the client. The lines have their ``"\r\n"`` line ending translated to - ``"\n"``. - - .. attribute:: smtp_state - - Holds the current state of the channel. This will be either - :attr:`COMMAND` initially and then :attr:`DATA` after the client sends - a "DATA" line. - - .. attribute:: seen_greeting - - Holds a string containing the greeting sent by the client in its "HELO". - - .. attribute:: mailfrom - - Holds a string containing the address identified in the "MAIL FROM:" line - from the client. - - .. attribute:: rcpttos - - Holds a list of strings containing the addresses identified in the - "RCPT TO:" lines from the client. - - .. attribute:: received_data - - Holds a string containing all of the data sent by the client during the - DATA state, up to but not including the terminating ``"\r\n.\r\n"``. - - .. attribute:: fqdn - - Holds the fully qualified domain name of the server as returned by - :func:`socket.getfqdn`. - - .. attribute:: peer - - Holds the name of the client peer as returned by ``conn.getpeername()`` - where ``conn`` is :attr:`conn`. - - The :class:`SMTPChannel` operates by invoking methods named ``smtp_`` - upon reception of a command line from the client. Built into the base - :class:`SMTPChannel` class are methods for handling the following commands - (and responding to them appropriately): - - ======== =================================================================== - Command Action taken - ======== =================================================================== - HELO Accepts the greeting from the client and stores it in - :attr:`seen_greeting`. Sets server to base command mode. - EHLO Accepts the greeting from the client and stores it in - :attr:`seen_greeting`. Sets server to extended command mode. - NOOP Takes no action. - QUIT Closes the connection cleanly. - MAIL Accepts the "MAIL FROM:" syntax and stores the supplied address as - :attr:`mailfrom`. In extended command mode, accepts the - :rfc:`1870` SIZE attribute and responds appropriately based on the - value of *data_size_limit*. - RCPT Accepts the "RCPT TO:" syntax and stores the supplied addresses in - the :attr:`rcpttos` list. - RSET Resets the :attr:`mailfrom`, :attr:`rcpttos`, and - :attr:`received_data`, but not the greeting. - DATA Sets the internal state to :attr:`DATA` and stores remaining lines - from the client in :attr:`received_data` until the terminator - ``"\r\n.\r\n"`` is received. - HELP Returns minimal information on command syntax - VRFY Returns code 252 (the server doesn't know if the address is valid) - EXPN Reports that the command is not implemented. - ======== =================================================================== diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index 3fade30c2562d3..06ed7af052f00e 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -34,6 +34,18 @@ This document includes four main sections: * :ref:`sqlite3-explanation` provides in-depth background on transaction control. +.. seealso:: + + https://www.sqlite.org + The SQLite web page; the documentation describes the syntax and the + available data types for the supported SQL dialect. + + https://www.w3schools.com/sql/ + Tutorial, reference and examples for learning SQL syntax. + + :pep:`249` - Database API Specification 2.0 + PEP written by Marc-André Lemburg. + .. _sqlite3-tutorial: @@ -94,6 +106,12 @@ using :meth:`~Cursor.executemany`:: ... ] >>> cur.executemany('INSERT INTO stocks VALUES(?, ?, ?, ?, ?)', data) +Notice that we used ``?`` placeholders to bind *data* to the query. +Always use placeholders instead of :ref:`string formatting ` +to bind Python values to SQL statements, +to avoid `SQL injection attacks`_. +See the :ref:`placeholders how-to ` for more details. + Then, retrieve the data by iterating over the result of a ``SELECT`` statement:: >>> for row in cur.execute('SELECT * FROM stocks ORDER BY price'): @@ -104,45 +122,9 @@ Then, retrieve the data by iterating over the result of a ``SELECT`` statement:: ('2006-04-06', 'SELL', 'IBM', 500, 53.0) ('2006-04-05', 'BUY', 'MSFT', 1000, 72.0) +You've now created an SQLite database using the :mod:`!sqlite3` module. -.. _sqlite3-placeholders: - -SQL operations usually need to use values from Python variables. However, -beware of using Python's string operations to assemble queries, as they -are vulnerable to SQL injection attacks (see the `xkcd webcomic -`_ for a humorous example of what can go wrong):: - - # Never do this -- insecure! - symbol = 'RHAT' - cur.execute("SELECT * FROM stocks WHERE symbol = '%s'" % symbol) - -Instead, use the DB-API's parameter substitution. To insert a variable into a -query string, use a placeholder in the string, and substitute the actual values -into the query by providing them as a :class:`tuple` of values to the second -argument of the cursor's :meth:`~Cursor.execute` method. An SQL statement may -use one of two kinds of placeholders: question marks (qmark style) or named -placeholders (named style). For the qmark style, ``parameters`` must be a -:term:`sequence `. For the named style, it can be either a -:term:`sequence ` or :class:`dict` instance. The length of the -:term:`sequence ` must match the number of placeholders, or a -:exc:`ProgrammingError` is raised. If a :class:`dict` is given, it must contain -keys for all named parameters. Any extra items are ignored. Here's an example of -both styles: - -.. literalinclude:: ../includes/sqlite3/execute_1.py - - -.. seealso:: - - https://www.sqlite.org - The SQLite web page; the documentation describes the syntax and the - available data types for the supported SQL dialect. - - https://www.w3schools.com/sql/ - Tutorial, reference and examples for learning SQL syntax. - - :pep:`249` - Database API Specification 2.0 - PEP written by Marc-André Lemburg. +.. _SQL injection attacks: https://en.wikipedia.org/wiki/SQL_injection .. _sqlite3-reference: @@ -150,142 +132,18 @@ both styles: Reference --------- +.. We keep the old sqlite3-module-contents ref to prevent breaking links. .. _sqlite3-module-contents: -Module functions and constants -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - - -.. data:: apilevel - - String constant stating the supported DB-API level. Required by the DB-API. - Hard-coded to ``"2.0"``. - -.. data:: paramstyle - - String constant stating the type of parameter marker formatting expected by - the :mod:`sqlite3` module. Required by the DB-API. Hard-coded to - ``"qmark"``. - - .. note:: - - The :mod:`sqlite3` module supports both ``qmark`` and ``numeric`` DB-API - parameter styles, because that is what the underlying SQLite library - supports. However, the DB-API does not allow multiple values for - the ``paramstyle`` attribute. - -.. data:: version - - Version number of this module as a :class:`string `. - This is not the version of the SQLite library. - - .. deprecated-removed:: 3.12 3.14 - This constant used to reflect the version number of the ``pysqlite`` - package, a third-party library which used to upstream changes to - ``sqlite3``. Today, it carries no meaning or practical value. - - -.. data:: version_info - - Version number of this module as a :class:`tuple` of :class:`integers `. - This is not the version of the SQLite library. - - .. deprecated-removed:: 3.12 3.14 - This constant used to reflect the version number of the ``pysqlite`` - package, a third-party library which used to upstream changes to - ``sqlite3``. Today, it carries no meaning or practical value. - - -.. data:: sqlite_version - - Version number of the runtime SQLite library as a :class:`string `. - - -.. data:: sqlite_version_info +.. _sqlite3-module-functions: - Version number of the runtime SQLite library as a :class:`tuple` of - :class:`integers `. +Module functions +^^^^^^^^^^^^^^^^ - -.. data:: threadsafety - - Integer constant required by the DB-API 2.0, stating the level of thread - safety the :mod:`sqlite3` module supports. This attribute is set based on - the default `threading mode `_ the - underlying SQLite library is compiled with. The SQLite threading modes are: - - 1. **Single-thread**: In this mode, all mutexes are disabled and SQLite is - unsafe to use in more than a single thread at once. - 2. **Multi-thread**: In this mode, SQLite can be safely used by multiple - threads provided that no single database connection is used - simultaneously in two or more threads. - 3. **Serialized**: In serialized mode, SQLite can be safely used by - multiple threads with no restriction. - - The mappings from SQLite threading modes to DB-API 2.0 threadsafety levels - are as follows: - - +------------------+-----------------+----------------------+-------------------------------+ - | SQLite threading | `threadsafety`_ | `SQLITE_THREADSAFE`_ | DB-API 2.0 meaning | - | mode | | | | - +==================+=================+======================+===============================+ - | single-thread | 0 | 0 | Threads may not share the | - | | | | module | - +------------------+-----------------+----------------------+-------------------------------+ - | multi-thread | 1 | 2 | Threads may share the module, | - | | | | but not connections | - +------------------+-----------------+----------------------+-------------------------------+ - | serialized | 3 | 1 | Threads may share the module, | - | | | | connections and cursors | - +------------------+-----------------+----------------------+-------------------------------+ - - .. _threadsafety: https://peps.python.org/pep-0249/#threadsafety - .. _SQLITE_THREADSAFE: https://sqlite.org/compile.html#threadsafe - - .. versionchanged:: 3.11 - Set *threadsafety* dynamically instead of hard-coding it to ``1``. - -.. data:: PARSE_DECLTYPES - - Pass this flag value to the *detect_types* parameter of - :func:`connect` to look up a converter function using - the declared types for each column. - The types are declared when the database table is created. - ``sqlite3`` will look up a converter function using the first word of the - declared type as the converter dictionary key. - For example: - - - .. code-block:: sql - - CREATE TABLE test( - i integer primary key, ! will look up a converter named "integer" - p point, ! will look up a converter named "point" - n number(10) ! will look up a converter named "number" - ) - - This flag may be combined with :const:`PARSE_COLNAMES` using the ``|`` - (bitwise or) operator. - - -.. data:: PARSE_COLNAMES - - Pass this flag value to the *detect_types* parameter of - :func:`connect` to look up a converter function by - using the type name, parsed from the query column name, - as the converter dictionary key. - The type name must be wrapped in square brackets (``[]``). - - .. code-block:: sql - - SELECT p as "p [point]" FROM test; ! will look up converter "point" - - This flag may be combined with :const:`PARSE_DECLTYPES` using the ``|`` - (bitwise or) operator. - - - -.. function:: connect(database, timeout=5.0, detect_types=0, isolation_level="DEFERRED", check_same_thread=True, factory=sqlite3.Connection, cached_statements=128, uri=False) +.. function:: connect(database, timeout=5.0, detect_types=0, \ + isolation_level="DEFERRED", check_same_thread=True, \ + factory=sqlite3.Connection, cached_statements=128, \ + uri=False) Open a connection to an SQLite database. @@ -295,15 +153,14 @@ Module functions and constants in RAM instead of on disk. :type database: :term:`path-like object` - :param timeout: + :param float timeout: How many seconds the connection should wait before raising an exception, if the database is locked by another connection. If another connection opens a transaction to modify the database, it will be locked until that transaction is committed. Default five seconds. - :type timeout: float - :param detect_types: + :param int detect_types: Control whether and how data types not :ref:`natively supported by SQLite ` are looked up to be converted to Python types, @@ -316,7 +173,6 @@ Module functions and constants even when the *detect_types* parameter is set; :class:`str` will be returned instead. By default (``0``), type detection is disabled. - :type detect_types: int :param isolation_level: The :attr:`~Connection.isolation_level` of the connection, @@ -326,25 +182,22 @@ Module functions and constants See :ref:`sqlite3-controlling-transactions` for more. :type isolation_level: str | None - :param check_same_thread: + :param bool check_same_thread: If ``True`` (default), only the creating thread may use the connection. If ``False``, the connection may be shared across multiple threads; if so, write operations should be serialized by the user to avoid data corruption. - :type check_same_thread: bool - :param factory: + :param Connection factory: A custom subclass of :class:`Connection` to create the connection with, if not the default :class:`Connection` class. - :type factory: :class:`Connection` - :param cached_statements: - The number of statements that ``sqlite3`` + :param int cached_statements: + The number of statements that :mod:`!sqlite3` should internally cache for this connection, to avoid parsing overhead. By default, 128 statements. - :type cached_statements: int - :param uri: + :param bool uri: If set to ``True``, *database* is interpreted as a :abbr:`URI (Uniform Resource Identifier)` with a file path and an optional query string. @@ -352,7 +205,6 @@ Module functions and constants and the path can be relative or absolute. The query string allows passing parameters to SQLite, enabling various :ref:`sqlite3-uri-tricks`. - :type uri: bool :rtype: Connection @@ -368,30 +220,6 @@ Module functions and constants .. versionadded:: 3.10 The ``sqlite3.connect/handle`` auditing event. - -.. function:: register_converter(typename, converter, /) - - Register the *converter* callable to convert SQLite objects of type - *typename* into a Python object of a specific type. - The converter is invoked for all SQLite values of type *typename*; - it is passed a :class:`bytes` object and should return an object of the - desired Python type. - Consult the parameter *detect_types* of - :func:`connect` for information regarding how type detection works. - - Note: *typename* and the name of the type in your query are matched - case-insensitively. - - -.. function:: register_adapter(type, adapter, /) - - Register an *adapter* callable to adapt the Python type *type* into an - SQLite type. - The adapter is called with a Python object of type *type* as its sole - argument, and must return a value of a - :ref:`type that SQLite natively understands`. - - .. function:: complete_statement(statement) Returns ``True`` if the string *statement* contains one or more complete SQL @@ -401,10 +229,8 @@ Module functions and constants This can be used to build a shell for SQLite, as in the following example: - .. literalinclude:: ../includes/sqlite3/complete_statement.py - .. function:: enable_callback_tracebacks(flag, /) Enable or disable callback tracebacks. @@ -432,6 +258,154 @@ Module functions and constants UnraisableHookArgs(exc_type=, exc_value=ZeroDivisionError('division by zero'), exc_traceback=, err_msg=None, object= at 0x10b4e3ee0>) +.. function:: register_adapter(type, adapter, /) + + Register an *adapter* callable to adapt the Python type *type* into an + SQLite type. + The adapter is called with a Python object of type *type* as its sole + argument, and must return a value of a + :ref:`type that SQLite natively understands `. + +.. function:: register_converter(typename, converter, /) + + Register the *converter* callable to convert SQLite objects of type + *typename* into a Python object of a specific type. + The converter is invoked for all SQLite values of type *typename*; + it is passed a :class:`bytes` object and should return an object of the + desired Python type. + Consult the parameter *detect_types* of + :func:`connect` for information regarding how type detection works. + + Note: *typename* and the name of the type in your query are matched + case-insensitively. + + +.. _sqlite3-module-constants: + +Module constants +^^^^^^^^^^^^^^^^ + +.. data:: PARSE_COLNAMES + + Pass this flag value to the *detect_types* parameter of + :func:`connect` to look up a converter function by + using the type name, parsed from the query column name, + as the converter dictionary key. + The type name must be wrapped in square brackets (``[]``). + + .. code-block:: sql + + SELECT p as "p [point]" FROM test; ! will look up converter "point" + + This flag may be combined with :const:`PARSE_DECLTYPES` using the ``|`` + (bitwise or) operator. + +.. data:: PARSE_DECLTYPES + + Pass this flag value to the *detect_types* parameter of + :func:`connect` to look up a converter function using + the declared types for each column. + The types are declared when the database table is created. + :mod:`!sqlite3` will look up a converter function using the first word of the + declared type as the converter dictionary key. + For example: + + .. code-block:: sql + + CREATE TABLE test( + i integer primary key, ! will look up a converter named "integer" + p point, ! will look up a converter named "point" + n number(10) ! will look up a converter named "number" + ) + + This flag may be combined with :const:`PARSE_COLNAMES` using the ``|`` + (bitwise or) operator. + +.. data:: apilevel + + String constant stating the supported DB-API level. Required by the DB-API. + Hard-coded to ``"2.0"``. + +.. data:: paramstyle + + String constant stating the type of parameter marker formatting expected by + the :mod:`!sqlite3` module. Required by the DB-API. Hard-coded to + ``"qmark"``. + + .. note:: + + The :mod:`!sqlite3` module supports both ``qmark`` and ``numeric`` DB-API + parameter styles, because that is what the underlying SQLite library + supports. However, the DB-API does not allow multiple values for + the ``paramstyle`` attribute. + +.. data:: sqlite_version + + Version number of the runtime SQLite library as a :class:`string `. + +.. data:: sqlite_version_info + + Version number of the runtime SQLite library as a :class:`tuple` of + :class:`integers `. + +.. data:: threadsafety + + Integer constant required by the DB-API 2.0, stating the level of thread + safety the :mod:`!sqlite3` module supports. This attribute is set based on + the default `threading mode `_ the + underlying SQLite library is compiled with. The SQLite threading modes are: + + 1. **Single-thread**: In this mode, all mutexes are disabled and SQLite is + unsafe to use in more than a single thread at once. + 2. **Multi-thread**: In this mode, SQLite can be safely used by multiple + threads provided that no single database connection is used + simultaneously in two or more threads. + 3. **Serialized**: In serialized mode, SQLite can be safely used by + multiple threads with no restriction. + + The mappings from SQLite threading modes to DB-API 2.0 threadsafety levels + are as follows: + + +------------------+-----------------+----------------------+-------------------------------+ + | SQLite threading | `threadsafety`_ | `SQLITE_THREADSAFE`_ | DB-API 2.0 meaning | + | mode | | | | + +==================+=================+======================+===============================+ + | single-thread | 0 | 0 | Threads may not share the | + | | | | module | + +------------------+-----------------+----------------------+-------------------------------+ + | multi-thread | 1 | 2 | Threads may share the module, | + | | | | but not connections | + +------------------+-----------------+----------------------+-------------------------------+ + | serialized | 3 | 1 | Threads may share the module, | + | | | | connections and cursors | + +------------------+-----------------+----------------------+-------------------------------+ + + .. _threadsafety: https://peps.python.org/pep-0249/#threadsafety + .. _SQLITE_THREADSAFE: https://sqlite.org/compile.html#threadsafe + + .. versionchanged:: 3.11 + Set *threadsafety* dynamically instead of hard-coding it to ``1``. + +.. data:: version + + Version number of this module as a :class:`string `. + This is not the version of the SQLite library. + + .. deprecated-removed:: 3.12 3.14 + This constant used to reflect the version number of the ``pysqlite`` + package, a third-party library which used to upstream changes to + :mod:`!sqlite3`. Today, it carries no meaning or practical value. + +.. data:: version_info + + Version number of this module as a :class:`tuple` of :class:`integers `. + This is not the version of the SQLite library. + + .. deprecated-removed:: 3.12 3.14 + This constant used to reflect the version number of the ``pysqlite`` + package, a third-party library which used to upstream changes to + :mod:`!sqlite3`. Today, it carries no meaning or practical value. + .. _sqlite3-connection-objects: @@ -455,7 +429,7 @@ Connection objects .. attribute:: isolation_level This attribute controls the :ref:`transaction handling - ` performed by ``sqlite3``. + ` performed by :mod:`!sqlite3`. If set to ``None``, transactions are never implicitly opened. If set to one of ``"DEFERRED"``, ``"IMMEDIATE"``, or ``"EXCLUSIVE"``, corresponding to the underlying `SQLite transaction behaviour`_, @@ -487,28 +461,23 @@ Connection objects Open a :class:`Blob` handle to an existing :abbr:`BLOB (Binary Large OBject)`. - :param table: + :param str table: The name of the table where the blob is located. - :type table: str - :param column: + :param str column: The name of the column where the blob is located. - :type column: str - :param row: + :param str row: The name of the row where the blob is located. - :type row: str - :param readonly: + :param bool readonly: Set to ``True`` if the blob should be opened without write permissions. Defaults to ``False``. - :type readonly: bool - :param name: + :param str name: The name of the database where the blob is located. Defaults to ``"main"``. - :type name: str :raises OperationalError: When trying to open a blob in a ``WITHOUT ROWID`` table. @@ -561,14 +530,12 @@ Connection objects Create or remove a user-defined SQL function. - :param name: + :param str name: The name of the SQL function. - :type name: str - :param narg: + :param int narg: The number of arguments the SQL function can accept. If ``-1``, it may take any number of arguments. - :type narg: int :param func: A callable that is called when the SQL function is invoked. @@ -577,11 +544,10 @@ Connection objects Set to ``None`` to remove an existing SQL function. :type func: :term:`callback` | None - :param deterministic: + :param bool deterministic: If ``True``, the created SQL function is marked as `deterministic `_, which allows SQLite to perform additional optimizations. - :type deterministic: bool :raises NotSupportedError: If *deterministic* is used with SQLite versions older than 3.8.3. @@ -598,14 +564,12 @@ Connection objects Create or remove a user-defined SQL aggregate function. - :param name: + :param str name: The name of the SQL aggregate function. - :type name: str - :param n_arg: + :param int n_arg: The number of arguments the SQL aggregate function can accept. If ``-1``, it may take any number of arguments. - :type n_arg: int :param aggregate_class: A class must implement the following methods: @@ -629,14 +593,12 @@ Connection objects Create or remove a user-defined aggregate window function. - :param name: + :param str name: The name of the SQL aggregate window function to create or remove. - :type name: str - :param num_params: + :param int num_params: The number of arguments the SQL aggregate window function can accept. If ``-1``, it may take any number of arguments. - :type num_params: int :param aggregate_class: A class that must implement the following methods: @@ -700,7 +662,7 @@ Connection objects :const:`SQLITE_OK` if access is allowed, :const:`SQLITE_DENY` if the entire SQL statement should be aborted with an error and :const:`SQLITE_IGNORE` if the column should be treated as a NULL value. These constants are available in the - :mod:`sqlite3` module. + :mod:`!sqlite3` module. The first argument to the callback signifies what kind of operation is to be authorized. The second and third argument will be arguments or ``None`` @@ -711,7 +673,7 @@ Connection objects Please consult the SQLite documentation about the possible values for the first argument and the meaning of the second and third argument depending on the first - one. All necessary constants are available in the :mod:`sqlite3` module. + one. All necessary constants are available in the :mod:`!sqlite3` module. Passing ``None`` as *authorizer_callback* will disable the authorizer. @@ -769,7 +731,7 @@ Connection objects .. note:: - The ``sqlite3`` module is not built with loadable extension support by + The :mod:`!sqlite3` module is not built with loadable extension support by default, because some platforms (notably macOS) have SQLite libraries which are compiled without this feature. To get loadable extension support, @@ -862,16 +824,14 @@ Connection objects Works even if the database is being accessed by other clients or concurrently by the same connection. - :param target: + :param Connection target: The database connection to save the backup to. - :type target: Connection - :param pages: + :param int pages: The number of pages to copy at a time. If equal to or less than ``0``, the entire database is copied in a single step. Defaults to ``-1``. - :type pages: int :param progress: If set to a callable, it is invoked with three integer arguments for @@ -882,18 +842,16 @@ Connection objects Defaults to ``None``. :type progress: :term:`callback` | None - :param name: + :param str name: The name of the database to back up. Either ``"main"`` (the default) for the main database, ``"temp"`` for the temporary database, or the name of a custom database as attached using the - ``ATTACH DATABASE`` SQL statment. - :type name: str + ``ATTACH DATABASE`` SQL statement. - :param sleep: + :param float sleep: The number of seconds to sleep between successive attempts to back up remaining pages. - :type sleep: float Example 1, copy an existing database into another:: @@ -919,11 +877,17 @@ Connection objects .. versionadded:: 3.7 - .. method:: getlimit(category, /) - Get a connection runtime limit. *category* is the limit category to be - queried. + Get a connection runtime limit. + + :param int category: + The `SQLite limit category`_ to be queried. + + :rtype: int + + :raises ProgrammingError: + If *category* is not recognised by the underlying SQLite library. Example, query the maximum length of an SQL statement:: @@ -937,14 +901,23 @@ Connection objects .. method:: setlimit(category, limit, /) - Set a connection runtime limit. *category* is the limit category to be - set. *limit* is the new limit. If the new limit is a negative number, the - limit is unchanged. - + Set a connection runtime limit. Attempts to increase a limit above its hard upper bound are silently truncated to the hard upper bound. Regardless of whether or not the limit was changed, the prior value of the limit is returned. + :param int category: + The `SQLite limit category`_ to be set. + + :param int limit: + The value of the new limit. + If negative, the current limit is unchanged. + + :rtype: int + + :raises ProgrammingError: + If *category* is not recognised by the underlying SQLite library. + Example, limit the number of attached databases to 1:: import sqlite3 @@ -953,6 +926,8 @@ Connection objects .. versionadded:: 3.11 + .. _SQLite limit category: https://www.sqlite.org/c3ref/c_limit_attached.html + .. method:: serialize(*, name="main") @@ -962,8 +937,11 @@ Connection objects serialization is the same sequence of bytes which would be written to disk if that database were backed up to disk. - *name* is the database to be serialized, and defaults to the main - database. + :param str name: + The database name to be serialized. + Defaults to ``"main"``. + + :rtype: bytes .. note:: @@ -979,12 +957,24 @@ Connection objects :class:`Connection`. This method causes the database connection to disconnect from database *name*, and reopen *name* as an in-memory database based on the - serialization contained in *data*. Deserialization will raise - :exc:`OperationalError` if the database connection is currently involved - in a read transaction or a backup operation. :exc:`OverflowError` will be - raised if ``len(data)`` is larger than ``2**63 - 1``, and - :exc:`DatabaseError` will be raised if *data* does not contain a valid - SQLite database. + serialization contained in *data*. + + :param bytes data: + A serialized database. + + :param str name: + The database name to deserialize into. + Defaults to ``"main"``. + + :raises OperationalError: + If the database connection is currently involved in a read + transaction or a backup operation. + + :raises DatabaseError: + If *data* does not contain a valid SQLite database. + + :raises OverflowError: + If :func:`len(data) ` is larger than ``2**63 - 1``. .. note:: @@ -1060,7 +1050,7 @@ Cursor objects .. method:: executescript(sql_script, /) Execute the SQL statements in *sql_script*. - If there is a pending transaciton, + If there is a pending transaction, an implicit ``COMMIT`` statement is executed first. No other implicit transaction control is performed; any transaction control must be added to *sql_script*. @@ -1081,13 +1071,13 @@ Cursor objects .. method:: fetchone() - Fetch the next row of a query result set as a :class:`tuple`. + Return the next row of a query result set as a :class:`tuple`. Return ``None`` if no more data is available. .. method:: fetchmany(size=cursor.arraysize) - Fetch the next set of rows of a query result as a :class:`list`. + Return the next set of rows of a query result as a :class:`list`. Return an empty list if no more rows are available. The number of rows to fetch per call is specified by the *size* parameter. @@ -1103,7 +1093,7 @@ Cursor objects .. method:: fetchall() - Fetch all (remaining) rows of a query result as a :class:`list`. + Return all (remaining) rows of a query result as a :class:`list`. Return an empty list if no rows are available. Note that the :attr:`arraysize` attribute can affect the performance of this operation. @@ -1117,11 +1107,11 @@ Cursor objects .. method:: setinputsizes(sizes, /) - Required by the DB-API. Does nothing in :mod:`sqlite3`. + Required by the DB-API. Does nothing in :mod:`!sqlite3`. .. method:: setoutputsize(size, column=None, /) - Required by the DB-API. Does nothing in :mod:`sqlite3`. + Required by the DB-API. Does nothing in :mod:`!sqlite3`. .. attribute:: rowcount @@ -1309,8 +1299,8 @@ The exception hierarchy is defined by the DB-API 2.0 (:pep:`249`). .. exception:: Warning - This exception is not currently raised by the ``sqlite3`` module, - but may be raised by applications using ``sqlite3``, + This exception is not currently raised by the :mod:`!sqlite3` module, + but may be raised by applications using :mod:`!sqlite3`, for example if a user-defined function truncates data while inserting. ``Warning`` is a subclass of :exc:`Exception`. @@ -1341,7 +1331,7 @@ The exception hierarchy is defined by the DB-API 2.0 (:pep:`249`). Exception raised for misuse of the low-level SQLite C API. In other words, if this exception is raised, it probably indicates a bug in the - ``sqlite3`` module. + :mod:`!sqlite3` module. ``InterfaceError`` is a subclass of :exc:`Error`. .. exception:: DatabaseError @@ -1379,7 +1369,7 @@ The exception hierarchy is defined by the DB-API 2.0 (:pep:`249`). .. exception:: ProgrammingError - Exception raised for ``sqlite3`` API programming errors, + Exception raised for :mod:`!sqlite3` API programming errors, for example supplying the wrong number of bindings to a query, or trying to operate on a closed :class:`Connection`. ``ProgrammingError`` is a subclass of :exc:`DatabaseError`. @@ -1435,10 +1425,10 @@ This is how SQLite types are converted to Python types by default: | ``BLOB`` | :class:`bytes` | +-------------+----------------------------------------------+ -The type system of the :mod:`sqlite3` module is extensible in two ways: you can +The type system of the :mod:`!sqlite3` module is extensible in two ways: you can store additional Python types in an SQLite database via :ref:`object adapters `, -and you can let the ``sqlite3`` module convert SQLite types to +and you can let the :mod:`!sqlite3` module convert SQLite types to Python types via :ref:`converters `. @@ -1447,7 +1437,7 @@ Python types via :ref:`converters `. Command-line interface ^^^^^^^^^^^^^^^^^^^^^^ -The ``sqlite3`` module can be invoked as a script +The :mod:`!sqlite3` module can be invoked as a script in order to provide a simple SQLite shell. Type ``.quit`` or CTRL-D to exit the shell. @@ -1469,6 +1459,36 @@ Type ``.quit`` or CTRL-D to exit the shell. How-to guides ------------- +.. _sqlite3-placeholders: + +Using placeholders to bind values in SQL queries +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +SQL operations usually need to use values from Python variables. However, +beware of using Python's string operations to assemble queries, as they +are vulnerable to `SQL injection attacks`_ (see the `xkcd webcomic +`_ for a humorous example of what can go wrong):: + + # Never do this -- insecure! + symbol = 'RHAT' + cur.execute("SELECT * FROM stocks WHERE symbol = '%s'" % symbol) + +Instead, use the DB-API's parameter substitution. To insert a variable into a +query string, use a placeholder in the string, and substitute the actual values +into the query by providing them as a :class:`tuple` of values to the second +argument of the cursor's :meth:`~Cursor.execute` method. An SQL statement may +use one of two kinds of placeholders: question marks (qmark style) or named +placeholders (named style). For the qmark style, ``parameters`` must be a +:term:`sequence `. For the named style, it can be either a +:term:`sequence ` or :class:`dict` instance. The length of the +:term:`sequence ` must match the number of placeholders, or a +:exc:`ProgrammingError` is raised. If a :class:`dict` is given, it must contain +keys for all named parameters. Any extra items are ignored. Here's an example of +both styles: + +.. literalinclude:: ../includes/sqlite3/execute_1.py + + .. _sqlite3-adapters: Using adapters to store custom Python types in SQLite databases @@ -1476,7 +1496,7 @@ Using adapters to store custom Python types in SQLite databases SQLite supports only a limited set of data types natively. To store custom Python types in SQLite databases, *adapt* them to one of the -:ref:`Python types SQLite natively understands`. +:ref:`Python types SQLite natively understands `. There are two ways to adapt Python objects to SQLite types: letting your object adapt itself, or using an *adapter callable*. @@ -1540,7 +1560,7 @@ and constructs a :class:`Point` object from it. x, y = map(float, s.split(b";")) return Point(x, y) -We now need to tell ``sqlite3`` when it should convert a given SQLite value. +We now need to tell :mod:`!sqlite3` when it should convert a given SQLite value. This is done when connecting to a database, using the *detect_types* parameter of :func:`connect`. There are three options: @@ -1657,7 +1677,7 @@ directly using only a single call on the :class:`Connection` object. Accessing columns by name instead of by index ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -One useful feature of the :mod:`sqlite3` module is the built-in +One useful feature of the :mod:`!sqlite3` module is the built-in :class:`sqlite3.Row` class designed to be used as a row factory. Rows wrapped with this class can be accessed both by index (like tuples) and @@ -1732,7 +1752,7 @@ Explanation Transaction control ^^^^^^^^^^^^^^^^^^^ -The ``sqlite3`` module does not adhere to the transaction handling recommended +The :mod:`!sqlite3` module does not adhere to the transaction handling recommended by :pep:`249`. If the connection attribute :attr:`~Connection.isolation_level` @@ -1743,7 +1763,7 @@ new transactions are implicitly opened before Use the :meth:`~Connection.commit` and :meth:`~Connection.rollback` methods to respectively commit and roll back pending transactions. You can choose the underlying `SQLite transaction behaviour`_ — -that is, whether and what type of ``BEGIN`` statements ``sqlite3`` +that is, whether and what type of ``BEGIN`` statements :mod:`!sqlite3` implicitly executes – via the :attr:`~Connection.isolation_level` attribute. @@ -1760,7 +1780,7 @@ any pending transaction before execution of the given SQL script, regardless of the value of :attr:`~Connection.isolation_level`. .. versionchanged:: 3.6 - :mod:`sqlite3` used to implicitly commit an open transaction before DDL + :mod:`!sqlite3` used to implicitly commit an open transaction before DDL statements. This is no longer the case. .. _autocommit mode: diff --git a/Doc/library/string.rst b/Doc/library/string.rst index 7d0d601799f7a1..3b96813e683864 100644 --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -8,6 +8,7 @@ -------------- + .. seealso:: :ref:`textseq` diff --git a/Doc/library/superseded.rst b/Doc/library/superseded.rst index b38f16691f6ea9..57ef9638d058d4 100644 --- a/Doc/library/superseded.rst +++ b/Doc/library/superseded.rst @@ -27,7 +27,6 @@ backwards compatibility. They have been superseded by other modules. optparse.rst ossaudiodev.rst pipes.rst - smtpd.rst sndhdr.rst spwd.rst sunau.rst diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst index 1e6b08f32a7aca..507f2b3763cae4 100644 --- a/Doc/reference/import.rst +++ b/Doc/reference/import.rst @@ -676,22 +676,10 @@ Here are the exact rules used: * Otherwise, just use the module's ``__name__`` in the repr. -.. versionchanged:: 3.4 - Use of :meth:`loader.module_repr() ` - has been deprecated and the module spec is now used by the import - machinery to generate a module repr. - - For backward compatibility with Python 3.3, the module repr will be - generated by calling the loader's - :meth:`~importlib.abc.Loader.module_repr` method, if defined, before - trying either approach described above. However, the method is deprecated. - -.. versionchanged:: 3.10 - - Calling :meth:`~importlib.abc.Loader.module_repr` now occurs after trying to - use a module's ``__spec__`` attribute but before falling back on - ``__file__``. Use of :meth:`~importlib.abc.Loader.module_repr` is slated to - stop in Python 3.12. +.. versionchanged:: 3.12 + Use of :meth:`module_repr`, having been deprecated since Python 3.4, was + removed in Python 3.12 and is no longer called during the resolution of a + module's repr. .. _pyc-invalidation: diff --git a/Doc/using/configure.rst b/Doc/using/configure.rst index 580cbd814f786e..4e50e73a11b817 100644 --- a/Doc/using/configure.rst +++ b/Doc/using/configure.rst @@ -41,12 +41,6 @@ General Options See :data:`sys.int_info.bits_per_digit `. -.. cmdoption:: --with-cxx-main -.. cmdoption:: --with-cxx-main=COMPILER - - Compile the Python ``main()`` function and link Python executable with C++ - compiler: ``$CXX``, or *COMPILER* if specified. - .. cmdoption:: --with-suffix=SUFFIX Set the Python executable suffix to *SUFFIX*. @@ -721,22 +715,10 @@ Compiler flags Example: ``gcc -pthread``. -.. envvar:: MAINCC - - C compiler command used to build the ``main()`` function of programs like - ``python``. - - Variable set by the :option:`--with-cxx-main` option of the configure - script. - - Default: ``$(CC)``. - .. envvar:: CXX C++ compiler command. - Used if the :option:`--with-cxx-main` option is used. - Example: ``g++ -pthread``. .. envvar:: CFLAGS @@ -854,7 +836,7 @@ Linker flags Linker command used to build programs like ``python`` and ``_testembed``. - Default: ``$(PURIFY) $(MAINCC)``. + Default: ``$(PURIFY) $(CC)``. .. envvar:: CONFIGURE_LDFLAGS diff --git a/Doc/using/mac.rst b/Doc/using/mac.rst index f85b5bd2e713d0..9ae0270eaee7ab 100644 --- a/Doc/using/mac.rst +++ b/Doc/using/mac.rst @@ -17,15 +17,16 @@ the IDE and the Package Manager that are worth pointing out. Getting and Installing MacPython ================================ -macOS since version 10.8 comes with Python 2.7 pre-installed by Apple. If you wish, you -are invited to install the most recent version of Python 3 from the Python +macOS used to come with Python 2.7 pre-installed between versions +10.8 and `12.3 `_. +You are invited to install the most recent version of Python 3 from the Python website (https://www.python.org). A current "universal binary" build of Python, which runs natively on the Mac's new Intel and legacy PPC CPU's, is available there. What you get after installing is a number of things: -* A :file:`Python 3.9` folder in your :file:`Applications` folder. In here +* A :file:`Python 3.12` folder in your :file:`Applications` folder. In here you find IDLE, the development environment that is a standard part of official Python distributions; and PythonLauncher, which handles double-clicking Python scripts from the Finder. diff --git a/Doc/using/venv-create.inc b/Doc/using/venv-create.inc index b9785832864e22..c2a9f521a148b6 100644 --- a/Doc/using/venv-create.inc +++ b/Doc/using/venv-create.inc @@ -26,7 +26,7 @@ re-used. On Windows, invoke the ``venv`` command as follows:: - c:\>c:\Python35\python -m venv c:\path\to\myenv + c:\>Python35\python -m venv c:\path\to\myenv Alternatively, if you configured the ``PATH`` and ``PATHEXT`` variables for your :ref:`Python installation `:: diff --git a/Doc/whatsnew/2.5.rst b/Doc/whatsnew/2.5.rst index ea785121db90d3..6c216826fee047 100644 --- a/Doc/whatsnew/2.5.rst +++ b/Doc/whatsnew/2.5.rst @@ -2019,7 +2019,7 @@ https://www.sqlite.org. .. seealso:: - http://www.pysqlite.org + https://www.pysqlite.org The pysqlite web page. https://www.sqlite.org diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index c976eddb08ba08..39f1dab590a976 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -49,13 +49,6 @@ This article explains the new features in Python 3.11, compared to 3.10. For full details, see the :ref:`changelog `. -.. note:: - - Prerelease users should be aware that this document is currently in draft - form. It will be updated substantially as Python 3.11 moves towards release, - so it's worth checking back even after reading earlier versions. - - Summary -- Release highlights ============================= @@ -1165,6 +1158,13 @@ contributors are volunteers from the community. CPython bytecode changes ======================== +* The bytecode now contains inline cache entries, which take the form of + :opcode:`CACHE` instructions. Many opcodes expect to be followed by an exact + number of caches, and instruct the interpreter to skip over them at runtime. + Populated caches can look like arbitrary instructions, so great care should be + taken when reading or modifying raw, adaptive bytecode containing quickened + data. + * Replaced all numeric ``BINARY_*`` and ``INPLACE_*`` instructions with a single :opcode:`BINARY_OP` implementation. @@ -1544,6 +1544,10 @@ Changes in the Python API :func:`compile` and other related functions. If invalid positions are detected, a :exc:`ValueError` will be raised. (Contributed by Pablo Galindo in :gh:`93351`) +* :c:member:`~PyTypeObject.tp_dictoffset` should be treated as write-only. + It can be set to describe C extension clases to the VM, but should be regarded + as meaningless when read. To get the pointer to the object's dictionary call + :c:func:`PyObject_GenericGetDict` instead. Build Changes ============= diff --git a/Doc/whatsnew/3.12.rst b/Doc/whatsnew/3.12.rst index 67396f8e02280b..6df122acba71d5 100644 --- a/Doc/whatsnew/3.12.rst +++ b/Doc/whatsnew/3.12.rst @@ -205,6 +205,10 @@ Pending Removal in Python 3.14 (Contributed by Jason R. Coombs and Hugo van Kemenade in :gh:`93963`.) +* Creating :c:data:`immutable types ` with mutable + bases using the C API. + + Pending Removal in Future Versions ---------------------------------- @@ -295,6 +299,14 @@ Removed (and corresponding ``EXPERIMENTAL_ISOLATED_SUBINTERPRETERS``) have been removed. +* ``smtpd`` has been removed according to the schedule in :pep:`594`, + having been deprecated in Python 3.4.7 and 3.5.4. + Use aiosmtpd_ PyPI module or any other + :mod:`asyncio`-based server instead. + (Contributed by Oleg Iarygin in :gh:`93243`.) + +.. _aiosmtpd: https://pypi.org/project/aiosmtpd/ + * Remove ``io.OpenWrapper`` and ``_pyio.OpenWrapper``, deprecated in Python 3.10: just use :func:`open` instead. The :func:`open` (:func:`io.open`) function is a built-in function. Since Python 3.10, :func:`_pyio.open` is @@ -378,6 +390,10 @@ Changes in the Python API to :term:`filesystem encoding and error handler`. Argument files should be encoded in UTF-8 instead of ANSI Codepage on Windows. +* Removed the ``asyncore``-based ``smtpd`` module deprecated in Python 3.4.7 + and 3.5.4. A recommended replacement is the + :mod:`asyncio`-based aiosmtpd_ PyPI module. + * :func:`shlex.split`: Passing ``None`` for *s* argument now raises an exception, rather than reading :data:`sys.stdin`. The feature was deprecated in Python 3.9. @@ -410,6 +426,23 @@ New Features an additional metaclass argument. (Contributed by Wenzel Jakob in :gh:`93012`.) +* API for creating objects that can be called using + :ref:`the vectorcall protocol ` was added to the + :ref:`Limited API `: + + * :const:`Py_TPFLAGS_HAVE_VECTORCALL` + * :c:func:`PyVectorcall_NARGS` + * :c:func:`PyVectorcall_Call` + * :c:type:`vectorcallfunc` + + The :const:`Py_TPFLAGS_HAVE_VECTORCALL` flag is now removed from a class + when the class's :py:meth:`~object.__call__` method is reassigned. + This makes vectorcall safe to use with mutable types (i.e. heap types + without the :const:`immutable ` flag). + Mutable types that do not override :c:member:`~PyTypeObject.tp_call` now + inherit the ``Py_TPFLAGS_HAVE_VECTORCALL`` flag. + (Contributed by Petr Viktorin in :gh:`93274`.) + Porting to Python 3.12 ---------------------- @@ -427,6 +460,22 @@ Porting to Python 3.12 using the existing public C-API instead, or, if necessary, the (internal-only) ``_PyObject_GET_WEAKREFS_LISTPTR()`` macro. +* This internal-only :c:member:`PyTypeObject.tp_subclasses` may now not be + a valid object pointer. Its type was changed to :c:expr:`void *` to + reflect this. We mention this in case someone happens to be accessing the + internal-only field directly. + + To get a list of subclasses, call the Python method + :py:meth:`~class.__subclasses__` (using :c:func:`PyObject_CallMethod`, + for example). + +* An unrecognized format character in :c:func:`PyUnicode_FromFormat` and + :c:func:`PyUnicode_FromFormatV` now sets a :exc:`SystemError`. + In previous versions it caused all the rest of the format string to be + copied as-is to the result string, and any extra arguments discarded. + (Contributed by Serhiy Storchaka in :gh:`95781`.) + + Deprecated ---------- @@ -458,6 +507,9 @@ Deprecated :c:type:`PyConfig` instead. (Contributed by Victor Stinner in :gh:`77782`.) +* Creating :c:data:`immutable types ` with mutable + bases is deprecated and will be disabled in Python 3.14. + Removed ------- diff --git a/Include/abstract.h b/Include/abstract.h index 576024e09c4101..784ff7e928676f 100644 --- a/Include/abstract.h +++ b/Include/abstract.h @@ -228,6 +228,16 @@ PyAPI_FUNC(PyObject *) PyObject_CallMethodObjArgs( PyObject *name, ...); +/* Given a vectorcall nargsf argument, return the actual number of arguments. + * (For use outside the limited API, this is re-defined as a static inline + * function in cpython/abstract.h) + */ +PyAPI_FUNC(Py_ssize_t) PyVectorcall_NARGS(size_t nargsf); + +/* Call "callable" (which must support vectorcall) with positional arguments + "tuple" and keyword arguments "dict". "dict" may also be NULL */ +PyAPI_FUNC(PyObject *) PyVectorcall_Call(PyObject *callable, PyObject *tuple, PyObject *dict); + /* Implemented elsewhere: diff --git a/Include/cpython/abstract.h b/Include/cpython/abstract.h index 7038918f018880..6da29cde9f6092 100644 --- a/Include/cpython/abstract.h +++ b/Include/cpython/abstract.h @@ -53,8 +53,12 @@ PyAPI_FUNC(PyObject *) _PyObject_MakeTpCall( #define PY_VECTORCALL_ARGUMENTS_OFFSET \ (_Py_STATIC_CAST(size_t, 1) << (8 * sizeof(size_t) - 1)) +// PyVectorcall_NARGS() is exported as a function for the stable ABI. +// Here (when we are not using the stable ABI), the name is overridden to +// call a static inline function for best performance. +#define PyVectorcall_NARGS(n) _PyVectorcall_NARGS(n) static inline Py_ssize_t -PyVectorcall_NARGS(size_t n) +_PyVectorcall_NARGS(size_t n) { return n & ~PY_VECTORCALL_ARGUMENTS_OFFSET; } @@ -84,10 +88,6 @@ PyAPI_FUNC(PyObject *) PyObject_VectorcallDict( size_t nargsf, PyObject *kwargs); -/* Call "callable" (which must support vectorcall) with positional arguments - "tuple" and keyword arguments "dict". "dict" may also be NULL */ -PyAPI_FUNC(PyObject *) PyVectorcall_Call(PyObject *callable, PyObject *tuple, PyObject *dict); - // Same as PyObject_Vectorcall(), except without keyword arguments PyAPI_FUNC(PyObject *) _PyObject_FastCall( PyObject *func, diff --git a/Include/cpython/object.h b/Include/cpython/object.h index 60c7c3e2aa6bfb..c80fc1df0e0ba4 100644 --- a/Include/cpython/object.h +++ b/Include/cpython/object.h @@ -54,9 +54,6 @@ typedef struct _Py_Identifier { typedef int (*getbufferproc)(PyObject *, Py_buffer *, int); typedef void (*releasebufferproc)(PyObject *, Py_buffer *); -typedef PyObject *(*vectorcallfunc)(PyObject *callable, PyObject *const *args, - size_t nargsf, PyObject *kwnames); - typedef struct { /* Number implementations must check *both* @@ -217,8 +214,8 @@ struct _typeobject { inquiry tp_is_gc; /* For PyObject_IS_GC */ PyObject *tp_bases; PyObject *tp_mro; /* method resolution order */ - PyObject *tp_cache; - PyObject *tp_subclasses; + PyObject *tp_cache; /* no longer used */ + void *tp_subclasses; /* for static builtin types this is an index */ PyObject *tp_weaklist; /* not used for static builtin types */ destructor tp_del; @@ -227,7 +224,6 @@ struct _typeobject { destructor tp_finalize; vectorcallfunc tp_vectorcall; - size_t tp_static_builtin_index; /* 0 means "not initialized" */ }; /* This struct is used by the specializer diff --git a/Include/internal/pycore_object.h b/Include/internal/pycore_object.h index 173d36784cf138..5a328f04bae72b 100644 --- a/Include/internal/pycore_object.h +++ b/Include/internal/pycore_object.h @@ -217,6 +217,16 @@ extern void _Py_PrintReferences(FILE *); extern void _Py_PrintReferenceAddresses(FILE *); #endif + +/* Return the *address* of the object's weaklist. The address may be + * dereferenced to get the current head of the weaklist. This is useful + * for iterating over the linked list of weakrefs, especially when the + * list is being modified externally (e.g. refs getting removed). + * + * The returned pointer should not be used to change the head of the list + * nor should it be used to add, remove, or swap any refs in the list. + * That is the sole responsibility of the code in weakrefobject.c. + */ static inline PyObject ** _PyObject_GET_WEAKREFS_LISTPTR(PyObject *op) { @@ -226,10 +236,33 @@ _PyObject_GET_WEAKREFS_LISTPTR(PyObject *op) (PyTypeObject *)op); return _PyStaticType_GET_WEAKREFS_LISTPTR(state); } + // Essentially _PyObject_GET_WEAKREFS_LISTPTR_FROM_OFFSET(): Py_ssize_t offset = Py_TYPE(op)->tp_weaklistoffset; return (PyObject **)((char *)op + offset); } +/* This is a special case of _PyObject_GET_WEAKREFS_LISTPTR(). + * Only the most fundamental lookup path is used. + * Consequently, static types should not be used. + * + * For static builtin types the returned pointer will always point + * to a NULL tp_weaklist. This is fine for any deallocation cases, + * since static types are never deallocated and static builtin types + * are only finalized at the end of runtime finalization. + * + * If the weaklist for static types is actually needed then use + * _PyObject_GET_WEAKREFS_LISTPTR(). + */ +static inline PyWeakReference ** +_PyObject_GET_WEAKREFS_LISTPTR_FROM_OFFSET(PyObject *op) +{ + assert(!PyType_Check(op) || + ((PyTypeObject *)op)->tp_flags & Py_TPFLAGS_HEAPTYPE); + Py_ssize_t offset = Py_TYPE(op)->tp_weaklistoffset; + return (PyWeakReference **)((char *)op + offset); +} + + // Fast inlined version of PyObject_IS_GC() static inline int _PyObject_IS_GC(PyObject *obj) @@ -318,6 +351,7 @@ _PyDictOrValues_SetValues(PyDictOrValues *ptr, PyDictValues *values) extern PyObject ** _PyObject_ComputedDictPointer(PyObject *); extern void _PyObject_FreeInstanceAttributes(PyObject *obj); extern int _PyObject_IsInstanceDictEmpty(PyObject *); +extern int _PyType_HasSubclasses(PyTypeObject *); extern PyObject* _PyType_GetSubclasses(PyTypeObject *); // Access macro to the members which are floating "behind" the object diff --git a/Include/internal/pycore_typeobject.h b/Include/internal/pycore_typeobject.h index 4f9d6b1c4d4ba2..4eb0efcedb5e0d 100644 --- a/Include/internal/pycore_typeobject.h +++ b/Include/internal/pycore_typeobject.h @@ -45,6 +45,7 @@ struct type_cache { typedef struct { PyTypeObject *type; + PyObject *tp_subclasses; /* We never clean up weakrefs for static builtin types since they will effectively never get triggered. However, there are also some diagnostic uses for the list of weakrefs, diff --git a/Include/object.h b/Include/object.h index c00b9eb583420a..7d499d8b306e57 100644 --- a/Include/object.h +++ b/Include/object.h @@ -228,6 +228,11 @@ typedef int (*initproc)(PyObject *, PyObject *, PyObject *); typedef PyObject *(*newfunc)(PyTypeObject *, PyObject *, PyObject *); typedef PyObject *(*allocfunc)(PyTypeObject *, Py_ssize_t); +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x030c0000 // 3.12 +typedef PyObject *(*vectorcallfunc)(PyObject *callable, PyObject *const *args, + size_t nargsf, PyObject *kwnames); +#endif + typedef struct{ int slot; /* slot id, see below */ void *pfunc; /* function pointer */ @@ -381,11 +386,13 @@ given type object has a specified feature. #define Py_TPFLAGS_BASETYPE (1UL << 10) /* Set if the type implements the vectorcall protocol (PEP 590) */ -#ifndef Py_LIMITED_API +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x030C0000 #define Py_TPFLAGS_HAVE_VECTORCALL (1UL << 11) +#ifndef Py_LIMITED_API // Backwards compatibility alias for API that was provisional in Python 3.8 #define _Py_TPFLAGS_HAVE_VECTORCALL Py_TPFLAGS_HAVE_VECTORCALL #endif +#endif /* Set if the type is 'ready' -- fully initialized */ #define Py_TPFLAGS_READY (1UL << 12) diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index ce95e2f9948b59..7fa7facf8cf780 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -4,6 +4,8 @@ Released on 2022-10-03 ========================= +gh-65802: Document handling of extensions in Save As dialogs. + gh-95191: Include prompts when saving Shell (interactive input/output). gh-95511: Fix the Shell context menu copy-with-prompts bug of copying diff --git a/Lib/idlelib/README.txt b/Lib/idlelib/README.txt index 67de2be26256df..76aec58912f00e 100644 --- a/Lib/idlelib/README.txt +++ b/Lib/idlelib/README.txt @@ -243,8 +243,8 @@ OTHER TOPICS Generally use PEP 8. -import ------- +import statements +----------------- Put imports at the top, unless there is a good reason otherwise. PEP 8 says to group stdlib, 3rd-party dependencies, and package imports. For idlelib, the groups are general stdlib, tkinter, and idlelib. @@ -259,3 +259,32 @@ htest function def or "if __name__ == '__main__'" clause. Within module imports like "from idlelib.mod import class" may cause circular imports to deadlock. Even without this, circular imports may require at least one of the imports to be delayed until a function call. + +What's New entries +------------------ + +Repository directory Doc/whatsnew/ has a file 3.n.rst for each 3.n +Python version. For the first entry in each file, add subsection +'IDLE and idlelib', in alphabetical position, to the 'Improved Modules' +section. For the rest of cpython, entries to 3.(n+1).rst begin with +the release of 3.n.0b1. For IDLE, entries for features backported from +'main' to '3.n' during its beta period do not got in 3.(n+1).rst. The +latter usually gets its first entry during the 3.n.0 candidate period +or after the 3.n.0 release. + +When, as per PEP 434, feature changes are backported, entries are placed +in the 3.n.rst file *in the main branch* for each Python version n that +gets the backport. (Note: the format of entries have varied between +versions.) Add a line "New in 3.n maintenance releases." before the +first back-ported feature after 3.n.0 is released. Since each older +version file gets a different number of backports, it is easiest to +make a separate PR for each file and label it with the backports +needed. + +Github repository and issues +---------------------------- + +The CPython repository is https://github.com/python/cpython. The +IDLE Issues listing is https://github.com/orgs/python/projects/31. +The main classification is by Topic, based on the IDLE menu. View the +topics list by clicking the [<]] button in the upper right. diff --git a/Lib/idlelib/help.html b/Lib/idlelib/help.html index e8e7d2876097ad..af5cbd5a5ba4c3 100644 --- a/Lib/idlelib/help.html +++ b/Lib/idlelib/help.html @@ -91,6 +91,7 @@

Table of Contents

  • Editor windows
  • Key bindings
  • Automatic indentation
    • +
  • Search and Replace
  • Completions
  • Calltips
  • Code Context
    • @@ -112,6 +113,7 @@

    Table of Contents

  • Setting preferences
  • IDLE on macOS
  • Extensions
    • +
  • idlelib
    • @@ -237,13 +239,13 @@

    File menu (Shell and Editor) -
    Class Browser

    Show functions, classes, and methods in the current Editor file in a +

    Module Browser

    Show functions, classes, and methods in the current Editor file in a tree structure. In the shell, open a module first.

    Path Browser

    Show sys.path directories, modules, functions, classes and methods in a @@ -255,10 +257,13 @@

    File menu (Shell and Editor) -
    Select All

    Select the entire contents of the current window.

    -
    Find…

    Open a search dialog with many options

    Find Again

    Repeat the last search, if there is one.

    @@ -309,17 +314,21 @@

    Edit menu (Shell and Editor)Calltips in the Editing and navigation section below.

    -
    Show surrounding parens

    Highlight the surrounding parenthesis.

    +
    Show Surrounding Parens

    Highlight the surrounding parenthesis.

    Format menu (Editor window only)

    +
    Format Paragraph

    Reformat the current blank-line-delimited paragraph in comment block or +multiline string or selected line in a string. All lines in the +paragraph will be formatted to less than N columns, where N defaults to 72.

    +
    Indent Region

    Shift selected lines right by the indent width (default 4 spaces).

    Dedent Region

    Shift selected lines left by the indent width (default 4 spaces).

    @@ -338,11 +347,7 @@

    Edit menu (Shell and Editor)See also the indent/dedent region commands on the Format menu.

    +
    +

    Search and Replace

    +

    Any selection becomes a search target. However, only selections within +a line work because searches are only performed within lines with the +terminal newline removed. If [x] Regular expresion is checked, the +target is interpreted according to the Python re module.

    +

    Completions

    Completions are supplied, when requested and available, for module @@ -990,6 +1002,17 @@

    Extensions +

    idlelib

    +

    Source code: Lib/idlelib

    +

    The Lib/idlelib package implements the IDLE application. See the top +of this file or content listing on the left for how to use IDLE.

    +

    The files in idlelib are described in idlelib/README.txt. Access it +either in idlelib or click Help => About IDLE on the IDLE menu. This +file also maps IDLE menu items to the code that implements the item. +Except for files listed under ‘Startup’, the idlelib code is ‘private’ in +sense that feature changes can be backported (see PEP 434).

    +

    @@ -1021,6 +1044,7 @@

    Table of Contents

  • Editor windows
  • Key bindings
  • Automatic indentation
    • +
  • Search and Replace
  • Completions
  • Calltips
  • Code Context
    • @@ -1042,6 +1066,7 @@

    Table of Contents

  • Setting preferences
  • IDLE on macOS
  • Extensions
    • +
  • idlelib
    • @@ -1141,7 +1166,7 @@

    Navigation



    - Last updated on Jul 03, 2022. + Last updated on Aug 07, 2022. Found a bug?
    diff --git a/Lib/importlib/_abc.py b/Lib/importlib/_abc.py index f80348fc7ffd1d..08320563896521 100644 --- a/Lib/importlib/_abc.py +++ b/Lib/importlib/_abc.py @@ -38,17 +38,3 @@ def load_module(self, fullname): raise ImportError # Warning implemented in _load_module_shim(). return _bootstrap._load_module_shim(self, fullname) - - def module_repr(self, module): - """Return a module's repr. - - Used by the module type when the method does not raise - NotImplementedError. - - This method is deprecated. - - """ - warnings.warn("importlib.abc.Loader.module_repr() is deprecated and " - "slated for removal in Python 3.12", DeprecationWarning) - # The exception will cause ModuleType.__repr__ to ignore this method. - raise NotImplementedError diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py index afb95f4e1df869..67989c500f21c0 100644 --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -296,11 +296,6 @@ def _module_repr(module): loader = getattr(module, '__loader__', None) if spec := getattr(module, "__spec__", None): return _module_repr_from_spec(spec) - elif hasattr(loader, 'module_repr'): - try: - return loader.module_repr(module) - except Exception: - pass # Fall through to a catch-all which always succeeds. try: name = module.__name__ @@ -582,7 +577,6 @@ def module_from_spec(spec): def _module_repr_from_spec(spec): """Return the repr to use for the module.""" - # We mostly replicate _module_repr() using the spec attributes. name = '?' if spec.name is None else spec.name if spec.origin is None: if spec.loader is None: diff --git a/Lib/importlib/_bootstrap_external.py b/Lib/importlib/_bootstrap_external.py index b6c6716e907734..82d204257ed7a3 100644 --- a/Lib/importlib/_bootstrap_external.py +++ b/Lib/importlib/_bootstrap_external.py @@ -1339,22 +1339,11 @@ def append(self, item): # This class is actually exposed publicly in a namespace package's __loader__ # attribute, so it should be available through a non-private name. -# https://bugs.python.org/issue35673 +# https://github.com/python/cpython/issues/92054 class NamespaceLoader: def __init__(self, name, path, path_finder): self._path = _NamespacePath(name, path, path_finder) - @staticmethod - def module_repr(module): - """Return repr for the module. - - The method is deprecated. The import machinery does the job itself. - - """ - _warnings.warn("NamespaceLoader.module_repr() is deprecated and " - "slated for removal in Python 3.12", DeprecationWarning) - return ''.format(module.__name__) - def is_package(self, fullname): return True diff --git a/Lib/pathlib.py b/Lib/pathlib.py index 2aee71742b23e4..54da1c8a625398 100644 --- a/Lib/pathlib.py +++ b/Lib/pathlib.py @@ -1211,23 +1211,9 @@ def is_file(self): def is_mount(self): """ - Check if this path is a POSIX mount point + Check if this path is a mount point """ - # Need to exist and be a dir - if not self.exists() or not self.is_dir(): - return False - - try: - parent_dev = self.parent.stat().st_dev - except OSError: - return False - - dev = self.stat().st_dev - if dev != parent_dev: - return True - ino = self.stat().st_ino - parent_ino = self.parent.stat().st_ino - return ino == parent_ino + return self._flavour.pathmod.ismount(self) def is_symlink(self): """ @@ -1378,6 +1364,3 @@ class WindowsPath(Path, PureWindowsPath): On a Windows system, instantiating a Path should return this object. """ __slots__ = () - - def is_mount(self): - raise NotImplementedError("Path.is_mount() is unsupported on this system") diff --git a/Lib/posixpath.py b/Lib/posixpath.py index a7b2f2d64824fa..5e1ebe3293d849 100644 --- a/Lib/posixpath.py +++ b/Lib/posixpath.py @@ -364,7 +364,7 @@ def normpath(path): initial_slashes = path.startswith(sep) # POSIX allows one or two initial slashes, but treats three or more # as single slash. - # (see http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_13) + # (see https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_13) if (initial_slashes and path.startswith(sep*2) and not path.startswith(sep*3)): initial_slashes = 2 diff --git a/Lib/random.py b/Lib/random.py index 76627309e288b9..c70294ee0cbf08 100644 --- a/Lib/random.py +++ b/Lib/random.py @@ -239,7 +239,7 @@ def _randbelow_with_getrandbits(self, n): "Return a random int in the range [0,n). Defined for n > 0." getrandbits = self.getrandbits - k = n.bit_length() # don't use (n-1) here because n can be 1 + k = n.bit_length() r = getrandbits(k) # 0 <= r < 2**k while r >= n: r = getrandbits(k) diff --git a/Lib/test/crashers/infinite_loop_re.py b/Lib/test/crashers/infinite_loop_re.py index 9aecc568d91fe0..c84f28d601f865 100644 --- a/Lib/test/crashers/infinite_loop_re.py +++ b/Lib/test/crashers/infinite_loop_re.py @@ -1,5 +1,5 @@ -# This was taken from http://python.org/sf/1541697 +# This was taken from https://bugs.python.org/issue1541697 # It's not technically a crasher. It may not even truly be infinite, # however, I haven't waited a long time to see the result. It takes # 100% of CPU while running this and should be fixed. diff --git a/Lib/test/mock_socket.py b/Lib/test/mock_socket.py index c7abddcf5fafd3..b85b955d7f6654 100644 --- a/Lib/test/mock_socket.py +++ b/Lib/test/mock_socket.py @@ -1,4 +1,4 @@ -"""Mock socket module used by the smtpd and smtplib tests. +"""Mock socket module used by the smtplib tests. """ # imported for _GLOBAL_DEFAULT_TIMEOUT @@ -33,7 +33,7 @@ def close(self): class MockSocket: - """Mock socket object used by smtpd and smtplib tests. + """Mock socket object used by the smtplib tests. """ def __init__(self, family=None): global _reply_data diff --git a/Lib/smtpd.py b/Lib/test/smtpd.py similarity index 98% rename from Lib/smtpd.py rename to Lib/test/smtpd.py index b23579f120716f..f9d4b048a83f68 100755 --- a/Lib/smtpd.py +++ b/Lib/test/smtpd.py @@ -77,25 +77,16 @@ import time import socket import collections -from warnings import _deprecated, warn +from test.support.import_helper import import_module +from warnings import warn from email._header_value_parser import get_addr_spec, get_angle_addr __all__ = [ "SMTPChannel", "SMTPServer", "DebuggingServer", "PureProxy", ] -_DEPRECATION_MSG = ('The {name} module is deprecated and unmaintained and will ' - 'be removed in Python {remove}. Please see aiosmtpd ' - '(https://aiosmtpd.readthedocs.io/) for the recommended ' - 'replacement.') -_deprecated(__name__, _DEPRECATION_MSG, remove=(3, 12)) - - -# These are imported after the above warning so that users get the correct -# deprecation warning. -import asyncore -import asynchat - +asyncore = import_module('asyncore', deprecated=True) +asynchat = import_module('asynchat', deprecated=True) program = sys.argv[0] __version__ = 'Python SMTP proxy version 0.3' diff --git a/Lib/test/test_asyncio/test_ssl.py b/Lib/test/test_asyncio/test_ssl.py index 5e3c1573c9c587..5de9b7a14e87da 100644 --- a/Lib/test/test_asyncio/test_ssl.py +++ b/Lib/test/test_asyncio/test_ssl.py @@ -5,6 +5,7 @@ import logging import select import socket +import sys import tempfile import threading import time @@ -20,6 +21,10 @@ from test.test_asyncio import utils as test_utils +MACOS = (sys.platform == 'darwin') +BUF_MULTIPLIER = 1024 if not MACOS else 64 + + def tearDownModule(): asyncio.set_event_loop_policy(None) @@ -58,6 +63,16 @@ def connection_lost(self, exc): self.done.set_result(None) +class MessageOutFilter(logging.Filter): + def __init__(self, msg): + self.msg = msg + + def filter(self, record): + if self.msg in record.msg: + return False + return True + + @unittest.skipIf(ssl is None, 'No ssl module') class TestSSL(test_utils.TestCase): @@ -149,7 +164,7 @@ def _create_client_ssl_context(self, *, disable_verify=True): def _silence_eof_received_warning(self): # TODO This warning has to be fixed in asyncio. logger = logging.getLogger('asyncio') - filter = logging.Filter('has no effect when using ssl') + filter = MessageOutFilter('has no effect when using ssl') logger.addFilter(filter) try: yield @@ -181,8 +196,8 @@ def test_create_server_ssl_1(self): TOTAL_CNT = 25 # total number of clients that test will create TIMEOUT = support.LONG_TIMEOUT # timeout for this test - A_DATA = b'A' * 1024 * 1024 - B_DATA = b'B' * 1024 * 1024 + A_DATA = b'A' * 1024 * BUF_MULTIPLIER + B_DATA = b'B' * 1024 * BUF_MULTIPLIER sslctx = self._create_server_ssl_context( test_utils.ONLYCERT, test_utils.ONLYKEY @@ -277,8 +292,8 @@ def test_create_connection_ssl_1(self): CNT = 0 TOTAL_CNT = 25 - A_DATA = b'A' * 1024 * 1024 - B_DATA = b'B' * 1024 * 1024 + A_DATA = b'A' * 1024 * BUF_MULTIPLIER + B_DATA = b'B' * 1024 * BUF_MULTIPLIER sslctx = self._create_server_ssl_context( test_utils.ONLYCERT, @@ -1024,8 +1039,8 @@ def test_create_server_ssl_over_ssl(self): TOTAL_CNT = 25 # total number of clients that test will create TIMEOUT = support.LONG_TIMEOUT # timeout for this test - A_DATA = b'A' * 1024 * 1024 - B_DATA = b'B' * 1024 * 1024 + A_DATA = b'A' * 1024 * BUF_MULTIPLIER + B_DATA = b'B' * 1024 * BUF_MULTIPLIER sslctx_1 = self._create_server_ssl_context( test_utils.ONLYCERT, test_utils.ONLYKEY) @@ -1168,7 +1183,7 @@ def test_shutdown_cleanly(self): CNT = 0 TOTAL_CNT = 25 - A_DATA = b'A' * 1024 * 1024 + A_DATA = b'A' * 1024 * BUF_MULTIPLIER sslctx = self._create_server_ssl_context( test_utils.ONLYCERT, test_utils.ONLYKEY) diff --git a/Lib/test/test_call.py b/Lib/test/test_call.py index 4c971bc5ed0586..131b45e6caaab3 100644 --- a/Lib/test/test_call.py +++ b/Lib/test/test_call.py @@ -9,6 +9,7 @@ import itertools import gc import contextlib +import sys class BadStr(str): @@ -606,9 +607,19 @@ def test_vectorcall_flag(self): self.assertFalse(_testcapi.MethodDescriptorNopGet.__flags__ & Py_TPFLAGS_HAVE_VECTORCALL) self.assertTrue(_testcapi.MethodDescriptor2.__flags__ & Py_TPFLAGS_HAVE_VECTORCALL) - # Mutable heap types should not inherit Py_TPFLAGS_HAVE_VECTORCALL + # Mutable heap types should inherit Py_TPFLAGS_HAVE_VECTORCALL, + # but should lose it when __call__ is overridden class MethodDescriptorHeap(_testcapi.MethodDescriptorBase): pass + self.assertTrue(MethodDescriptorHeap.__flags__ & Py_TPFLAGS_HAVE_VECTORCALL) + MethodDescriptorHeap.__call__ = print + self.assertFalse(MethodDescriptorHeap.__flags__ & Py_TPFLAGS_HAVE_VECTORCALL) + + # Mutable heap types should not inherit Py_TPFLAGS_HAVE_VECTORCALL if + # they define __call__ directly + class MethodDescriptorHeap(_testcapi.MethodDescriptorBase): + def __call__(self): + pass self.assertFalse(MethodDescriptorHeap.__flags__ & Py_TPFLAGS_HAVE_VECTORCALL) def test_vectorcall_override(self): @@ -621,6 +632,58 @@ def test_vectorcall_override(self): f = _testcapi.MethodDescriptorNopGet() self.assertIs(f(*args), args) + def test_vectorcall_override_on_mutable_class(self): + """Setting __call__ should disable vectorcall""" + TestType = _testcapi.make_vectorcall_class() + instance = TestType() + self.assertEqual(instance(), "tp_call") + instance.set_vectorcall(TestType) + self.assertEqual(instance(), "vectorcall") # assume vectorcall is used + TestType.__call__ = lambda self: "custom" + self.assertEqual(instance(), "custom") + + def test_vectorcall_override_with_subclass(self): + """Setting __call__ on a superclass should disable vectorcall""" + SuperType = _testcapi.make_vectorcall_class() + class DerivedType(SuperType): + pass + + instance = DerivedType() + + # Derived types with its own vectorcall should be unaffected + UnaffectedType1 = _testcapi.make_vectorcall_class(DerivedType) + UnaffectedType2 = _testcapi.make_vectorcall_class(SuperType) + + # Aside: Quickly check that the C helper actually made derived types + self.assertTrue(issubclass(UnaffectedType1, DerivedType)) + self.assertTrue(issubclass(UnaffectedType2, SuperType)) + + # Initial state: tp_call + self.assertEqual(instance(), "tp_call") + self.assertEqual(_testcapi.has_vectorcall_flag(SuperType), True) + self.assertEqual(_testcapi.has_vectorcall_flag(DerivedType), True) + self.assertEqual(_testcapi.has_vectorcall_flag(UnaffectedType1), True) + self.assertEqual(_testcapi.has_vectorcall_flag(UnaffectedType2), True) + + # Setting the vectorcall function + instance.set_vectorcall(SuperType) + + self.assertEqual(instance(), "vectorcall") + self.assertEqual(_testcapi.has_vectorcall_flag(SuperType), True) + self.assertEqual(_testcapi.has_vectorcall_flag(DerivedType), True) + self.assertEqual(_testcapi.has_vectorcall_flag(UnaffectedType1), True) + self.assertEqual(_testcapi.has_vectorcall_flag(UnaffectedType2), True) + + # Setting __call__ should remove vectorcall from all subclasses + SuperType.__call__ = lambda self: "custom" + + self.assertEqual(instance(), "custom") + self.assertEqual(_testcapi.has_vectorcall_flag(SuperType), False) + self.assertEqual(_testcapi.has_vectorcall_flag(DerivedType), False) + self.assertEqual(_testcapi.has_vectorcall_flag(UnaffectedType1), True) + self.assertEqual(_testcapi.has_vectorcall_flag(UnaffectedType2), True) + + def test_vectorcall(self): # Test a bunch of different ways to call objects: # 1. vectorcall using PyVectorcall_Call() @@ -697,6 +760,14 @@ def __call__(self, *args): self.assertEqual(expected, meth(*args1, **kwargs)) self.assertEqual(expected, wrapped(*args, **kwargs)) + @unittest.skipIf( + hasattr(sys, 'getobjects'), + "Limited API is not compatible with Py_TRACE_REFS") + def test_vectorcall_limited(self): + from _testcapi import pyobject_vectorcall + obj = _testcapi.LimitedVectorCallClass() + self.assertEqual(pyobject_vectorcall(obj, (), ()), "vectorcall called") + class A: def method_two_args(self, x, y): diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py index 013229a6cdc97a..1ff14e7bc56fd5 100644 --- a/Lib/test/test_capi.py +++ b/Lib/test/test_capi.py @@ -15,6 +15,7 @@ import threading import time import unittest +import warnings import weakref from test import support from test.support import MISSING_C_DOCSTRINGS @@ -644,6 +645,35 @@ def test_pytype_fromspec_with_repeated_slots(self): with self.assertRaises(SystemError): _testcapi.create_type_from_repeated_slots(variant) + @warnings_helper.ignore_warnings(category=DeprecationWarning) + def test_immutable_type_with_mutable_base(self): + # Add deprecation warning here so it's removed in 3.14 + warnings._deprecated( + 'creating immutable classes with mutable bases', remove=(3, 14)) + + class MutableBase: + def meth(self): + return 'original' + + with self.assertWarns(DeprecationWarning): + ImmutableSubclass = _testcapi.make_immutable_type_with_base( + MutableBase) + instance = ImmutableSubclass() + + self.assertEqual(instance.meth(), 'original') + + # Cannot override the static type's method + with self.assertRaisesRegex( + TypeError, + "cannot set 'meth' attribute of immutable type"): + ImmutableSubclass.meth = lambda self: 'overridden' + self.assertEqual(instance.meth(), 'original') + + # Can change the method on the mutable base + MutableBase.meth = lambda self: 'changed' + self.assertEqual(instance.meth(), 'changed') + + def test_pynumber_tobase(self): from _testcapi import pynumber_tobase self.assertEqual(pynumber_tobase(123, 2), '0b1111011') diff --git a/Lib/test/test_check_c_globals.py b/Lib/test/test_check_c_globals.py index 030debc452e409..898807a5e69259 100644 --- a/Lib/test/test_check_c_globals.py +++ b/Lib/test/test_check_c_globals.py @@ -1,9 +1,14 @@ import unittest import test.test_tools +from test.support.warnings_helper import save_restore_warnings_filters test.test_tools.skip_if_missing('c-analyzer') with test.test_tools.imports_under_tool('c-analyzer'): - from cpython.__main__ import main + # gh-95349: Save/restore warnings filters to leave them unchanged. + # Importing the c-analyzer imports docutils which imports pkg_resources + # which adds a warnings filter. + with save_restore_warnings_filters(): + from cpython.__main__ import main class ActualChecks(unittest.TestCase): diff --git a/Lib/test/test_configparser.py b/Lib/test/test_configparser.py index 5e2715a96b9943..da17c00063c56d 100644 --- a/Lib/test/test_configparser.py +++ b/Lib/test/test_configparser.py @@ -114,7 +114,7 @@ def basic_test(self, cf): # The use of spaces in the section names serves as a # regression test for SourceForge bug #583248: - # http://www.python.org/sf/583248 + # https://bugs.python.org/issue583248 # API access eq(cf.get('Foo Bar', 'foo'), 'bar1') @@ -932,7 +932,7 @@ def test_items(self): ('name', 'value')]) def test_safe_interpolation(self): - # See http://www.python.org/sf/511737 + # See https://bugs.python.org/issue511737 cf = self.fromstring("[section]\n" "option1{eq}xxx\n" "option2{eq}%(option1)s/xxx\n" diff --git a/Lib/test/test_ctypes/test_functions.py b/Lib/test/test_ctypes/test_functions.py index a3db0033533fbb..95633dfa8b38fd 100644 --- a/Lib/test/test_ctypes/test_functions.py +++ b/Lib/test/test_ctypes/test_functions.py @@ -371,7 +371,7 @@ class S8I(Structure): (9*2, 8*3, 7*4, 6*5, 5*6, 4*7, 3*8, 2*9)) def test_sf1651235(self): - # see https://www.python.org/sf/1651235 + # see https://bugs.python.org/issue1651235 proto = CFUNCTYPE(c_int, RECT, POINT) def callback(*args): diff --git a/Lib/test/test_ctypes/test_loading.py b/Lib/test/test_ctypes/test_loading.py index ea892277c4eae9..8d8632a4eb64d3 100644 --- a/Lib/test/test_ctypes/test_loading.py +++ b/Lib/test/test_ctypes/test_loading.py @@ -93,7 +93,7 @@ def test_1703286_A(self): # NOT fit into a 32-bit integer. FreeLibrary must be able # to accept this address. - # These are tests for https://www.python.org/sf/1703286 + # These are tests for https://bugs.python.org/issue1703286 handle = LoadLibrary("advapi32") FreeLibrary(handle) diff --git a/Lib/test/test_dataclasses.py b/Lib/test/test_dataclasses.py index cfa82093dfbe00..e2eab695789de7 100644 --- a/Lib/test/test_dataclasses.py +++ b/Lib/test/test_dataclasses.py @@ -2215,12 +2215,12 @@ class C(B): self.assertEqual(c.z, 100) def test_no_init(self): - dataclass(init=False) + @dataclass(init=False) class C: i: int = 0 self.assertEqual(C().i, 0) - dataclass(init=False) + @dataclass(init=False) class C: i: int = 2 def __init__(self): diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py index 4386e113745a4f..037c859e97d41b 100644 --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -845,7 +845,7 @@ def __delattr__(self, name): ("getattr", "foo"), ("delattr", "foo")]) - # http://python.org/sf/1174712 + # https://bugs.python.org/issue1174712 try: class Module(types.ModuleType, str): pass diff --git a/Lib/test/test_descrtut.py b/Lib/test/test_descrtut.py index b4158eb23a56f0..7796031ed0602f 100644 --- a/Lib/test/test_descrtut.py +++ b/Lib/test/test_descrtut.py @@ -1,7 +1,7 @@ # This contains most of the executable examples from Guido's descr # tutorial, once at # -# http://www.python.org/2.2/descrintro.html +# https://www.python.org/download/releases/2.2.3/descrintro/ # # A few examples left implicit in the writeup were fleshed out, a few were # skipped due to lack of interest (e.g., faking super() by hand isn't diff --git a/Lib/test/test_file.py b/Lib/test/test_file.py index 1146a37323c9bf..9df55278693531 100644 --- a/Lib/test/test_file.py +++ b/Lib/test/test_file.py @@ -217,7 +217,7 @@ def testSetBufferSize(self): self._checkBufferSize(1) def testTruncateOnWindows(self): - # SF bug + # SF bug # "file.truncate fault on windows" f = self.open(TESTFN, 'wb') diff --git a/Lib/test/test_fileio.py b/Lib/test/test_fileio.py index c26cdc028cc890..2263604ed1f97d 100644 --- a/Lib/test/test_fileio.py +++ b/Lib/test/test_fileio.py @@ -503,7 +503,7 @@ def testTruncate(self): def testTruncateOnWindows(self): def bug801631(): - # SF bug + # SF bug # "file.truncate fault on windows" f = self.FileIO(TESTFN, 'w') f.write(bytes(range(11))) diff --git a/Lib/test/test_getopt.py b/Lib/test/test_getopt.py index 64b9ce01e05ea2..c96a33b77fe272 100644 --- a/Lib/test/test_getopt.py +++ b/Lib/test/test_getopt.py @@ -83,7 +83,7 @@ def test_do_longs(self): # Much like the preceding, except with a non-alpha character ("-") in # option name that precedes "="; failed in - # http://python.org/sf/126863 + # https://bugs.python.org/issue126863 opts, args = getopt.do_longs([], 'foo=42', ['foo-bar', 'foo=',], []) self.assertEqual(opts, [('--foo', '42')]) self.assertEqual(args, []) diff --git a/Lib/test/test_grammar.py b/Lib/test/test_grammar.py index da88519862712a..58f907eac09d53 100644 --- a/Lib/test/test_grammar.py +++ b/Lib/test/test_grammar.py @@ -1592,7 +1592,7 @@ def test_selectors(self): s = a[-5:] s = a[:-1] s = a[-4:-3] - # A rough test of SF bug 1333982. http://python.org/sf/1333982 + # A rough test of SF bug 1333982. https://bugs.python.org/issue1333982 # The testing here is fairly incomplete. # Test cases should include: commas with 1 and 2 colons d = {} diff --git a/Lib/test/test_grp.py b/Lib/test/test_grp.py index c7ec03ec0e4388..e52e17b8dc7366 100644 --- a/Lib/test/test_grp.py +++ b/Lib/test/test_grp.py @@ -49,10 +49,12 @@ def test_values_extended(self): def test_errors(self): self.assertRaises(TypeError, grp.getgrgid) + self.assertRaises(TypeError, grp.getgrgid, 3.14) self.assertRaises(TypeError, grp.getgrnam) + self.assertRaises(TypeError, grp.getgrnam, 42) self.assertRaises(TypeError, grp.getgrall, 42) # embedded null character - self.assertRaises(ValueError, grp.getgrnam, 'a\x00b') + self.assertRaisesRegex(ValueError, 'null', grp.getgrnam, 'a\x00b') # try to get some errors bynames = {} diff --git a/Lib/test/test_importlib/frozen/test_loader.py b/Lib/test/test_importlib/frozen/test_loader.py index f2df7e60bf8e38..32f951cb1aca28 100644 --- a/Lib/test/test_importlib/frozen/test_loader.py +++ b/Lib/test/test_importlib/frozen/test_loader.py @@ -103,14 +103,6 @@ def test_lacking_parent(self): expected=value)) self.assertEqual(output, 'Hello world!\n') - def test_module_repr(self): - name = '__hello__' - module, output = self.exec_module(name) - with deprecated(): - repr_str = self.machinery.FrozenImporter.module_repr(module) - self.assertEqual(repr_str, - "") - def test_module_repr_indirect(self): name = '__hello__' module, output = self.exec_module(name) diff --git a/Lib/test/test_importlib/source/test_file_loader.py b/Lib/test/test_importlib/source/test_file_loader.py index 378dcbe08a8050..f35adec1a8e800 100644 --- a/Lib/test/test_importlib/source/test_file_loader.py +++ b/Lib/test/test_importlib/source/test_file_loader.py @@ -51,7 +51,6 @@ class Tester(self.abc.FileLoader): def get_code(self, _): pass def get_source(self, _): pass def is_package(self, _): pass - def module_repr(self, _): pass path = 'some_path' name = 'some_name' diff --git a/Lib/test/test_importlib/test_abc.py b/Lib/test/test_importlib/test_abc.py index d59b663a43ed06..88bf100efaad80 100644 --- a/Lib/test/test_importlib/test_abc.py +++ b/Lib/test/test_importlib/test_abc.py @@ -221,8 +221,6 @@ def test_module_repr(self): mod = types.ModuleType('blah') with warnings.catch_warnings(): warnings.simplefilter("ignore", DeprecationWarning) - with self.assertRaises(NotImplementedError): - self.ins.module_repr(mod) original_repr = repr(mod) mod.__loader__ = self.ins # Should still return a proper repr. diff --git a/Lib/test/test_importlib/test_namespace_pkgs.py b/Lib/test/test_importlib/test_namespace_pkgs.py index cd08498545e802..f451f7547b35bb 100644 --- a/Lib/test/test_importlib/test_namespace_pkgs.py +++ b/Lib/test/test_importlib/test_namespace_pkgs.py @@ -79,13 +79,6 @@ def test_cant_import_other(self): with self.assertRaises(ImportError): import foo.two - def test_module_repr(self): - import foo.one - with warnings.catch_warnings(): - warnings.simplefilter("ignore") - self.assertEqual(foo.__spec__.loader.module_repr(foo), - "") - class DynamicPathNamespacePackage(NamespacePackageTest): paths = ['portion1'] diff --git a/Lib/test/test_importlib/test_spec.py b/Lib/test/test_importlib/test_spec.py index 21e2c02094f22e..f1ab16c7b2a9be 100644 --- a/Lib/test/test_importlib/test_spec.py +++ b/Lib/test/test_importlib/test_spec.py @@ -407,101 +407,6 @@ def test_reload_legacy(self): machinery=machinery) -class ModuleReprTests: - - @property - def bootstrap(self): - return self.init._bootstrap - - def setUp(self): - self.module = type(os)('spam') - self.spec = self.machinery.ModuleSpec('spam', TestLoader()) - - def test_module___loader___module_repr(self): - class Loader: - def module_repr(self, module): - return ''.format(module.__name__) - self.module.__loader__ = Loader() - modrepr = self.bootstrap._module_repr(self.module) - - self.assertEqual(modrepr, '') - - def test_module___loader___module_repr_bad(self): - class Loader(TestLoader): - def module_repr(self, module): - raise Exception - self.module.__loader__ = Loader() - modrepr = self.bootstrap._module_repr(self.module) - - self.assertEqual(modrepr, - ')>'.format('spam')) - - def test_module___spec__(self): - origin = 'in a hole, in the ground' - self.spec.origin = origin - self.module.__spec__ = self.spec - modrepr = self.bootstrap._module_repr(self.module) - - self.assertEqual(modrepr, ''.format('spam', origin)) - - def test_module___spec___location(self): - location = 'in_a_galaxy_far_far_away.py' - self.spec.origin = location - self.spec._set_fileattr = True - self.module.__spec__ = self.spec - modrepr = self.bootstrap._module_repr(self.module) - - self.assertEqual(modrepr, - ''.format('spam', location)) - - def test_module___spec___no_origin(self): - self.spec.loader = TestLoader() - self.module.__spec__ = self.spec - modrepr = self.bootstrap._module_repr(self.module) - - self.assertEqual(modrepr, - ')>'.format('spam')) - - def test_module___spec___no_origin_no_loader(self): - self.spec.loader = None - self.module.__spec__ = self.spec - modrepr = self.bootstrap._module_repr(self.module) - - self.assertEqual(modrepr, ''.format('spam')) - - def test_module_no_name(self): - del self.module.__name__ - modrepr = self.bootstrap._module_repr(self.module) - - self.assertEqual(modrepr, ''.format('?')) - - def test_module_with_file(self): - filename = 'e/i/e/i/o/spam.py' - self.module.__file__ = filename - modrepr = self.bootstrap._module_repr(self.module) - - self.assertEqual(modrepr, - ''.format('spam', filename)) - - def test_module_no_file(self): - self.module.__loader__ = TestLoader() - modrepr = self.bootstrap._module_repr(self.module) - - self.assertEqual(modrepr, - ')>'.format('spam')) - - def test_module_no_file_no_loader(self): - modrepr = self.bootstrap._module_repr(self.module) - - self.assertEqual(modrepr, ''.format('spam')) - - -(Frozen_ModuleReprTests, - Source_ModuleReprTests - ) = test_util.test_both(ModuleReprTests, init=init, util=util, - machinery=machinery) - - class FactoryTests: def setUp(self): diff --git a/Lib/test/test_importlib/test_util.py b/Lib/test/test_importlib/test_util.py index c77c7814a9ccd3..a62d68fcd8b333 100644 --- a/Lib/test/test_importlib/test_util.py +++ b/Lib/test/test_importlib/test_util.py @@ -860,7 +860,7 @@ def test_magic_number(self): # stakeholders such as OS package maintainers must be notified # in advance. Such exceptional releases will then require an # adjustment to this test case. - EXPECTED_MAGIC_NUMBER = 3413 + EXPECTED_MAGIC_NUMBER = 3495 actual = int.from_bytes(importlib.util.MAGIC_NUMBER[:2], 'little') msg = ( diff --git a/Lib/test/test_lib2to3/data/py2_test_grammar.py b/Lib/test/test_lib2to3/data/py2_test_grammar.py index f9e4ea1374f907..1a631510f4dba7 100644 --- a/Lib/test/test_lib2to3/data/py2_test_grammar.py +++ b/Lib/test/test_lib2to3/data/py2_test_grammar.py @@ -735,7 +735,7 @@ def testSelectors(self): s = a[-5:] s = a[:-1] s = a[-4:-3] - # A rough test of SF bug 1333982. https://python.org/sf/1333982 + # A rough test of SF bug 1333982. https://bugs.python.org/issue1333982 # The testing here is fairly incomplete. # Test cases should include: commas with 1 and 2 colons d = {} diff --git a/Lib/test/test_lib2to3/data/py3_test_grammar.py b/Lib/test/test_lib2to3/data/py3_test_grammar.py index a4a3f7eac0dded..774851f5bd7e85 100644 --- a/Lib/test/test_lib2to3/data/py3_test_grammar.py +++ b/Lib/test/test_lib2to3/data/py3_test_grammar.py @@ -714,7 +714,7 @@ def testSelectors(self): s = a[-5:] s = a[:-1] s = a[-4:-3] - # A rough test of SF bug 1333982. https://python.org/sf/1333982 + # A rough test of SF bug 1333982. https://bugs.python.org/issue1333982 # The testing here is fairly incomplete. # Test cases should include: commas with 1 and 2 colons d = {} diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py index a900dabec6774b..a505e8000daab7 100644 --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -61,9 +61,10 @@ from socketserver import (ThreadingUDPServer, DatagramRequestHandler, ThreadingTCPServer, StreamRequestHandler) +with warnings.catch_warnings(): + from . import smtpd asyncore = warnings_helper.import_deprecated('asyncore') -smtpd = warnings_helper.import_deprecated('smtpd') try: diff --git a/Lib/test/test_module.py b/Lib/test/test_module.py index f72177dda3702a..6c83d76c8e3c68 100644 --- a/Lib/test/test_module.py +++ b/Lib/test/test_module.py @@ -8,10 +8,10 @@ import sys ModuleType = type(sys) + class FullLoader: - @classmethod - def module_repr(cls, m): - return "".format(m.__name__) + pass + class BareLoader: pass @@ -236,7 +236,7 @@ def test_module_repr_with_full_loader(self): # Yes, a class not an instance. m.__loader__ = FullLoader self.assertEqual( - repr(m), "") + repr(m), ")>") def test_module_repr_with_bare_loader_and_filename(self): # Because the loader has no module_repr(), use the file name. @@ -252,7 +252,7 @@ def test_module_repr_with_full_loader_and_filename(self): # Yes, a class not an instance. m.__loader__ = FullLoader m.__file__ = '/tmp/foo.py' - self.assertEqual(repr(m), "") + self.assertEqual(repr(m), "") def test_module_repr_builtin(self): self.assertEqual(repr(sys), "") diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py index e14b0fca55360f..668af8030c0c6a 100644 --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -2294,10 +2294,12 @@ def test_is_file(self): self.assertIs((P / 'fileA\udfff').is_file(), False) self.assertIs((P / 'fileA\x00').is_file(), False) - @only_posix def test_is_mount(self): P = self.cls(BASE) - R = self.cls('/') # TODO: Work out Windows. + if os.name == 'nt': + R = self.cls('c:\\') + else: + R = self.cls('/') self.assertFalse((P / 'fileA').is_mount()) self.assertFalse((P / 'dirA').is_mount()) self.assertFalse((P / 'non-existing').is_mount()) @@ -2305,8 +2307,7 @@ def test_is_mount(self): self.assertTrue(R.is_mount()) if os_helper.can_symlink(): self.assertFalse((P / 'linkA').is_mount()) - self.assertIs(self.cls('/\udfff').is_mount(), False) - self.assertIs(self.cls('/\x00').is_mount(), False) + self.assertIs((R / '\udfff').is_mount(), False) def test_is_symlink(self): P = self.cls(BASE) diff --git a/Lib/test/test_pwd.py b/Lib/test/test_pwd.py index c789326425be32..aa090b464a7222 100644 --- a/Lib/test/test_pwd.py +++ b/Lib/test/test_pwd.py @@ -59,6 +59,8 @@ def test_errors(self): self.assertRaises(TypeError, pwd.getpwnam) self.assertRaises(TypeError, pwd.getpwnam, 42) self.assertRaises(TypeError, pwd.getpwall, 42) + # embedded null character + self.assertRaisesRegex(ValueError, 'null', pwd.getpwnam, 'a\x00b') # try to get some errors bynames = {} diff --git a/Lib/test/test_pyexpat.py b/Lib/test/test_pyexpat.py index 6f0441b66d9b88..863c1194672c1c 100644 --- a/Lib/test/test_pyexpat.py +++ b/Lib/test/test_pyexpat.py @@ -508,7 +508,7 @@ def test(self): class sf1296433Test(unittest.TestCase): def test_parse_only_xml_data(self): - # http://python.org/sf/1296433 + # https://bugs.python.org/issue1296433 # xml = "%s" % ('a' * 1025) # this one doesn't crash diff --git a/Lib/test/test_set.py b/Lib/test/test_set.py index 43f23dbbf9bf7d..2dd65240f5faec 100644 --- a/Lib/test/test_set.py +++ b/Lib/test/test_set.py @@ -427,7 +427,7 @@ def test_remove(self): self.assertRaises(KeyError, self.s.remove, self.thetype(self.word)) def test_remove_keyerror_unpacking(self): - # bug: www.python.org/sf/1576657 + # https://bugs.python.org/issue1576657 for v1 in ['Q', (1,)]: try: self.s.remove(v1) diff --git a/Lib/test/test_smtpd.py b/Lib/test/test_smtpd.py deleted file mode 100644 index 39ff8793648ba6..00000000000000 --- a/Lib/test/test_smtpd.py +++ /dev/null @@ -1,1019 +0,0 @@ -import unittest -import textwrap -from test import support, mock_socket -from test.support import socket_helper -from test.support import warnings_helper -import socket -import io - - -smtpd = warnings_helper.import_deprecated('smtpd') -asyncore = warnings_helper.import_deprecated('asyncore') - -if not socket_helper.has_gethostname: - raise unittest.SkipTest("test requires gethostname()") - - -class DummyServer(smtpd.SMTPServer): - def __init__(self, *args, **kwargs): - smtpd.SMTPServer.__init__(self, *args, **kwargs) - self.messages = [] - if self._decode_data: - self.return_status = 'return status' - else: - self.return_status = b'return status' - - def process_message(self, peer, mailfrom, rcpttos, data, **kw): - self.messages.append((peer, mailfrom, rcpttos, data)) - if data == self.return_status: - return '250 Okish' - if 'mail_options' in kw and 'SMTPUTF8' in kw['mail_options']: - return '250 SMTPUTF8 message okish' - - -class DummyDispatcherBroken(Exception): - pass - - -class BrokenDummyServer(DummyServer): - def listen(self, num): - raise DummyDispatcherBroken() - - -class SMTPDServerTest(unittest.TestCase): - def setUp(self): - smtpd.socket = asyncore.socket = mock_socket - - def test_process_message_unimplemented(self): - server = smtpd.SMTPServer((socket_helper.HOST, 0), ('b', 0), - decode_data=True) - conn, addr = server.accept() - channel = smtpd.SMTPChannel(server, conn, addr, decode_data=True) - - def write_line(line): - channel.socket.queue_recv(line) - channel.handle_read() - - write_line(b'HELO example') - write_line(b'MAIL From:eggs@example') - write_line(b'RCPT To:spam@example') - write_line(b'DATA') - self.assertRaises(NotImplementedError, write_line, b'spam\r\n.\r\n') - - def test_decode_data_and_enable_SMTPUTF8_raises(self): - self.assertRaises( - ValueError, - smtpd.SMTPServer, - (socket_helper.HOST, 0), - ('b', 0), - enable_SMTPUTF8=True, - decode_data=True) - - def tearDown(self): - asyncore.close_all() - asyncore.socket = smtpd.socket = socket - - -class DebuggingServerTest(unittest.TestCase): - - def setUp(self): - smtpd.socket = asyncore.socket = mock_socket - - def send_data(self, channel, data, enable_SMTPUTF8=False): - def write_line(line): - channel.socket.queue_recv(line) - channel.handle_read() - write_line(b'EHLO example') - if enable_SMTPUTF8: - write_line(b'MAIL From:eggs@example BODY=8BITMIME SMTPUTF8') - else: - write_line(b'MAIL From:eggs@example') - write_line(b'RCPT To:spam@example') - write_line(b'DATA') - write_line(data) - write_line(b'.') - - def test_process_message_with_decode_data_true(self): - server = smtpd.DebuggingServer((socket_helper.HOST, 0), ('b', 0), - decode_data=True) - conn, addr = server.accept() - channel = smtpd.SMTPChannel(server, conn, addr, decode_data=True) - with support.captured_stdout() as s: - self.send_data(channel, b'From: test\n\nhello\n') - stdout = s.getvalue() - self.assertEqual(stdout, textwrap.dedent("""\ - ---------- MESSAGE FOLLOWS ---------- - From: test - X-Peer: peer-address - - hello - ------------ END MESSAGE ------------ - """)) - - def test_process_message_with_decode_data_false(self): - server = smtpd.DebuggingServer((socket_helper.HOST, 0), ('b', 0)) - conn, addr = server.accept() - channel = smtpd.SMTPChannel(server, conn, addr) - with support.captured_stdout() as s: - self.send_data(channel, b'From: test\n\nh\xc3\xa9llo\xff\n') - stdout = s.getvalue() - self.assertEqual(stdout, textwrap.dedent("""\ - ---------- MESSAGE FOLLOWS ---------- - b'From: test' - b'X-Peer: peer-address' - b'' - b'h\\xc3\\xa9llo\\xff' - ------------ END MESSAGE ------------ - """)) - - def test_process_message_with_enable_SMTPUTF8_true(self): - server = smtpd.DebuggingServer((socket_helper.HOST, 0), ('b', 0), - enable_SMTPUTF8=True) - conn, addr = server.accept() - channel = smtpd.SMTPChannel(server, conn, addr, enable_SMTPUTF8=True) - with support.captured_stdout() as s: - self.send_data(channel, b'From: test\n\nh\xc3\xa9llo\xff\n') - stdout = s.getvalue() - self.assertEqual(stdout, textwrap.dedent("""\ - ---------- MESSAGE FOLLOWS ---------- - b'From: test' - b'X-Peer: peer-address' - b'' - b'h\\xc3\\xa9llo\\xff' - ------------ END MESSAGE ------------ - """)) - - def test_process_SMTPUTF8_message_with_enable_SMTPUTF8_true(self): - server = smtpd.DebuggingServer((socket_helper.HOST, 0), ('b', 0), - enable_SMTPUTF8=True) - conn, addr = server.accept() - channel = smtpd.SMTPChannel(server, conn, addr, enable_SMTPUTF8=True) - with support.captured_stdout() as s: - self.send_data(channel, b'From: test\n\nh\xc3\xa9llo\xff\n', - enable_SMTPUTF8=True) - stdout = s.getvalue() - self.assertEqual(stdout, textwrap.dedent("""\ - ---------- MESSAGE FOLLOWS ---------- - mail options: ['BODY=8BITMIME', 'SMTPUTF8'] - b'From: test' - b'X-Peer: peer-address' - b'' - b'h\\xc3\\xa9llo\\xff' - ------------ END MESSAGE ------------ - """)) - - def tearDown(self): - asyncore.close_all() - asyncore.socket = smtpd.socket = socket - - -class TestFamilyDetection(unittest.TestCase): - def setUp(self): - smtpd.socket = asyncore.socket = mock_socket - - def tearDown(self): - asyncore.close_all() - asyncore.socket = smtpd.socket = socket - - @unittest.skipUnless(socket_helper.IPV6_ENABLED, "IPv6 not enabled") - def test_socket_uses_IPv6(self): - server = smtpd.SMTPServer((socket_helper.HOSTv6, 0), (socket_helper.HOSTv4, 0)) - self.assertEqual(server.socket.family, socket.AF_INET6) - - def test_socket_uses_IPv4(self): - server = smtpd.SMTPServer((socket_helper.HOSTv4, 0), (socket_helper.HOSTv6, 0)) - self.assertEqual(server.socket.family, socket.AF_INET) - - -class TestRcptOptionParsing(unittest.TestCase): - error_response = (b'555 RCPT TO parameters not recognized or not ' - b'implemented\r\n') - - def setUp(self): - smtpd.socket = asyncore.socket = mock_socket - self.old_debugstream = smtpd.DEBUGSTREAM - self.debug = smtpd.DEBUGSTREAM = io.StringIO() - - def tearDown(self): - asyncore.close_all() - asyncore.socket = smtpd.socket = socket - smtpd.DEBUGSTREAM = self.old_debugstream - - def write_line(self, channel, line): - channel.socket.queue_recv(line) - channel.handle_read() - - def test_params_rejected(self): - server = DummyServer((socket_helper.HOST, 0), ('b', 0)) - conn, addr = server.accept() - channel = smtpd.SMTPChannel(server, conn, addr) - self.write_line(channel, b'EHLO example') - self.write_line(channel, b'MAIL from: size=20') - self.write_line(channel, b'RCPT to: foo=bar') - self.assertEqual(channel.socket.last, self.error_response) - - def test_nothing_accepted(self): - server = DummyServer((socket_helper.HOST, 0), ('b', 0)) - conn, addr = server.accept() - channel = smtpd.SMTPChannel(server, conn, addr) - self.write_line(channel, b'EHLO example') - self.write_line(channel, b'MAIL from: size=20') - self.write_line(channel, b'RCPT to: ') - self.assertEqual(channel.socket.last, b'250 OK\r\n') - - -class TestMailOptionParsing(unittest.TestCase): - error_response = (b'555 MAIL FROM parameters not recognized or not ' - b'implemented\r\n') - - def setUp(self): - smtpd.socket = asyncore.socket = mock_socket - self.old_debugstream = smtpd.DEBUGSTREAM - self.debug = smtpd.DEBUGSTREAM = io.StringIO() - - def tearDown(self): - asyncore.close_all() - asyncore.socket = smtpd.socket = socket - smtpd.DEBUGSTREAM = self.old_debugstream - - def write_line(self, channel, line): - channel.socket.queue_recv(line) - channel.handle_read() - - def test_with_decode_data_true(self): - server = DummyServer((socket_helper.HOST, 0), ('b', 0), decode_data=True) - conn, addr = server.accept() - channel = smtpd.SMTPChannel(server, conn, addr, decode_data=True) - self.write_line(channel, b'EHLO example') - for line in [ - b'MAIL from: size=20 SMTPUTF8', - b'MAIL from: size=20 SMTPUTF8 BODY=8BITMIME', - b'MAIL from: size=20 BODY=UNKNOWN', - b'MAIL from: size=20 body=8bitmime', - ]: - self.write_line(channel, line) - self.assertEqual(channel.socket.last, self.error_response) - self.write_line(channel, b'MAIL from: size=20') - self.assertEqual(channel.socket.last, b'250 OK\r\n') - - def test_with_decode_data_false(self): - server = DummyServer((socket_helper.HOST, 0), ('b', 0)) - conn, addr = server.accept() - channel = smtpd.SMTPChannel(server, conn, addr) - self.write_line(channel, b'EHLO example') - for line in [ - b'MAIL from: size=20 SMTPUTF8', - b'MAIL from: size=20 SMTPUTF8 BODY=8BITMIME', - ]: - self.write_line(channel, line) - self.assertEqual(channel.socket.last, self.error_response) - self.write_line( - channel, - b'MAIL from: size=20 SMTPUTF8 BODY=UNKNOWN') - self.assertEqual( - channel.socket.last, - b'501 Error: BODY can only be one of 7BIT, 8BITMIME\r\n') - self.write_line( - channel, b'MAIL from: size=20 body=8bitmime') - self.assertEqual(channel.socket.last, b'250 OK\r\n') - - def test_with_enable_smtputf8_true(self): - server = DummyServer((socket_helper.HOST, 0), ('b', 0), enable_SMTPUTF8=True) - conn, addr = server.accept() - channel = smtpd.SMTPChannel(server, conn, addr, enable_SMTPUTF8=True) - self.write_line(channel, b'EHLO example') - self.write_line( - channel, - b'MAIL from: size=20 body=8bitmime smtputf8') - self.assertEqual(channel.socket.last, b'250 OK\r\n') - - -class SMTPDChannelTest(unittest.TestCase): - def setUp(self): - smtpd.socket = asyncore.socket = mock_socket - self.old_debugstream = smtpd.DEBUGSTREAM - self.debug = smtpd.DEBUGSTREAM = io.StringIO() - self.server = DummyServer((socket_helper.HOST, 0), ('b', 0), - decode_data=True) - conn, addr = self.server.accept() - self.channel = smtpd.SMTPChannel(self.server, conn, addr, - decode_data=True) - - def tearDown(self): - asyncore.close_all() - asyncore.socket = smtpd.socket = socket - smtpd.DEBUGSTREAM = self.old_debugstream - - def write_line(self, line): - self.channel.socket.queue_recv(line) - self.channel.handle_read() - - def test_broken_connect(self): - self.assertRaises( - DummyDispatcherBroken, BrokenDummyServer, - (socket_helper.HOST, 0), ('b', 0), decode_data=True) - - def test_decode_data_and_enable_SMTPUTF8_raises(self): - self.assertRaises( - ValueError, smtpd.SMTPChannel, - self.server, self.channel.conn, self.channel.addr, - enable_SMTPUTF8=True, decode_data=True) - - def test_server_accept(self): - self.server.handle_accept() - - def test_missing_data(self): - self.write_line(b'') - self.assertEqual(self.channel.socket.last, - b'500 Error: bad syntax\r\n') - - def test_EHLO(self): - self.write_line(b'EHLO example') - self.assertEqual(self.channel.socket.last, b'250 HELP\r\n') - - def test_EHLO_bad_syntax(self): - self.write_line(b'EHLO') - self.assertEqual(self.channel.socket.last, - b'501 Syntax: EHLO hostname\r\n') - - def test_EHLO_duplicate(self): - self.write_line(b'EHLO example') - self.write_line(b'EHLO example') - self.assertEqual(self.channel.socket.last, - b'503 Duplicate HELO/EHLO\r\n') - - def test_EHLO_HELO_duplicate(self): - self.write_line(b'EHLO example') - self.write_line(b'HELO example') - self.assertEqual(self.channel.socket.last, - b'503 Duplicate HELO/EHLO\r\n') - - def test_HELO(self): - name = smtpd.socket.getfqdn() - self.write_line(b'HELO example') - self.assertEqual(self.channel.socket.last, - '250 {}\r\n'.format(name).encode('ascii')) - - def test_HELO_EHLO_duplicate(self): - self.write_line(b'HELO example') - self.write_line(b'EHLO example') - self.assertEqual(self.channel.socket.last, - b'503 Duplicate HELO/EHLO\r\n') - - def test_HELP(self): - self.write_line(b'HELP') - self.assertEqual(self.channel.socket.last, - b'250 Supported commands: EHLO HELO MAIL RCPT ' + \ - b'DATA RSET NOOP QUIT VRFY\r\n') - - def test_HELP_command(self): - self.write_line(b'HELP MAIL') - self.assertEqual(self.channel.socket.last, - b'250 Syntax: MAIL FROM:
    \r\n') - - def test_HELP_command_unknown(self): - self.write_line(b'HELP SPAM') - self.assertEqual(self.channel.socket.last, - b'501 Supported commands: EHLO HELO MAIL RCPT ' + \ - b'DATA RSET NOOP QUIT VRFY\r\n') - - def test_HELO_bad_syntax(self): - self.write_line(b'HELO') - self.assertEqual(self.channel.socket.last, - b'501 Syntax: HELO hostname\r\n') - - def test_HELO_duplicate(self): - self.write_line(b'HELO example') - self.write_line(b'HELO example') - self.assertEqual(self.channel.socket.last, - b'503 Duplicate HELO/EHLO\r\n') - - def test_HELO_parameter_rejected_when_extensions_not_enabled(self): - self.extended_smtp = False - self.write_line(b'HELO example') - self.write_line(b'MAIL from: SIZE=1234') - self.assertEqual(self.channel.socket.last, - b'501 Syntax: MAIL FROM:
    \r\n') - - def test_MAIL_allows_space_after_colon(self): - self.write_line(b'HELO example') - self.write_line(b'MAIL from: ') - self.assertEqual(self.channel.socket.last, - b'250 OK\r\n') - - def test_extended_MAIL_allows_space_after_colon(self): - self.write_line(b'EHLO example') - self.write_line(b'MAIL from: size=20') - self.assertEqual(self.channel.socket.last, - b'250 OK\r\n') - - def test_NOOP(self): - self.write_line(b'NOOP') - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - - def test_HELO_NOOP(self): - self.write_line(b'HELO example') - self.write_line(b'NOOP') - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - - def test_NOOP_bad_syntax(self): - self.write_line(b'NOOP hi') - self.assertEqual(self.channel.socket.last, - b'501 Syntax: NOOP\r\n') - - def test_QUIT(self): - self.write_line(b'QUIT') - self.assertEqual(self.channel.socket.last, b'221 Bye\r\n') - - def test_HELO_QUIT(self): - self.write_line(b'HELO example') - self.write_line(b'QUIT') - self.assertEqual(self.channel.socket.last, b'221 Bye\r\n') - - def test_QUIT_arg_ignored(self): - self.write_line(b'QUIT bye bye') - self.assertEqual(self.channel.socket.last, b'221 Bye\r\n') - - def test_bad_state(self): - self.channel.smtp_state = 'BAD STATE' - self.write_line(b'HELO example') - self.assertEqual(self.channel.socket.last, - b'451 Internal confusion\r\n') - - def test_command_too_long(self): - self.write_line(b'HELO example') - self.write_line(b'MAIL from: ' + - b'a' * self.channel.command_size_limit + - b'@example') - self.assertEqual(self.channel.socket.last, - b'500 Error: line too long\r\n') - - def test_MAIL_command_limit_extended_with_SIZE(self): - self.write_line(b'EHLO example') - fill_len = self.channel.command_size_limit - len('MAIL from:<@example>') - self.write_line(b'MAIL from:<' + - b'a' * fill_len + - b'@example> SIZE=1234') - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - - self.write_line(b'MAIL from:<' + - b'a' * (fill_len + 26) + - b'@example> SIZE=1234') - self.assertEqual(self.channel.socket.last, - b'500 Error: line too long\r\n') - - def test_MAIL_command_rejects_SMTPUTF8_by_default(self): - self.write_line(b'EHLO example') - self.write_line( - b'MAIL from: BODY=8BITMIME SMTPUTF8') - self.assertEqual(self.channel.socket.last[0:1], b'5') - - def test_data_longer_than_default_data_size_limit(self): - # Hack the default so we don't have to generate so much data. - self.channel.data_size_limit = 1048 - self.write_line(b'HELO example') - self.write_line(b'MAIL From:eggs@example') - self.write_line(b'RCPT To:spam@example') - self.write_line(b'DATA') - self.write_line(b'A' * self.channel.data_size_limit + - b'A\r\n.') - self.assertEqual(self.channel.socket.last, - b'552 Error: Too much mail data\r\n') - - def test_MAIL_size_parameter(self): - self.write_line(b'EHLO example') - self.write_line(b'MAIL FROM: SIZE=512') - self.assertEqual(self.channel.socket.last, - b'250 OK\r\n') - - def test_MAIL_invalid_size_parameter(self): - self.write_line(b'EHLO example') - self.write_line(b'MAIL FROM: SIZE=invalid') - self.assertEqual(self.channel.socket.last, - b'501 Syntax: MAIL FROM:
    [SP ]\r\n') - - def test_MAIL_RCPT_unknown_parameters(self): - self.write_line(b'EHLO example') - self.write_line(b'MAIL FROM: ham=green') - self.assertEqual(self.channel.socket.last, - b'555 MAIL FROM parameters not recognized or not implemented\r\n') - - self.write_line(b'MAIL FROM:') - self.write_line(b'RCPT TO: ham=green') - self.assertEqual(self.channel.socket.last, - b'555 RCPT TO parameters not recognized or not implemented\r\n') - - def test_MAIL_size_parameter_larger_than_default_data_size_limit(self): - self.channel.data_size_limit = 1048 - self.write_line(b'EHLO example') - self.write_line(b'MAIL FROM: SIZE=2096') - self.assertEqual(self.channel.socket.last, - b'552 Error: message size exceeds fixed maximum message size\r\n') - - def test_need_MAIL(self): - self.write_line(b'HELO example') - self.write_line(b'RCPT to:spam@example') - self.assertEqual(self.channel.socket.last, - b'503 Error: need MAIL command\r\n') - - def test_MAIL_syntax_HELO(self): - self.write_line(b'HELO example') - self.write_line(b'MAIL from eggs@example') - self.assertEqual(self.channel.socket.last, - b'501 Syntax: MAIL FROM:
    \r\n') - - def test_MAIL_syntax_EHLO(self): - self.write_line(b'EHLO example') - self.write_line(b'MAIL from eggs@example') - self.assertEqual(self.channel.socket.last, - b'501 Syntax: MAIL FROM:
    [SP ]\r\n') - - def test_MAIL_missing_address(self): - self.write_line(b'HELO example') - self.write_line(b'MAIL from:') - self.assertEqual(self.channel.socket.last, - b'501 Syntax: MAIL FROM:
    \r\n') - - def test_MAIL_chevrons(self): - self.write_line(b'HELO example') - self.write_line(b'MAIL from:') - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - - def test_MAIL_empty_chevrons(self): - self.write_line(b'EHLO example') - self.write_line(b'MAIL from:<>') - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - - def test_MAIL_quoted_localpart(self): - self.write_line(b'EHLO example') - self.write_line(b'MAIL from: <"Fred Blogs"@example.com>') - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - self.assertEqual(self.channel.mailfrom, '"Fred Blogs"@example.com') - - def test_MAIL_quoted_localpart_no_angles(self): - self.write_line(b'EHLO example') - self.write_line(b'MAIL from: "Fred Blogs"@example.com') - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - self.assertEqual(self.channel.mailfrom, '"Fred Blogs"@example.com') - - def test_MAIL_quoted_localpart_with_size(self): - self.write_line(b'EHLO example') - self.write_line(b'MAIL from: <"Fred Blogs"@example.com> SIZE=1000') - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - self.assertEqual(self.channel.mailfrom, '"Fred Blogs"@example.com') - - def test_MAIL_quoted_localpart_with_size_no_angles(self): - self.write_line(b'EHLO example') - self.write_line(b'MAIL from: "Fred Blogs"@example.com SIZE=1000') - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - self.assertEqual(self.channel.mailfrom, '"Fred Blogs"@example.com') - - def test_nested_MAIL(self): - self.write_line(b'HELO example') - self.write_line(b'MAIL from:eggs@example') - self.write_line(b'MAIL from:spam@example') - self.assertEqual(self.channel.socket.last, - b'503 Error: nested MAIL command\r\n') - - def test_VRFY(self): - self.write_line(b'VRFY eggs@example') - self.assertEqual(self.channel.socket.last, - b'252 Cannot VRFY user, but will accept message and attempt ' + \ - b'delivery\r\n') - - def test_VRFY_syntax(self): - self.write_line(b'VRFY') - self.assertEqual(self.channel.socket.last, - b'501 Syntax: VRFY
    \r\n') - - def test_EXPN_not_implemented(self): - self.write_line(b'EXPN') - self.assertEqual(self.channel.socket.last, - b'502 EXPN not implemented\r\n') - - def test_no_HELO_MAIL(self): - self.write_line(b'MAIL from:') - self.assertEqual(self.channel.socket.last, - b'503 Error: send HELO first\r\n') - - def test_need_RCPT(self): - self.write_line(b'HELO example') - self.write_line(b'MAIL From:eggs@example') - self.write_line(b'DATA') - self.assertEqual(self.channel.socket.last, - b'503 Error: need RCPT command\r\n') - - def test_RCPT_syntax_HELO(self): - self.write_line(b'HELO example') - self.write_line(b'MAIL From: eggs@example') - self.write_line(b'RCPT to eggs@example') - self.assertEqual(self.channel.socket.last, - b'501 Syntax: RCPT TO:
    \r\n') - - def test_RCPT_syntax_EHLO(self): - self.write_line(b'EHLO example') - self.write_line(b'MAIL From: eggs@example') - self.write_line(b'RCPT to eggs@example') - self.assertEqual(self.channel.socket.last, - b'501 Syntax: RCPT TO:
    [SP ]\r\n') - - def test_RCPT_lowercase_to_OK(self): - self.write_line(b'HELO example') - self.write_line(b'MAIL From: eggs@example') - self.write_line(b'RCPT to: ') - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - - def test_no_HELO_RCPT(self): - self.write_line(b'RCPT to eggs@example') - self.assertEqual(self.channel.socket.last, - b'503 Error: send HELO first\r\n') - - def test_data_dialog(self): - self.write_line(b'HELO example') - self.write_line(b'MAIL From:eggs@example') - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - self.write_line(b'RCPT To:spam@example') - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - - self.write_line(b'DATA') - self.assertEqual(self.channel.socket.last, - b'354 End data with .\r\n') - self.write_line(b'data\r\nmore\r\n.') - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - self.assertEqual(self.server.messages, - [(('peer-address', 'peer-port'), - 'eggs@example', - ['spam@example'], - 'data\nmore')]) - - def test_DATA_syntax(self): - self.write_line(b'HELO example') - self.write_line(b'MAIL From:eggs@example') - self.write_line(b'RCPT To:spam@example') - self.write_line(b'DATA spam') - self.assertEqual(self.channel.socket.last, b'501 Syntax: DATA\r\n') - - def test_no_HELO_DATA(self): - self.write_line(b'DATA spam') - self.assertEqual(self.channel.socket.last, - b'503 Error: send HELO first\r\n') - - def test_data_transparency_section_4_5_2(self): - self.write_line(b'HELO example') - self.write_line(b'MAIL From:eggs@example') - self.write_line(b'RCPT To:spam@example') - self.write_line(b'DATA') - self.write_line(b'..\r\n.\r\n') - self.assertEqual(self.channel.received_data, '.') - - def test_multiple_RCPT(self): - self.write_line(b'HELO example') - self.write_line(b'MAIL From:eggs@example') - self.write_line(b'RCPT To:spam@example') - self.write_line(b'RCPT To:ham@example') - self.write_line(b'DATA') - self.write_line(b'data\r\n.') - self.assertEqual(self.server.messages, - [(('peer-address', 'peer-port'), - 'eggs@example', - ['spam@example','ham@example'], - 'data')]) - - def test_manual_status(self): - # checks that the Channel is able to return a custom status message - self.write_line(b'HELO example') - self.write_line(b'MAIL From:eggs@example') - self.write_line(b'RCPT To:spam@example') - self.write_line(b'DATA') - self.write_line(b'return status\r\n.') - self.assertEqual(self.channel.socket.last, b'250 Okish\r\n') - - def test_RSET(self): - self.write_line(b'HELO example') - self.write_line(b'MAIL From:eggs@example') - self.write_line(b'RCPT To:spam@example') - self.write_line(b'RSET') - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - self.write_line(b'MAIL From:foo@example') - self.write_line(b'RCPT To:eggs@example') - self.write_line(b'DATA') - self.write_line(b'data\r\n.') - self.assertEqual(self.server.messages, - [(('peer-address', 'peer-port'), - 'foo@example', - ['eggs@example'], - 'data')]) - - def test_HELO_RSET(self): - self.write_line(b'HELO example') - self.write_line(b'RSET') - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - - def test_RSET_syntax(self): - self.write_line(b'RSET hi') - self.assertEqual(self.channel.socket.last, b'501 Syntax: RSET\r\n') - - def test_unknown_command(self): - self.write_line(b'UNKNOWN_CMD') - self.assertEqual(self.channel.socket.last, - b'500 Error: command "UNKNOWN_CMD" not ' + \ - b'recognized\r\n') - - def test_attribute_deprecations(self): - with warnings_helper.check_warnings(('', DeprecationWarning)): - spam = self.channel._SMTPChannel__server - with warnings_helper.check_warnings(('', DeprecationWarning)): - self.channel._SMTPChannel__server = 'spam' - with warnings_helper.check_warnings(('', DeprecationWarning)): - spam = self.channel._SMTPChannel__line - with warnings_helper.check_warnings(('', DeprecationWarning)): - self.channel._SMTPChannel__line = 'spam' - with warnings_helper.check_warnings(('', DeprecationWarning)): - spam = self.channel._SMTPChannel__state - with warnings_helper.check_warnings(('', DeprecationWarning)): - self.channel._SMTPChannel__state = 'spam' - with warnings_helper.check_warnings(('', DeprecationWarning)): - spam = self.channel._SMTPChannel__greeting - with warnings_helper.check_warnings(('', DeprecationWarning)): - self.channel._SMTPChannel__greeting = 'spam' - with warnings_helper.check_warnings(('', DeprecationWarning)): - spam = self.channel._SMTPChannel__mailfrom - with warnings_helper.check_warnings(('', DeprecationWarning)): - self.channel._SMTPChannel__mailfrom = 'spam' - with warnings_helper.check_warnings(('', DeprecationWarning)): - spam = self.channel._SMTPChannel__rcpttos - with warnings_helper.check_warnings(('', DeprecationWarning)): - self.channel._SMTPChannel__rcpttos = 'spam' - with warnings_helper.check_warnings(('', DeprecationWarning)): - spam = self.channel._SMTPChannel__data - with warnings_helper.check_warnings(('', DeprecationWarning)): - self.channel._SMTPChannel__data = 'spam' - with warnings_helper.check_warnings(('', DeprecationWarning)): - spam = self.channel._SMTPChannel__fqdn - with warnings_helper.check_warnings(('', DeprecationWarning)): - self.channel._SMTPChannel__fqdn = 'spam' - with warnings_helper.check_warnings(('', DeprecationWarning)): - spam = self.channel._SMTPChannel__peer - with warnings_helper.check_warnings(('', DeprecationWarning)): - self.channel._SMTPChannel__peer = 'spam' - with warnings_helper.check_warnings(('', DeprecationWarning)): - spam = self.channel._SMTPChannel__conn - with warnings_helper.check_warnings(('', DeprecationWarning)): - self.channel._SMTPChannel__conn = 'spam' - with warnings_helper.check_warnings(('', DeprecationWarning)): - spam = self.channel._SMTPChannel__addr - with warnings_helper.check_warnings(('', DeprecationWarning)): - self.channel._SMTPChannel__addr = 'spam' - -@unittest.skipUnless(socket_helper.IPV6_ENABLED, "IPv6 not enabled") -class SMTPDChannelIPv6Test(SMTPDChannelTest): - def setUp(self): - smtpd.socket = asyncore.socket = mock_socket - self.old_debugstream = smtpd.DEBUGSTREAM - self.debug = smtpd.DEBUGSTREAM = io.StringIO() - self.server = DummyServer((socket_helper.HOSTv6, 0), ('b', 0), - decode_data=True) - conn, addr = self.server.accept() - self.channel = smtpd.SMTPChannel(self.server, conn, addr, - decode_data=True) - -class SMTPDChannelWithDataSizeLimitTest(unittest.TestCase): - - def setUp(self): - smtpd.socket = asyncore.socket = mock_socket - self.old_debugstream = smtpd.DEBUGSTREAM - self.debug = smtpd.DEBUGSTREAM = io.StringIO() - self.server = DummyServer((socket_helper.HOST, 0), ('b', 0), - decode_data=True) - conn, addr = self.server.accept() - # Set DATA size limit to 32 bytes for easy testing - self.channel = smtpd.SMTPChannel(self.server, conn, addr, 32, - decode_data=True) - - def tearDown(self): - asyncore.close_all() - asyncore.socket = smtpd.socket = socket - smtpd.DEBUGSTREAM = self.old_debugstream - - def write_line(self, line): - self.channel.socket.queue_recv(line) - self.channel.handle_read() - - def test_data_limit_dialog(self): - self.write_line(b'HELO example') - self.write_line(b'MAIL From:eggs@example') - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - self.write_line(b'RCPT To:spam@example') - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - - self.write_line(b'DATA') - self.assertEqual(self.channel.socket.last, - b'354 End data with .\r\n') - self.write_line(b'data\r\nmore\r\n.') - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - self.assertEqual(self.server.messages, - [(('peer-address', 'peer-port'), - 'eggs@example', - ['spam@example'], - 'data\nmore')]) - - def test_data_limit_dialog_too_much_data(self): - self.write_line(b'HELO example') - self.write_line(b'MAIL From:eggs@example') - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - self.write_line(b'RCPT To:spam@example') - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - - self.write_line(b'DATA') - self.assertEqual(self.channel.socket.last, - b'354 End data with .\r\n') - self.write_line(b'This message is longer than 32 bytes\r\n.') - self.assertEqual(self.channel.socket.last, - b'552 Error: Too much mail data\r\n') - - -class SMTPDChannelWithDecodeDataFalse(unittest.TestCase): - - def setUp(self): - smtpd.socket = asyncore.socket = mock_socket - self.old_debugstream = smtpd.DEBUGSTREAM - self.debug = smtpd.DEBUGSTREAM = io.StringIO() - self.server = DummyServer((socket_helper.HOST, 0), ('b', 0)) - conn, addr = self.server.accept() - self.channel = smtpd.SMTPChannel(self.server, conn, addr) - - def tearDown(self): - asyncore.close_all() - asyncore.socket = smtpd.socket = socket - smtpd.DEBUGSTREAM = self.old_debugstream - - def write_line(self, line): - self.channel.socket.queue_recv(line) - self.channel.handle_read() - - def test_ascii_data(self): - self.write_line(b'HELO example') - self.write_line(b'MAIL From:eggs@example') - self.write_line(b'RCPT To:spam@example') - self.write_line(b'DATA') - self.write_line(b'plain ascii text') - self.write_line(b'.') - self.assertEqual(self.channel.received_data, b'plain ascii text') - - def test_utf8_data(self): - self.write_line(b'HELO example') - self.write_line(b'MAIL From:eggs@example') - self.write_line(b'RCPT To:spam@example') - self.write_line(b'DATA') - self.write_line(b'utf8 enriched text: \xc5\xbc\xc5\xba\xc4\x87') - self.write_line(b'and some plain ascii') - self.write_line(b'.') - self.assertEqual( - self.channel.received_data, - b'utf8 enriched text: \xc5\xbc\xc5\xba\xc4\x87\n' - b'and some plain ascii') - - -class SMTPDChannelWithDecodeDataTrue(unittest.TestCase): - - def setUp(self): - smtpd.socket = asyncore.socket = mock_socket - self.old_debugstream = smtpd.DEBUGSTREAM - self.debug = smtpd.DEBUGSTREAM = io.StringIO() - self.server = DummyServer((socket_helper.HOST, 0), ('b', 0), - decode_data=True) - conn, addr = self.server.accept() - # Set decode_data to True - self.channel = smtpd.SMTPChannel(self.server, conn, addr, - decode_data=True) - - def tearDown(self): - asyncore.close_all() - asyncore.socket = smtpd.socket = socket - smtpd.DEBUGSTREAM = self.old_debugstream - - def write_line(self, line): - self.channel.socket.queue_recv(line) - self.channel.handle_read() - - def test_ascii_data(self): - self.write_line(b'HELO example') - self.write_line(b'MAIL From:eggs@example') - self.write_line(b'RCPT To:spam@example') - self.write_line(b'DATA') - self.write_line(b'plain ascii text') - self.write_line(b'.') - self.assertEqual(self.channel.received_data, 'plain ascii text') - - def test_utf8_data(self): - self.write_line(b'HELO example') - self.write_line(b'MAIL From:eggs@example') - self.write_line(b'RCPT To:spam@example') - self.write_line(b'DATA') - self.write_line(b'utf8 enriched text: \xc5\xbc\xc5\xba\xc4\x87') - self.write_line(b'and some plain ascii') - self.write_line(b'.') - self.assertEqual( - self.channel.received_data, - 'utf8 enriched text: żźć\nand some plain ascii') - - -class SMTPDChannelTestWithEnableSMTPUTF8True(unittest.TestCase): - def setUp(self): - smtpd.socket = asyncore.socket = mock_socket - self.old_debugstream = smtpd.DEBUGSTREAM - self.debug = smtpd.DEBUGSTREAM = io.StringIO() - self.server = DummyServer((socket_helper.HOST, 0), ('b', 0), - enable_SMTPUTF8=True) - conn, addr = self.server.accept() - self.channel = smtpd.SMTPChannel(self.server, conn, addr, - enable_SMTPUTF8=True) - - def tearDown(self): - asyncore.close_all() - asyncore.socket = smtpd.socket = socket - smtpd.DEBUGSTREAM = self.old_debugstream - - def write_line(self, line): - self.channel.socket.queue_recv(line) - self.channel.handle_read() - - def test_MAIL_command_accepts_SMTPUTF8_when_announced(self): - self.write_line(b'EHLO example') - self.write_line( - 'MAIL from: BODY=8BITMIME SMTPUTF8'.encode( - 'utf-8') - ) - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - - def test_process_smtputf8_message(self): - self.write_line(b'EHLO example') - for mail_parameters in [b'', b'BODY=8BITMIME SMTPUTF8']: - self.write_line(b'MAIL from: ' + mail_parameters) - self.assertEqual(self.channel.socket.last[0:3], b'250') - self.write_line(b'rcpt to:') - self.assertEqual(self.channel.socket.last[0:3], b'250') - self.write_line(b'data') - self.assertEqual(self.channel.socket.last[0:3], b'354') - self.write_line(b'c\r\n.') - if mail_parameters == b'': - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - else: - self.assertEqual(self.channel.socket.last, - b'250 SMTPUTF8 message okish\r\n') - - def test_utf8_data(self): - self.write_line(b'EHLO example') - self.write_line( - 'MAIL From: naïve@examplé BODY=8BITMIME SMTPUTF8'.encode('utf-8')) - self.assertEqual(self.channel.socket.last[0:3], b'250') - self.write_line('RCPT To:späm@examplé'.encode('utf-8')) - self.assertEqual(self.channel.socket.last[0:3], b'250') - self.write_line(b'DATA') - self.assertEqual(self.channel.socket.last[0:3], b'354') - self.write_line(b'utf8 enriched text: \xc5\xbc\xc5\xba\xc4\x87') - self.write_line(b'.') - self.assertEqual( - self.channel.received_data, - b'utf8 enriched text: \xc5\xbc\xc5\xba\xc4\x87') - - def test_MAIL_command_limit_extended_with_SIZE_and_SMTPUTF8(self): - self.write_line(b'ehlo example') - fill_len = (512 + 26 + 10) - len('mail from:<@example>') - self.write_line(b'MAIL from:<' + - b'a' * (fill_len + 1) + - b'@example>') - self.assertEqual(self.channel.socket.last, - b'500 Error: line too long\r\n') - self.write_line(b'MAIL from:<' + - b'a' * fill_len + - b'@example>') - self.assertEqual(self.channel.socket.last, b'250 OK\r\n') - - def test_multiple_emails_with_extended_command_length(self): - self.write_line(b'ehlo example') - fill_len = (512 + 26 + 10) - len('mail from:<@example>') - for char in [b'a', b'b', b'c']: - self.write_line(b'MAIL from:<' + char * fill_len + b'a@example>') - self.assertEqual(self.channel.socket.last[0:3], b'500') - self.write_line(b'MAIL from:<' + char * fill_len + b'@example>') - self.assertEqual(self.channel.socket.last[0:3], b'250') - self.write_line(b'rcpt to:') - self.assertEqual(self.channel.socket.last[0:3], b'250') - self.write_line(b'data') - self.assertEqual(self.channel.socket.last[0:3], b'354') - self.write_line(b'test\r\n.') - self.assertEqual(self.channel.socket.last[0:3], b'250') - - -class MiscTestCase(unittest.TestCase): - def test__all__(self): - not_exported = { - "program", "Devnull", "DEBUGSTREAM", "NEWLINE", "COMMASPACE", - "DATA_SIZE_DEFAULT", "usage", "Options", "parseargs", - } - support.check__all__(self, smtpd, not_exported=not_exported) - - -if __name__ == "__main__": - unittest.main() diff --git a/Lib/test/test_smtplib.py b/Lib/test/test_smtplib.py index a4074c02f1926f..3210ef217703e5 100644 --- a/Lib/test/test_smtplib.py +++ b/Lib/test/test_smtplib.py @@ -24,9 +24,9 @@ from test.support import warnings_helper from unittest.mock import Mock +from . import smtpd asyncore = warnings_helper.import_deprecated('asyncore') -smtpd = warnings_helper.import_deprecated('smtpd') support.requires_working_socket(module=True) diff --git a/Lib/test/test_stable_abi_ctypes.py b/Lib/test/test_stable_abi_ctypes.py index 53e93ab6b9b4c9..a803e3a5025985 100644 --- a/Lib/test/test_stable_abi_ctypes.py +++ b/Lib/test/test_stable_abi_ctypes.py @@ -782,6 +782,8 @@ def test_windows_feature_macros(self): "PyUnicode_Translate", "PyUnicode_Type", "PyUnicode_WriteChar", + "PyVectorcall_Call", + "PyVectorcall_NARGS", "PyWeakref_GetObject", "PyWeakref_NewProxy", "PyWeakref_NewRef", diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index c4798afc92016c..05fbcc19b6b7d7 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -1507,7 +1507,7 @@ def delx(self): del self.__x check((1,2,3), vsize('') + 3*self.P) # type # static type: PyTypeObject - fmt = 'P2nPI13Pl4Pn9Pn12PIPI' + fmt = 'P2nPI13Pl4Pn9Pn12PIP' s = vsize('2P' + fmt) check(int, s) # class diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py index 9765ed97a60a44..63bccb72e04646 100644 --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -2641,8 +2641,6 @@ def check_format(expected, format, *args): b'%c%c', c_int(0x10000), c_int(0x100000)) # test "%" - check_format('%', - b'%') check_format('%', b'%%') check_format('%s', @@ -2819,23 +2817,22 @@ def check_format(expected, format, *args): check_format('repr=abc\ufffd', b'repr=%V', None, b'abc\xff') - # not supported: copy the raw format string. these tests are just here - # to check for crashes and should not be considered as specifications - check_format('%s', - b'%1%s', b'abc') - check_format('%1abc', - b'%1abc') - check_format('%+i', - b'%+i', c_int(10)) - check_format('%.%s', - b'%.%s', b'abc') - # Issue #33817: empty strings check_format('', b'') check_format('', b'%s', b'') + # check for crashes + for fmt in (b'%', b'%0', b'%01', b'%.', b'%.1', + b'%0%s', b'%1%s', b'%.%s', b'%.1%s', b'%1abc', + b'%l', b'%ll', b'%z', b'%ls', b'%lls', b'%zs'): + with self.subTest(fmt=fmt): + self.assertRaisesRegex(SystemError, 'invalid format string', + PyUnicode_FromFormat, fmt, b'abc') + self.assertRaisesRegex(SystemError, 'invalid format string', + PyUnicode_FromFormat, b'%+i', c_int(10)) + # Test PyUnicode_AsWideChar() @support.cpython_only @unittest.skipIf(_testcapi is None, 'need _testcapi module') diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py index 88270b7537b5e7..28f88412fdcaac 100644 --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -1556,11 +1556,11 @@ def test_proxy_basic_auth(self): def test_basic_and_digest_auth_handlers(self): # HTTPDigestAuthHandler raised an exception if it couldn't handle a 40* - # response (http://python.org/sf/1479302), where it should instead + # response (https://bugs.python.org/issue1479302), where it should instead # return None to allow another handler (especially # HTTPBasicAuthHandler) to handle the response. - # Also (http://python.org/sf/14797027, RFC 2617 section 1.2), we must + # Also (https://bugs.python.org/issue14797027, RFC 2617 section 1.2), we must # try digest first (since it's the strongest auth scheme), so we record # order of calls here to check digest comes first: class RecordingOpenerDirector(OpenerDirector): diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py index fa0ca5aa7428ac..21257785159b37 100644 --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -2032,6 +2032,7 @@ def test_seek_tell(self): fp.seek(bloc, os.SEEK_CUR) self.assertEqual(fp.tell(), bloc) self.assertEqual(fp.read(5), txt[bloc:bloc+5]) + self.assertEqual(fp.tell(), bloc + 5) fp.seek(0, os.SEEK_END) self.assertEqual(fp.tell(), len(txt)) fp.seek(0, os.SEEK_SET) @@ -2049,6 +2050,7 @@ def test_seek_tell(self): fp.seek(bloc, os.SEEK_CUR) self.assertEqual(fp.tell(), bloc) self.assertEqual(fp.read(5), txt[bloc:bloc+5]) + self.assertEqual(fp.tell(), bloc + 5) fp.seek(0, os.SEEK_END) self.assertEqual(fp.tell(), len(txt)) fp.seek(0, os.SEEK_SET) diff --git a/Lib/zipfile.py b/Lib/zipfile.py index e3b7a61a6399be..981560082cab7a 100644 --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -847,6 +847,7 @@ def __init__(self, fileobj, mode, zipinfo, pwd=None, self._orig_compress_size = zipinfo.compress_size self._orig_file_size = zipinfo.file_size self._orig_start_crc = self._running_crc + self._orig_crc = self._expected_crc self._seekable = True except AttributeError: pass @@ -1069,17 +1070,17 @@ def seekable(self): raise ValueError("I/O operation on closed file.") return self._seekable - def seek(self, offset, whence=0): + def seek(self, offset, whence=os.SEEK_SET): if self.closed: raise ValueError("seek on closed file.") if not self._seekable: raise io.UnsupportedOperation("underlying stream is not seekable") curr_pos = self.tell() - if whence == 0: # Seek from start of file + if whence == os.SEEK_SET: new_pos = offset - elif whence == 1: # Seek from current position + elif whence == os.SEEK_CUR: new_pos = curr_pos + offset - elif whence == 2: # Seek from EOF + elif whence == os.SEEK_END: new_pos = self._orig_file_size + offset else: raise ValueError("whence must be os.SEEK_SET (0), " @@ -1094,7 +1095,19 @@ def seek(self, offset, whence=0): read_offset = new_pos - curr_pos buff_offset = read_offset + self._offset - if buff_offset >= 0 and buff_offset < len(self._readbuffer): + # Fast seek uncompressed unencrypted file + if self._compress_type == ZIP_STORED and self._decrypter is None and read_offset > 0: + # disable CRC checking after first seeking - it would be invalid + self._expected_crc = None + # seek actual file taking already buffered data into account + read_offset -= len(self._readbuffer) - self._offset + self._fileobj.seek(read_offset, os.SEEK_CUR) + self._left -= read_offset + read_offset = 0 + # flush read buffer + self._readbuffer = b'' + self._offset = 0 + elif buff_offset >= 0 and buff_offset < len(self._readbuffer): # Just move the _offset index if the new position is in the _readbuffer self._offset = buff_offset read_offset = 0 @@ -1102,6 +1115,7 @@ def seek(self, offset, whence=0): # Position is before the current position. Reset the ZipExtFile self._fileobj.seek(self._orig_compress_start) self._running_crc = self._orig_start_crc + self._expected_crc = self._orig_crc self._compress_left = self._orig_compress_size self._left = self._orig_file_size self._readbuffer = b'' diff --git a/Makefile.pre.in b/Makefile.pre.in index 746ff4226e6c43..79616160e495e3 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -36,7 +36,6 @@ abs_builddir= @abs_builddir@ CC= @CC@ CXX= @CXX@ -MAINCC= @MAINCC@ LINKCC= @LINKCC@ AR= @AR@ READELF= @READELF@ @@ -1222,10 +1221,10 @@ Modules/getpath.o: $(srcdir)/Modules/getpath.c Python/frozen_modules/getpath.h M -o $@ $(srcdir)/Modules/getpath.c Programs/python.o: $(srcdir)/Programs/python.c - $(MAINCC) -c $(PY_CORE_CFLAGS) -o $@ $(srcdir)/Programs/python.c + $(CC) -c $(PY_CORE_CFLAGS) -o $@ $(srcdir)/Programs/python.c Programs/_testembed.o: $(srcdir)/Programs/_testembed.c Programs/test_frozenmain.h - $(MAINCC) -c $(PY_CORE_CFLAGS) -o $@ $(srcdir)/Programs/_testembed.c + $(CC) -c $(PY_CORE_CFLAGS) -o $@ $(srcdir)/Programs/_testembed.c Modules/_sre/sre.o: $(srcdir)/Modules/_sre/sre.c $(srcdir)/Modules/_sre/sre.h $(srcdir)/Modules/_sre/sre_constants.h $(srcdir)/Modules/_sre/sre_lib.h diff --git a/Misc/ACKS b/Misc/ACKS index b18fabe09ef4f8..7065267379deb1 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -609,6 +609,7 @@ Jan-Philip Gehrcke Thomas Gellekum Gabriel Genellina Christos Georgiou +Philip Georgi Elazar (אלעזר) Gershuni Ben Gertzfield Nadim Ghaznavi diff --git a/Misc/HISTORY b/Misc/HISTORY index 570638869f92ed..8ca89854ed13b5 100644 --- a/Misc/HISTORY +++ b/Misc/HISTORY @@ -21790,7 +21790,7 @@ Library x == y is False, and x != y is True. This is akin to the change made for mixed-type comparisons of datetime objects in 2.3a2; more info about the rationale is in the NEWS entry for that. See also SF bug - report . + report . - On Unix platforms, if os.listdir() is called with a Unicode argument, it now returns Unicode strings. (This behavior was added earlier @@ -22025,7 +22025,7 @@ Extension modules now. today() and now() now round system timestamps to the closest - microsecond . This repairs an + microsecond . This repairs an irritation most likely seen on Windows systems. In dt.astimezone(tz), if tz.utcoffset(dt) returns a duration, @@ -22080,7 +22080,7 @@ Extension modules datetime.fromtimestamp(): Like datetime.now() above, this had less than useful behavior when the optional tinzo argument was specified. See - also SF bug report . + also SF bug report . date and datetime comparison: In order to prevent comparison from falling back to the default compare-object-addresses strategy, these @@ -22139,10 +22139,10 @@ Library dependent path modules (e.g. ntpath.py) rather than os.py, so these variables are now available via os.path. They continue to be available from the os module. - (see ). + (see ). - array.array was added to the types repr.py knows about (see - ). + ). - The new pickletools.py contains lots of documentation about pickle internals, and supplies some helpers for working with pickles, such as @@ -22527,7 +22527,7 @@ Core and builtins potential drawback is that list.sort() may require temp space of len(list)*2 bytes (``*4`` on a 64-bit machine). It's therefore possible for list.sort() to raise MemoryError now, even if a comparison function - does not. See for full details. + does not. See for full details. - All standard iterators now ensure that, once StopIteration has been raised, all future calls to next() on the same iterator will also diff --git a/Misc/NEWS.d/3.9.0a1.rst b/Misc/NEWS.d/3.9.0a1.rst index 45f232f1948d5d..eace8755a0d171 100644 --- a/Misc/NEWS.d/3.9.0a1.rst +++ b/Misc/NEWS.d/3.9.0a1.rst @@ -5769,4 +5769,4 @@ Convert posixmodule.c statically allocated types ``DirEntryType`` and .. section: C API Use singular/plural noun in error message when instantiating an abstract -class with non-overriden abstract method(s). +class with non-overridden abstract method(s). diff --git a/Misc/NEWS.d/next/Build/2022-08-04-15-29-35.gh-issue-93744.svRuqm.rst b/Misc/NEWS.d/next/Build/2022-08-04-15-29-35.gh-issue-93744.svRuqm.rst new file mode 100644 index 00000000000000..fa9ade25718fcb --- /dev/null +++ b/Misc/NEWS.d/next/Build/2022-08-04-15-29-35.gh-issue-93744.svRuqm.rst @@ -0,0 +1,3 @@ +Remove the ``configure --with-cxx-main`` build option: it didn't work for +many years. Remove the ``MAINCC`` variable from ``configure`` and +``Makefile``. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/C API/2022-07-29-10-41-59.gh-issue-95388.aiRSgr.rst b/Misc/NEWS.d/next/C API/2022-07-29-10-41-59.gh-issue-95388.aiRSgr.rst new file mode 100644 index 00000000000000..c389d13db6afc2 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2022-07-29-10-41-59.gh-issue-95388.aiRSgr.rst @@ -0,0 +1,2 @@ +Creating :c:data:`immutable types ` with mutable +bases is deprecated and is planned to be disabled in Python 3.14. diff --git a/Misc/NEWS.d/next/C API/2022-07-29-15-24-45.gh-issue-93012.-DdGEy.rst b/Misc/NEWS.d/next/C API/2022-07-29-15-24-45.gh-issue-93012.-DdGEy.rst new file mode 100644 index 00000000000000..199c2d1fc197dd --- /dev/null +++ b/Misc/NEWS.d/next/C API/2022-07-29-15-24-45.gh-issue-93012.-DdGEy.rst @@ -0,0 +1,6 @@ +The :const:`Py_TPFLAGS_HAVE_VECTORCALL` flag is now removed from a class +when the class's :py:meth:`~object.__call__` method is reassigned. This +makes vectorcall safe to use with mutable types (i.e. heap types without the +:const:`immutable ` flag). Mutable types that do +not override :c:member:`~PyTypeObject.tp_call` now inherit the +:const:`Py_TPFLAGS_HAVE_VECTORCALL` flag. diff --git a/Misc/NEWS.d/next/C API/2022-08-01-16-21-39.gh-issue-93274.QoDHEu.rst b/Misc/NEWS.d/next/C API/2022-08-01-16-21-39.gh-issue-93274.QoDHEu.rst new file mode 100644 index 00000000000000..da6cce4a7b283a --- /dev/null +++ b/Misc/NEWS.d/next/C API/2022-08-01-16-21-39.gh-issue-93274.QoDHEu.rst @@ -0,0 +1,3 @@ +API for implementing vectorcall (:c:data:`Py_TPFLAGS_HAVE_VECTORCALL`, +:c:func:`PyVectorcall_NARGS` and :c:func:`PyVectorcall_Call`) was added to +the limited API and stable ABI. diff --git a/Misc/NEWS.d/next/C API/2022-08-08-14-36-31.gh-issue-95781.W_G8YW.rst b/Misc/NEWS.d/next/C API/2022-08-08-14-36-31.gh-issue-95781.W_G8YW.rst new file mode 100644 index 00000000000000..eb2fd7e9da3d81 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2022-08-08-14-36-31.gh-issue-95781.W_G8YW.rst @@ -0,0 +1,4 @@ +An unrecognized format character in :c:func:`PyUnicode_FromFormat` and +:c:func:`PyUnicode_FromFormatV` now sets a :exc:`SystemError`. +In previous versions it caused all the rest of the format string to be +copied as-is to the result string, and any extra arguments discarded. diff --git a/Misc/NEWS.d/next/IDLE/2022-08-04-20-07-51.gh-issue-65802.xnThWe.rst b/Misc/NEWS.d/next/IDLE/2022-08-04-20-07-51.gh-issue-65802.xnThWe.rst new file mode 100644 index 00000000000000..a62a784b6e6926 --- /dev/null +++ b/Misc/NEWS.d/next/IDLE/2022-08-04-20-07-51.gh-issue-65802.xnThWe.rst @@ -0,0 +1 @@ +Document handling of extensions in Save As dialogs. diff --git a/Misc/NEWS.d/next/Library/2021-08-27-18-07-35.bpo-44173.oW92Ev.rst b/Misc/NEWS.d/next/Library/2021-08-27-18-07-35.bpo-44173.oW92Ev.rst new file mode 100644 index 00000000000000..abc98266afb0ce --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-08-27-18-07-35.bpo-44173.oW92Ev.rst @@ -0,0 +1 @@ +Enable fast seeking of uncompressed unencrypted :class:`zipfile.ZipExtFile` diff --git a/Misc/NEWS.d/next/Library/2022-02-21-01-37-00.bpo-42777.nWK3E6.rst b/Misc/NEWS.d/next/Library/2022-02-21-01-37-00.bpo-42777.nWK3E6.rst new file mode 100644 index 00000000000000..24912380fb590a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-02-21-01-37-00.bpo-42777.nWK3E6.rst @@ -0,0 +1 @@ +Implement :meth:`pathlib.Path.is_mount` for Windows paths. diff --git a/Misc/NEWS.d/next/Library/2022-05-26-08-41-34.gh-issue-93243.uw6x5z.rst b/Misc/NEWS.d/next/Library/2022-05-26-08-41-34.gh-issue-93243.uw6x5z.rst new file mode 100644 index 00000000000000..f03ed7b5efc546 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-05-26-08-41-34.gh-issue-93243.uw6x5z.rst @@ -0,0 +1 @@ +The :mod:`smtpd` module was removed per the schedule in :pep:`594`. diff --git a/Misc/NEWS.d/next/Library/2022-07-06-14-57-33.gh-issue-94619.PRqKVX.rst b/Misc/NEWS.d/next/Library/2022-07-06-14-57-33.gh-issue-94619.PRqKVX.rst new file mode 100644 index 00000000000000..a7905034424684 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-07-06-14-57-33.gh-issue-94619.PRqKVX.rst @@ -0,0 +1 @@ +Remove the long-deprecated `module_repr()` from `importlib`. diff --git a/Misc/NEWS.d/next/Library/2022-07-28-17-14-38.gh-issue-95385.6YlsDI.rst b/Misc/NEWS.d/next/Library/2022-07-28-17-14-38.gh-issue-95385.6YlsDI.rst new file mode 100644 index 00000000000000..89fa9c2b27664d --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-07-28-17-14-38.gh-issue-95385.6YlsDI.rst @@ -0,0 +1 @@ +Faster ``json.dumps()`` when sorting of keys is not requested (default). diff --git a/Misc/NEWS.d/next/Tests/2022-08-05-09-57-43.gh-issue-95573.edMdQB.rst b/Misc/NEWS.d/next/Tests/2022-08-05-09-57-43.gh-issue-95573.edMdQB.rst new file mode 100644 index 00000000000000..8580556965e1da --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2022-08-05-09-57-43.gh-issue-95573.edMdQB.rst @@ -0,0 +1,6 @@ +:source:`Lib/test/test_asyncio/test_ssl.py` exposed a bug in the macOS +kernel where intense concurrent load on non-blocking sockets occasionally +causes :const:`errno.ENOBUFS` ("No buffer space available") to be emitted. +FB11063974 filed with Apple, in the mean time as a workaround buffer size +used in tests on macOS is decreased to avoid intermittent failures. Patch +by Fantix King. diff --git a/Misc/NEWS.d/next/Windows/2022-08-04-01-12-27.gh-issue-95587.Fvdv5q.rst b/Misc/NEWS.d/next/Windows/2022-08-04-01-12-27.gh-issue-95587.Fvdv5q.rst new file mode 100644 index 00000000000000..1033e892876c91 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2022-08-04-01-12-27.gh-issue-95587.Fvdv5q.rst @@ -0,0 +1,2 @@ +Fixes some issues where the Windows installer would incorrectly detect +certain features of an existing install when upgrading. diff --git a/Misc/NEWS.d/next/Windows/2022-08-04-18-47-54.gh-issue-95656.VJ1d13.rst b/Misc/NEWS.d/next/Windows/2022-08-04-18-47-54.gh-issue-95656.VJ1d13.rst new file mode 100644 index 00000000000000..77fea4c33f7a92 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2022-08-04-18-47-54.gh-issue-95656.VJ1d13.rst @@ -0,0 +1,2 @@ +Enable the :meth:`~sqlite3.Connection.enable_load_extension` :mod:`sqlite3` +API. diff --git a/Misc/stable_abi.toml b/Misc/stable_abi.toml index 84bec827096050..4da002a0586299 100644 --- a/Misc/stable_abi.toml +++ b/Misc/stable_abi.toml @@ -2275,5 +2275,14 @@ added = '3.11' [function.PyErr_SetHandledException] added = '3.11' + [function.PyType_FromMetaclass] added = '3.12' +[const.Py_TPFLAGS_HAVE_VECTORCALL] + added = '3.12' +[function.PyVectorcall_NARGS] + added = '3.12' +[function.PyVectorcall_Call] + added = '3.12' +[typedef.vectorcallfunc] + added = '3.12' diff --git a/Modules/Setup.stdlib.in b/Modules/Setup.stdlib.in index c5dc1e8eb45377..ac8959ebea5bf2 100644 --- a/Modules/Setup.stdlib.in +++ b/Modules/Setup.stdlib.in @@ -169,7 +169,7 @@ @MODULE__XXTESTFUZZ_TRUE@_xxtestfuzz _xxtestfuzz/_xxtestfuzz.c _xxtestfuzz/fuzzer.c @MODULE__TESTBUFFER_TRUE@_testbuffer _testbuffer.c @MODULE__TESTINTERNALCAPI_TRUE@_testinternalcapi _testinternalcapi.c -@MODULE__TESTCAPI_TRUE@_testcapi _testcapimodule.c _testcapi/vectorcall.c _testcapi/heaptype.c +@MODULE__TESTCAPI_TRUE@_testcapi _testcapimodule.c _testcapi/vectorcall.c _testcapi/vectorcall_limited.c _testcapi/heaptype.c _testcapi/unicode.c # Some testing modules MUST be built as shared libraries. *shared* diff --git a/Modules/_json.c b/Modules/_json.c index 7ea84efdb911bd..1c39b46937d792 100644 --- a/Modules/_json.c +++ b/Modules/_json.c @@ -12,6 +12,7 @@ #include "Python.h" #include "pycore_ceval.h" // _Py_EnterRecursiveCall() #include "structmember.h" // PyMemberDef +#include // bool typedef struct _PyScannerObject { @@ -1491,17 +1492,79 @@ encoder_listencode_obj(PyEncoderObject *s, _PyUnicodeWriter *writer, } } +static int +encoder_encode_key_value(PyEncoderObject *s, _PyUnicodeWriter *writer, bool *first, + PyObject *key, PyObject *value, Py_ssize_t indent_level) +{ + PyObject *keystr = NULL; + PyObject *encoded; + + if (PyUnicode_Check(key)) { + Py_INCREF(key); + keystr = key; + } + else if (PyFloat_Check(key)) { + keystr = encoder_encode_float(s, key); + } + else if (key == Py_True || key == Py_False || key == Py_None) { + /* This must come before the PyLong_Check because + True and False are also 1 and 0.*/ + keystr = _encoded_const(key); + } + else if (PyLong_Check(key)) { + keystr = PyLong_Type.tp_repr(key); + } + else if (s->skipkeys) { + return 0; + } + else { + PyErr_Format(PyExc_TypeError, + "keys must be str, int, float, bool or None, " + "not %.100s", Py_TYPE(key)->tp_name); + return -1; + } + + if (keystr == NULL) { + return -1; + } + + if (*first) { + *first = false; + } + else { + if (_PyUnicodeWriter_WriteStr(writer, s->item_separator) < 0) { + Py_DECREF(keystr); + return -1; + } + } + + encoded = encoder_encode_string(s, keystr); + Py_DECREF(keystr); + if (encoded == NULL) { + return -1; + } + + if (_steal_accumulate(writer, encoded) < 0) { + return -1; + } + if (_PyUnicodeWriter_WriteStr(writer, s->key_separator) < 0) { + return -1; + } + if (encoder_listencode_obj(s, writer, value, indent_level) < 0) { + return -1; + } + return 0; +} + static int encoder_listencode_dict(PyEncoderObject *s, _PyUnicodeWriter *writer, PyObject *dct, Py_ssize_t indent_level) { /* Encode Python dict dct a JSON term */ - PyObject *kstr = NULL; PyObject *ident = NULL; - PyObject *it = NULL; - PyObject *items; - PyObject *item = NULL; - Py_ssize_t idx; + PyObject *items = NULL; + PyObject *key, *value; + bool first = true; if (PyDict_GET_SIZE(dct) == 0) /* Fast path */ return _PyUnicodeWriter_WriteASCIIString(writer, "{}", 2); @@ -1535,84 +1598,34 @@ encoder_listencode_dict(PyEncoderObject *s, _PyUnicodeWriter *writer, */ } - items = PyMapping_Items(dct); - if (items == NULL) - goto bail; - if (s->sort_keys && PyList_Sort(items) < 0) { - Py_DECREF(items); - goto bail; - } - it = PyObject_GetIter(items); - Py_DECREF(items); - if (it == NULL) - goto bail; - idx = 0; - while ((item = PyIter_Next(it)) != NULL) { - PyObject *encoded, *key, *value; - if (!PyTuple_Check(item) || PyTuple_GET_SIZE(item) != 2) { - PyErr_SetString(PyExc_ValueError, "items must return 2-tuples"); + if (s->sort_keys) { + + items = PyDict_Items(dct); + if (items == NULL || PyList_Sort(items) < 0) goto bail; - } - key = PyTuple_GET_ITEM(item, 0); - if (PyUnicode_Check(key)) { - Py_INCREF(key); - kstr = key; - } - else if (PyFloat_Check(key)) { - kstr = encoder_encode_float(s, key); - if (kstr == NULL) - goto bail; - } - else if (key == Py_True || key == Py_False || key == Py_None) { - /* This must come before the PyLong_Check because - True and False are also 1 and 0.*/ - kstr = _encoded_const(key); - if (kstr == NULL) - goto bail; - } - else if (PyLong_Check(key)) { - kstr = PyLong_Type.tp_repr(key); - if (kstr == NULL) { + + for (Py_ssize_t i = 0; i < PyList_GET_SIZE(items); i++) { + PyObject *item = PyList_GET_ITEM(items, i); + + if (!PyTuple_Check(item) || PyTuple_GET_SIZE(item) != 2) { + PyErr_SetString(PyExc_ValueError, "items must return 2-tuples"); goto bail; } - } - else if (s->skipkeys) { - Py_DECREF(item); - continue; - } - else { - PyErr_Format(PyExc_TypeError, - "keys must be str, int, float, bool or None, " - "not %.100s", Py_TYPE(key)->tp_name); - goto bail; - } - if (idx) { - if (_PyUnicodeWriter_WriteStr(writer, s->item_separator)) + key = PyTuple_GET_ITEM(item, 0); + value = PyTuple_GET_ITEM(item, 1); + if (encoder_encode_key_value(s, writer, &first, key, value, indent_level) < 0) goto bail; } + Py_CLEAR(items); - encoded = encoder_encode_string(s, kstr); - Py_CLEAR(kstr); - if (encoded == NULL) - goto bail; - if (_PyUnicodeWriter_WriteStr(writer, encoded)) { - Py_DECREF(encoded); - goto bail; + } else { + Py_ssize_t pos = 0; + while (PyDict_Next(dct, &pos, &key, &value)) { + if (encoder_encode_key_value(s, writer, &first, key, value, indent_level) < 0) + goto bail; } - Py_DECREF(encoded); - if (_PyUnicodeWriter_WriteStr(writer, s->key_separator)) - goto bail; - - value = PyTuple_GET_ITEM(item, 1); - if (encoder_listencode_obj(s, writer, value, indent_level)) - goto bail; - idx += 1; - Py_DECREF(item); } - if (PyErr_Occurred()) - goto bail; - Py_CLEAR(it); if (ident != NULL) { if (PyDict_DelItem(s->markers, ident)) @@ -1630,14 +1643,11 @@ encoder_listencode_dict(PyEncoderObject *s, _PyUnicodeWriter *writer, return 0; bail: - Py_XDECREF(it); - Py_XDECREF(item); - Py_XDECREF(kstr); + Py_XDECREF(items); Py_XDECREF(ident); return -1; } - static int encoder_listencode_list(PyEncoderObject *s, _PyUnicodeWriter *writer, PyObject *seq, Py_ssize_t indent_level) diff --git a/Modules/_testcapi/clinic/vectorcall.c.h b/Modules/_testcapi/clinic/vectorcall.c.h new file mode 100644 index 00000000000000..14cdf23304be94 --- /dev/null +++ b/Modules/_testcapi/clinic/vectorcall.c.h @@ -0,0 +1,107 @@ +/*[clinic input] +preserve +[clinic start generated code]*/ + +PyDoc_STRVAR(_testcapi_VectorCallClass_set_vectorcall__doc__, +"set_vectorcall($self, type, /)\n" +"--\n" +"\n" +"Set self\'s vectorcall function for `type` to one that returns \"vectorcall\""); + +#define _TESTCAPI_VECTORCALLCLASS_SET_VECTORCALL_METHODDEF \ + {"set_vectorcall", (PyCFunction)_testcapi_VectorCallClass_set_vectorcall, METH_O, _testcapi_VectorCallClass_set_vectorcall__doc__}, + +static PyObject * +_testcapi_VectorCallClass_set_vectorcall_impl(PyObject *self, + PyTypeObject *type); + +static PyObject * +_testcapi_VectorCallClass_set_vectorcall(PyObject *self, PyObject *arg) +{ + PyObject *return_value = NULL; + PyTypeObject *type; + + if (!PyObject_TypeCheck(arg, &PyType_Type)) { + _PyArg_BadArgument("set_vectorcall", "argument", (&PyType_Type)->tp_name, arg); + goto exit; + } + type = (PyTypeObject *)arg; + return_value = _testcapi_VectorCallClass_set_vectorcall_impl(self, type); + +exit: + return return_value; +} + +PyDoc_STRVAR(_testcapi_make_vectorcall_class__doc__, +"make_vectorcall_class($module, base=, /)\n" +"--\n" +"\n" +"Create a class whose instances return \"tpcall\" when called.\n" +"\n" +"When the \"set_vectorcall\" method is called on an instance, a vectorcall\n" +"function that returns \"vectorcall\" will be installed."); + +#define _TESTCAPI_MAKE_VECTORCALL_CLASS_METHODDEF \ + {"make_vectorcall_class", _PyCFunction_CAST(_testcapi_make_vectorcall_class), METH_FASTCALL, _testcapi_make_vectorcall_class__doc__}, + +static PyObject * +_testcapi_make_vectorcall_class_impl(PyObject *module, PyTypeObject *base); + +static PyObject * +_testcapi_make_vectorcall_class(PyObject *module, PyObject *const *args, Py_ssize_t nargs) +{ + PyObject *return_value = NULL; + PyTypeObject *base = NULL; + + if (!_PyArg_CheckPositional("make_vectorcall_class", nargs, 0, 1)) { + goto exit; + } + if (nargs < 1) { + goto skip_optional; + } + if (!PyObject_TypeCheck(args[0], &PyType_Type)) { + _PyArg_BadArgument("make_vectorcall_class", "argument 1", (&PyType_Type)->tp_name, args[0]); + goto exit; + } + base = (PyTypeObject *)args[0]; +skip_optional: + return_value = _testcapi_make_vectorcall_class_impl(module, base); + +exit: + return return_value; +} + +PyDoc_STRVAR(_testcapi_has_vectorcall_flag__doc__, +"has_vectorcall_flag($module, type, /)\n" +"--\n" +"\n" +"Return true iff Py_TPFLAGS_HAVE_VECTORCALL is set on the class."); + +#define _TESTCAPI_HAS_VECTORCALL_FLAG_METHODDEF \ + {"has_vectorcall_flag", (PyCFunction)_testcapi_has_vectorcall_flag, METH_O, _testcapi_has_vectorcall_flag__doc__}, + +static int +_testcapi_has_vectorcall_flag_impl(PyObject *module, PyTypeObject *type); + +static PyObject * +_testcapi_has_vectorcall_flag(PyObject *module, PyObject *arg) +{ + PyObject *return_value = NULL; + PyTypeObject *type; + int _return_value; + + if (!PyObject_TypeCheck(arg, &PyType_Type)) { + _PyArg_BadArgument("has_vectorcall_flag", "argument", (&PyType_Type)->tp_name, arg); + goto exit; + } + type = (PyTypeObject *)arg; + _return_value = _testcapi_has_vectorcall_flag_impl(module, type); + if ((_return_value == -1) && PyErr_Occurred()) { + goto exit; + } + return_value = PyBool_FromLong((long)_return_value); + +exit: + return return_value; +} +/*[clinic end generated code: output=cf39927be151aebd input=a9049054013a1b77]*/ diff --git a/Modules/_testcapi/heaptype.c b/Modules/_testcapi/heaptype.c index 12889e825d5581..514541ce348dab 100644 --- a/Modules/_testcapi/heaptype.c +++ b/Modules/_testcapi/heaptype.c @@ -365,6 +365,21 @@ create_type_from_repeated_slots(PyObject *self, PyObject *variant_obj) } + +static PyObject * +make_immutable_type_with_base(PyObject *self, PyObject *base) +{ + assert(PyType_Check(base)); + PyType_Spec ImmutableSubclass_spec = { + .name = "ImmutableSubclass", + .basicsize = (int)((PyTypeObject*)base)->tp_basicsize, + .slots = empty_type_slots, + .flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_IMMUTABLETYPE, + }; + return PyType_FromSpecWithBases(&ImmutableSubclass_spec, base); +} + + static PyMethodDef TestMethods[] = { {"pytype_fromspec_meta", pytype_fromspec_meta, METH_O}, {"test_type_from_ephemeral_spec", test_type_from_ephemeral_spec, METH_NOARGS}, @@ -375,6 +390,7 @@ static PyMethodDef TestMethods[] = { {"test_from_spec_invalid_metatype_inheritance", test_from_spec_invalid_metatype_inheritance, METH_NOARGS}, + {"make_immutable_type_with_base", make_immutable_type_with_base, METH_O}, {NULL}, }; diff --git a/Modules/_testcapi/parts.h b/Modules/_testcapi/parts.h index e6d2ed23cb18e7..c0e0f3aa1cc21b 100644 --- a/Modules/_testcapi/parts.h +++ b/Modules/_testcapi/parts.h @@ -1,4 +1,6 @@ #include "Python.h" int _PyTestCapi_Init_Vectorcall(PyObject *module); +int _PyTestCapi_Init_VectorcallLimited(PyObject *module); int _PyTestCapi_Init_Heaptype(PyObject *module); +int _PyTestCapi_Init_Unicode(PyObject *module); diff --git a/Modules/_testcapi/unicode.c b/Modules/_testcapi/unicode.c new file mode 100644 index 00000000000000..58214249e22527 --- /dev/null +++ b/Modules/_testcapi/unicode.c @@ -0,0 +1,653 @@ +#include "parts.h" + +static struct PyModuleDef *_testcapimodule = NULL; // set at initialization + +static PyObject * +codec_incrementalencoder(PyObject *self, PyObject *args) +{ + const char *encoding, *errors = NULL; + if (!PyArg_ParseTuple(args, "s|s:test_incrementalencoder", + &encoding, &errors)) + return NULL; + return PyCodec_IncrementalEncoder(encoding, errors); +} + +static PyObject * +codec_incrementaldecoder(PyObject *self, PyObject *args) +{ + const char *encoding, *errors = NULL; + if (!PyArg_ParseTuple(args, "s|s:test_incrementaldecoder", + &encoding, &errors)) + return NULL; + return PyCodec_IncrementalDecoder(encoding, errors); +} + +static PyObject * +test_unicode_compare_with_ascii(PyObject *self, PyObject *Py_UNUSED(ignored)) { + PyObject *py_s = PyUnicode_FromStringAndSize("str\0", 4); + int result; + if (py_s == NULL) + return NULL; + result = PyUnicode_CompareWithASCIIString(py_s, "str"); + Py_DECREF(py_s); + if (!result) { + PyErr_SetString(PyExc_AssertionError, "Python string ending in NULL " + "should not compare equal to c string."); + return NULL; + } + Py_RETURN_NONE; +} + +static PyObject * +test_widechar(PyObject *self, PyObject *Py_UNUSED(ignored)) +{ +#if defined(SIZEOF_WCHAR_T) && (SIZEOF_WCHAR_T == 4) + const wchar_t wtext[2] = {(wchar_t)0x10ABCDu}; + size_t wtextlen = 1; + const wchar_t invalid[1] = {(wchar_t)0x110000u}; +#else + const wchar_t wtext[3] = {(wchar_t)0xDBEAu, (wchar_t)0xDFCDu}; + size_t wtextlen = 2; +#endif + PyObject *wide, *utf8; + + wide = PyUnicode_FromWideChar(wtext, wtextlen); + if (wide == NULL) + return NULL; + + utf8 = PyUnicode_FromString("\xf4\x8a\xaf\x8d"); + if (utf8 == NULL) { + Py_DECREF(wide); + return NULL; + } + + if (PyUnicode_GET_LENGTH(wide) != PyUnicode_GET_LENGTH(utf8)) { + Py_DECREF(wide); + Py_DECREF(utf8); + PyErr_SetString(PyExc_AssertionError, + "test_widechar: " + "wide string and utf8 string " + "have different length"); + return NULL; + } + if (PyUnicode_Compare(wide, utf8)) { + Py_DECREF(wide); + Py_DECREF(utf8); + if (PyErr_Occurred()) + return NULL; + PyErr_SetString(PyExc_AssertionError, + "test_widechar: " + "wide string and utf8 string " + "are different"); + return NULL; + } + + Py_DECREF(wide); + Py_DECREF(utf8); + +#if defined(SIZEOF_WCHAR_T) && (SIZEOF_WCHAR_T == 4) + wide = PyUnicode_FromWideChar(invalid, 1); + if (wide == NULL) + PyErr_Clear(); + else { + PyErr_SetString(PyExc_AssertionError, + "test_widechar: " + "PyUnicode_FromWideChar(L\"\\U00110000\", 1) didn't fail"); + return NULL; + } +#endif + Py_RETURN_NONE; +} + +static PyObject * +unicode_aswidechar(PyObject *self, PyObject *args) +{ + PyObject *unicode, *result; + Py_ssize_t buflen, size; + wchar_t *buffer; + + if (!PyArg_ParseTuple(args, "Un", &unicode, &buflen)) + return NULL; + buffer = PyMem_New(wchar_t, buflen); + if (buffer == NULL) + return PyErr_NoMemory(); + + size = PyUnicode_AsWideChar(unicode, buffer, buflen); + if (size == -1) { + PyMem_Free(buffer); + return NULL; + } + + if (size < buflen) + buflen = size + 1; + else + buflen = size; + result = PyUnicode_FromWideChar(buffer, buflen); + PyMem_Free(buffer); + if (result == NULL) + return NULL; + + return Py_BuildValue("(Nn)", result, size); +} + +static PyObject * +unicode_aswidecharstring(PyObject *self, PyObject *args) +{ + PyObject *unicode, *result; + Py_ssize_t size; + wchar_t *buffer; + + if (!PyArg_ParseTuple(args, "U", &unicode)) + return NULL; + + buffer = PyUnicode_AsWideCharString(unicode, &size); + if (buffer == NULL) + return NULL; + + result = PyUnicode_FromWideChar(buffer, size + 1); + PyMem_Free(buffer); + if (result == NULL) + return NULL; + return Py_BuildValue("(Nn)", result, size); +} + +static PyObject * +unicode_asucs4(PyObject *self, PyObject *args) +{ + PyObject *unicode, *result; + Py_UCS4 *buffer; + int copy_null; + Py_ssize_t str_len, buf_len; + + if (!PyArg_ParseTuple(args, "Unp:unicode_asucs4", &unicode, &str_len, ©_null)) { + return NULL; + } + + buf_len = str_len + 1; + buffer = PyMem_NEW(Py_UCS4, buf_len); + if (buffer == NULL) { + return PyErr_NoMemory(); + } + memset(buffer, 0, sizeof(Py_UCS4)*buf_len); + buffer[str_len] = 0xffffU; + + if (!PyUnicode_AsUCS4(unicode, buffer, buf_len, copy_null)) { + PyMem_Free(buffer); + return NULL; + } + + result = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND, buffer, buf_len); + PyMem_Free(buffer); + return result; +} + +static PyObject * +unicode_asutf8(PyObject *self, PyObject *args) +{ + PyObject *unicode; + const char *buffer; + + if (!PyArg_ParseTuple(args, "U", &unicode)) { + return NULL; + } + + buffer = PyUnicode_AsUTF8(unicode); + if (buffer == NULL) { + return NULL; + } + + return PyBytes_FromString(buffer); +} + +static PyObject * +unicode_asutf8andsize(PyObject *self, PyObject *args) +{ + PyObject *unicode, *result; + const char *buffer; + Py_ssize_t utf8_len; + + if(!PyArg_ParseTuple(args, "U", &unicode)) { + return NULL; + } + + buffer = PyUnicode_AsUTF8AndSize(unicode, &utf8_len); + if (buffer == NULL) { + return NULL; + } + + result = PyBytes_FromString(buffer); + if (result == NULL) { + return NULL; + } + + return Py_BuildValue("(Nn)", result, utf8_len); +} + +static PyObject * +unicode_findchar(PyObject *self, PyObject *args) +{ + PyObject *str; + int direction; + unsigned int ch; + Py_ssize_t result; + Py_ssize_t start, end; + + if (!PyArg_ParseTuple(args, "UInni:unicode_findchar", &str, &ch, + &start, &end, &direction)) { + return NULL; + } + + result = PyUnicode_FindChar(str, (Py_UCS4)ch, start, end, direction); + if (result == -2) + return NULL; + else + return PyLong_FromSsize_t(result); +} + +static PyObject * +unicode_copycharacters(PyObject *self, PyObject *args) +{ + PyObject *from, *to, *to_copy; + Py_ssize_t from_start, to_start, how_many, copied; + + if (!PyArg_ParseTuple(args, "UnOnn:unicode_copycharacters", &to, &to_start, + &from, &from_start, &how_many)) { + return NULL; + } + + if (!(to_copy = PyUnicode_New(PyUnicode_GET_LENGTH(to), + PyUnicode_MAX_CHAR_VALUE(to)))) { + return NULL; + } + if (PyUnicode_Fill(to_copy, 0, PyUnicode_GET_LENGTH(to_copy), 0U) < 0) { + Py_DECREF(to_copy); + return NULL; + } + + if ((copied = PyUnicode_CopyCharacters(to_copy, to_start, from, + from_start, how_many)) < 0) { + Py_DECREF(to_copy); + return NULL; + } + + return Py_BuildValue("(Nn)", to_copy, copied); +} + +static int +check_raised_systemerror(PyObject *result, char* msg) +{ + if (result) { + // no exception + PyErr_Format(PyExc_AssertionError, + "SystemError not raised: %s", + msg); + return 0; + } + if (PyErr_ExceptionMatches(PyExc_SystemError)) { + // expected exception + PyErr_Clear(); + return 1; + } + // unexpected exception + return 0; +} + +static PyObject * +test_string_from_format(PyObject *self, PyObject *Py_UNUSED(ignored)) +{ + PyObject *result; + PyObject *unicode = PyUnicode_FromString("None"); + +#define CHECK_FORMAT_2(FORMAT, EXPECTED, ARG1, ARG2) \ + result = PyUnicode_FromFormat(FORMAT, ARG1, ARG2); \ + if (EXPECTED == NULL) { \ + if (!check_raised_systemerror(result, FORMAT)) { \ + goto Fail; \ + } \ + } \ + else if (result == NULL) \ + return NULL; \ + else if (!_PyUnicode_EqualToASCIIString(result, EXPECTED)) { \ + PyErr_Format(PyExc_AssertionError, \ + "test_string_from_format: failed at \"%s\" " \ + "expected \"%s\" got \"%s\"", \ + FORMAT, EXPECTED, PyUnicode_AsUTF8(result)); \ + goto Fail; \ + } \ + Py_XDECREF(result) + +#define CHECK_FORMAT_1(FORMAT, EXPECTED, ARG) \ + CHECK_FORMAT_2(FORMAT, EXPECTED, ARG, 0) + +#define CHECK_FORMAT_0(FORMAT, EXPECTED) \ + CHECK_FORMAT_2(FORMAT, EXPECTED, 0, 0) + + // Unrecognized + CHECK_FORMAT_2("%u %? %u", NULL, 1, 2); + + // "%%" (options are rejected) + CHECK_FORMAT_0( "%%", "%"); + CHECK_FORMAT_0( "%0%", NULL); + CHECK_FORMAT_0("%00%", NULL); + CHECK_FORMAT_0( "%2%", NULL); + CHECK_FORMAT_0("%02%", NULL); + CHECK_FORMAT_0("%.0%", NULL); + CHECK_FORMAT_0("%.2%", NULL); + + // "%c" + CHECK_FORMAT_1( "%c", "c", 'c'); + CHECK_FORMAT_1( "%0c", "c", 'c'); + CHECK_FORMAT_1("%00c", "c", 'c'); + CHECK_FORMAT_1( "%2c", "c", 'c'); + CHECK_FORMAT_1("%02c", "c", 'c'); + CHECK_FORMAT_1("%.0c", "c", 'c'); + CHECK_FORMAT_1("%.2c", "c", 'c'); + + // Integers + CHECK_FORMAT_1("%d", "123", (int)123); + CHECK_FORMAT_1("%i", "123", (int)123); + CHECK_FORMAT_1("%u", "123", (unsigned int)123); + CHECK_FORMAT_1("%ld", "123", (long)123); + CHECK_FORMAT_1("%li", "123", (long)123); + CHECK_FORMAT_1("%lu", "123", (unsigned long)123); + CHECK_FORMAT_1("%lld", "123", (long long)123); + CHECK_FORMAT_1("%lli", "123", (long long)123); + CHECK_FORMAT_1("%llu", "123", (unsigned long long)123); + CHECK_FORMAT_1("%zd", "123", (Py_ssize_t)123); + CHECK_FORMAT_1("%zi", "123", (Py_ssize_t)123); + CHECK_FORMAT_1("%zu", "123", (size_t)123); + CHECK_FORMAT_1("%x", "7b", (int)123); + + CHECK_FORMAT_1("%d", "-123", (int)-123); + CHECK_FORMAT_1("%i", "-123", (int)-123); + CHECK_FORMAT_1("%ld", "-123", (long)-123); + CHECK_FORMAT_1("%li", "-123", (long)-123); + CHECK_FORMAT_1("%lld", "-123", (long long)-123); + CHECK_FORMAT_1("%lli", "-123", (long long)-123); + CHECK_FORMAT_1("%zd", "-123", (Py_ssize_t)-123); + CHECK_FORMAT_1("%zi", "-123", (Py_ssize_t)-123); + CHECK_FORMAT_1("%x", "ffffff85", (int)-123); + + // Integers: width < length + CHECK_FORMAT_1("%1d", "123", (int)123); + CHECK_FORMAT_1("%1i", "123", (int)123); + CHECK_FORMAT_1("%1u", "123", (unsigned int)123); + CHECK_FORMAT_1("%1ld", "123", (long)123); + CHECK_FORMAT_1("%1li", "123", (long)123); + CHECK_FORMAT_1("%1lu", "123", (unsigned long)123); + CHECK_FORMAT_1("%1lld", "123", (long long)123); + CHECK_FORMAT_1("%1lli", "123", (long long)123); + CHECK_FORMAT_1("%1llu", "123", (unsigned long long)123); + CHECK_FORMAT_1("%1zd", "123", (Py_ssize_t)123); + CHECK_FORMAT_1("%1zi", "123", (Py_ssize_t)123); + CHECK_FORMAT_1("%1zu", "123", (size_t)123); + CHECK_FORMAT_1("%1x", "7b", (int)123); + + CHECK_FORMAT_1("%1d", "-123", (int)-123); + CHECK_FORMAT_1("%1i", "-123", (int)-123); + CHECK_FORMAT_1("%1ld", "-123", (long)-123); + CHECK_FORMAT_1("%1li", "-123", (long)-123); + CHECK_FORMAT_1("%1lld", "-123", (long long)-123); + CHECK_FORMAT_1("%1lli", "-123", (long long)-123); + CHECK_FORMAT_1("%1zd", "-123", (Py_ssize_t)-123); + CHECK_FORMAT_1("%1zi", "-123", (Py_ssize_t)-123); + CHECK_FORMAT_1("%1x", "ffffff85", (int)-123); + + // Integers: width > length + CHECK_FORMAT_1("%5d", " 123", (int)123); + CHECK_FORMAT_1("%5i", " 123", (int)123); + CHECK_FORMAT_1("%5u", " 123", (unsigned int)123); + CHECK_FORMAT_1("%5ld", " 123", (long)123); + CHECK_FORMAT_1("%5li", " 123", (long)123); + CHECK_FORMAT_1("%5lu", " 123", (unsigned long)123); + CHECK_FORMAT_1("%5lld", " 123", (long long)123); + CHECK_FORMAT_1("%5lli", " 123", (long long)123); + CHECK_FORMAT_1("%5llu", " 123", (unsigned long long)123); + CHECK_FORMAT_1("%5zd", " 123", (Py_ssize_t)123); + CHECK_FORMAT_1("%5zi", " 123", (Py_ssize_t)123); + CHECK_FORMAT_1("%5zu", " 123", (size_t)123); + CHECK_FORMAT_1("%5x", " 7b", (int)123); + + CHECK_FORMAT_1("%5d", " -123", (int)-123); + CHECK_FORMAT_1("%5i", " -123", (int)-123); + CHECK_FORMAT_1("%5ld", " -123", (long)-123); + CHECK_FORMAT_1("%5li", " -123", (long)-123); + CHECK_FORMAT_1("%5lld", " -123", (long long)-123); + CHECK_FORMAT_1("%5lli", " -123", (long long)-123); + CHECK_FORMAT_1("%5zd", " -123", (Py_ssize_t)-123); + CHECK_FORMAT_1("%5zi", " -123", (Py_ssize_t)-123); + CHECK_FORMAT_1("%9x", " ffffff85", (int)-123); + + // Integers: width > length, 0-flag + CHECK_FORMAT_1("%05d", "00123", (int)123); + CHECK_FORMAT_1("%05i", "00123", (int)123); + CHECK_FORMAT_1("%05u", "00123", (unsigned int)123); + CHECK_FORMAT_1("%05ld", "00123", (long)123); + CHECK_FORMAT_1("%05li", "00123", (long)123); + CHECK_FORMAT_1("%05lu", "00123", (unsigned long)123); + CHECK_FORMAT_1("%05lld", "00123", (long long)123); + CHECK_FORMAT_1("%05lli", "00123", (long long)123); + CHECK_FORMAT_1("%05llu", "00123", (unsigned long long)123); + CHECK_FORMAT_1("%05zd", "00123", (Py_ssize_t)123); + CHECK_FORMAT_1("%05zi", "00123", (Py_ssize_t)123); + CHECK_FORMAT_1("%05zu", "00123", (size_t)123); + CHECK_FORMAT_1("%05x", "0007b", (int)123); + + // Integers: precision < length + CHECK_FORMAT_1("%.1d", "123", (int)123); + CHECK_FORMAT_1("%.1i", "123", (int)123); + CHECK_FORMAT_1("%.1u", "123", (unsigned int)123); + CHECK_FORMAT_1("%.1ld", "123", (long)123); + CHECK_FORMAT_1("%.1li", "123", (long)123); + CHECK_FORMAT_1("%.1lu", "123", (unsigned long)123); + CHECK_FORMAT_1("%.1lld", "123", (long long)123); + CHECK_FORMAT_1("%.1lli", "123", (long long)123); + CHECK_FORMAT_1("%.1llu", "123", (unsigned long long)123); + CHECK_FORMAT_1("%.1zd", "123", (Py_ssize_t)123); + CHECK_FORMAT_1("%.1zi", "123", (Py_ssize_t)123); + CHECK_FORMAT_1("%.1zu", "123", (size_t)123); + CHECK_FORMAT_1("%.1x", "7b", (int)123); + + CHECK_FORMAT_1("%.1d", "-123", (int)-123); + CHECK_FORMAT_1("%.1i", "-123", (int)-123); + CHECK_FORMAT_1("%.1ld", "-123", (long)-123); + CHECK_FORMAT_1("%.1li", "-123", (long)-123); + CHECK_FORMAT_1("%.1lld", "-123", (long long)-123); + CHECK_FORMAT_1("%.1lli", "-123", (long long)-123); + CHECK_FORMAT_1("%.1zd", "-123", (Py_ssize_t)-123); + CHECK_FORMAT_1("%.1zi", "-123", (Py_ssize_t)-123); + CHECK_FORMAT_1("%.1x", "ffffff85", (int)-123); + + // Integers: precision > length + CHECK_FORMAT_1("%.5d", "00123", (int)123); + CHECK_FORMAT_1("%.5i", "00123", (int)123); + CHECK_FORMAT_1("%.5u", "00123", (unsigned int)123); + CHECK_FORMAT_1("%.5ld", "00123", (long)123); + CHECK_FORMAT_1("%.5li", "00123", (long)123); + CHECK_FORMAT_1("%.5lu", "00123", (unsigned long)123); + CHECK_FORMAT_1("%.5lld", "00123", (long long)123); + CHECK_FORMAT_1("%.5lli", "00123", (long long)123); + CHECK_FORMAT_1("%.5llu", "00123", (unsigned long long)123); + CHECK_FORMAT_1("%.5zd", "00123", (Py_ssize_t)123); + CHECK_FORMAT_1("%.5zi", "00123", (Py_ssize_t)123); + CHECK_FORMAT_1("%.5zu", "00123", (size_t)123); + CHECK_FORMAT_1("%.5x", "0007b", (int)123); + + // Integers: width > precision > length + CHECK_FORMAT_1("%7.5d", " 00123", (int)123); + CHECK_FORMAT_1("%7.5i", " 00123", (int)123); + CHECK_FORMAT_1("%7.5u", " 00123", (unsigned int)123); + CHECK_FORMAT_1("%7.5ld", " 00123", (long)123); + CHECK_FORMAT_1("%7.5li", " 00123", (long)123); + CHECK_FORMAT_1("%7.5lu", " 00123", (unsigned long)123); + CHECK_FORMAT_1("%7.5lld", " 00123", (long long)123); + CHECK_FORMAT_1("%7.5lli", " 00123", (long long)123); + CHECK_FORMAT_1("%7.5llu", " 00123", (unsigned long long)123); + CHECK_FORMAT_1("%7.5zd", " 00123", (Py_ssize_t)123); + CHECK_FORMAT_1("%7.5zi", " 00123", (Py_ssize_t)123); + CHECK_FORMAT_1("%7.5zu", " 00123", (size_t)123); + CHECK_FORMAT_1("%7.5x", " 0007b", (int)123); + + // Integers: width > precision > length, 0-flag + CHECK_FORMAT_1("%07.5d", "0000123", (int)123); + CHECK_FORMAT_1("%07.5i", "0000123", (int)123); + CHECK_FORMAT_1("%07.5u", "0000123", (unsigned int)123); + CHECK_FORMAT_1("%07.5ld", "0000123", (long)123); + CHECK_FORMAT_1("%07.5li", "0000123", (long)123); + CHECK_FORMAT_1("%07.5lu", "0000123", (unsigned long)123); + CHECK_FORMAT_1("%07.5lld", "0000123", (long long)123); + CHECK_FORMAT_1("%07.5lli", "0000123", (long long)123); + CHECK_FORMAT_1("%07.5llu", "0000123", (unsigned long long)123); + CHECK_FORMAT_1("%07.5zd", "0000123", (Py_ssize_t)123); + CHECK_FORMAT_1("%07.5zi", "0000123", (Py_ssize_t)123); + CHECK_FORMAT_1("%07.5zu", "0000123", (size_t)123); + CHECK_FORMAT_1("%07.5x", "000007b", (int)123); + + // Integers: precision > width > length + CHECK_FORMAT_1("%5.7d", "0000123", (int)123); + CHECK_FORMAT_1("%5.7i", "0000123", (int)123); + CHECK_FORMAT_1("%5.7u", "0000123", (unsigned int)123); + CHECK_FORMAT_1("%5.7ld", "0000123", (long)123); + CHECK_FORMAT_1("%5.7li", "0000123", (long)123); + CHECK_FORMAT_1("%5.7lu", "0000123", (unsigned long)123); + CHECK_FORMAT_1("%5.7lld", "0000123", (long long)123); + CHECK_FORMAT_1("%5.7lli", "0000123", (long long)123); + CHECK_FORMAT_1("%5.7llu", "0000123", (unsigned long long)123); + CHECK_FORMAT_1("%5.7zd", "0000123", (Py_ssize_t)123); + CHECK_FORMAT_1("%5.7zi", "0000123", (Py_ssize_t)123); + CHECK_FORMAT_1("%5.7zu", "0000123", (size_t)123); + CHECK_FORMAT_1("%5.7x", "000007b", (int)123); + + // Integers: precision > width > length, 0-flag + CHECK_FORMAT_1("%05.7d", "0000123", (int)123); + CHECK_FORMAT_1("%05.7i", "0000123", (int)123); + CHECK_FORMAT_1("%05.7u", "0000123", (unsigned int)123); + CHECK_FORMAT_1("%05.7ld", "0000123", (long)123); + CHECK_FORMAT_1("%05.7li", "0000123", (long)123); + CHECK_FORMAT_1("%05.7lu", "0000123", (unsigned long)123); + CHECK_FORMAT_1("%05.7lld", "0000123", (long long)123); + CHECK_FORMAT_1("%05.7lli", "0000123", (long long)123); + CHECK_FORMAT_1("%05.7llu", "0000123", (unsigned long long)123); + CHECK_FORMAT_1("%05.7zd", "0000123", (Py_ssize_t)123); + CHECK_FORMAT_1("%05.7zi", "0000123", (Py_ssize_t)123); + CHECK_FORMAT_1("%05.7zu", "0000123", (size_t)123); + CHECK_FORMAT_1("%05.7x", "000007b", (int)123); + + // Integers: precision = 0, arg = 0 (empty string in C) + CHECK_FORMAT_1("%.0d", "0", (int)0); + CHECK_FORMAT_1("%.0i", "0", (int)0); + CHECK_FORMAT_1("%.0u", "0", (unsigned int)0); + CHECK_FORMAT_1("%.0ld", "0", (long)0); + CHECK_FORMAT_1("%.0li", "0", (long)0); + CHECK_FORMAT_1("%.0lu", "0", (unsigned long)0); + CHECK_FORMAT_1("%.0lld", "0", (long long)0); + CHECK_FORMAT_1("%.0lli", "0", (long long)0); + CHECK_FORMAT_1("%.0llu", "0", (unsigned long long)0); + CHECK_FORMAT_1("%.0zd", "0", (Py_ssize_t)0); + CHECK_FORMAT_1("%.0zi", "0", (Py_ssize_t)0); + CHECK_FORMAT_1("%.0zu", "0", (size_t)0); + CHECK_FORMAT_1("%.0x", "0", (int)0); + + // Strings + CHECK_FORMAT_1("%s", "None", "None"); + CHECK_FORMAT_1("%U", "None", unicode); + CHECK_FORMAT_1("%A", "None", Py_None); + CHECK_FORMAT_1("%S", "None", Py_None); + CHECK_FORMAT_1("%R", "None", Py_None); + CHECK_FORMAT_2("%V", "None", unicode, "ignored"); + CHECK_FORMAT_2("%V", "None", NULL, "None"); + + // Strings: width < length + CHECK_FORMAT_1("%1s", "None", "None"); + CHECK_FORMAT_1("%1U", "None", unicode); + CHECK_FORMAT_1("%1A", "None", Py_None); + CHECK_FORMAT_1("%1S", "None", Py_None); + CHECK_FORMAT_1("%1R", "None", Py_None); + CHECK_FORMAT_2("%1V", "None", unicode, "ignored"); + CHECK_FORMAT_2("%1V", "None", NULL, "None"); + + // Strings: width > length + CHECK_FORMAT_1("%5s", " None", "None"); + CHECK_FORMAT_1("%5U", " None", unicode); + CHECK_FORMAT_1("%5A", " None", Py_None); + CHECK_FORMAT_1("%5S", " None", Py_None); + CHECK_FORMAT_1("%5R", " None", Py_None); + CHECK_FORMAT_2("%5V", " None", unicode, "ignored"); + CHECK_FORMAT_2("%5V", " None", NULL, "None"); + + // Strings: precision < length + CHECK_FORMAT_1("%.1s", "N", "None"); + CHECK_FORMAT_1("%.1U", "N", unicode); + CHECK_FORMAT_1("%.1A", "N", Py_None); + CHECK_FORMAT_1("%.1S", "N", Py_None); + CHECK_FORMAT_1("%.1R", "N", Py_None); + CHECK_FORMAT_2("%.1V", "N", unicode, "ignored"); + CHECK_FORMAT_2("%.1V", "N", NULL, "None"); + + // Strings: precision > length + CHECK_FORMAT_1("%.5s", "None", "None"); + CHECK_FORMAT_1("%.5U", "None", unicode); + CHECK_FORMAT_1("%.5A", "None", Py_None); + CHECK_FORMAT_1("%.5S", "None", Py_None); + CHECK_FORMAT_1("%.5R", "None", Py_None); + CHECK_FORMAT_2("%.5V", "None", unicode, "ignored"); + CHECK_FORMAT_2("%.5V", "None", NULL, "None"); + + // Strings: precision < length, width > length + CHECK_FORMAT_1("%5.1s", " N", "None"); + CHECK_FORMAT_1("%5.1U", " N", unicode); + CHECK_FORMAT_1("%5.1A", " N", Py_None); + CHECK_FORMAT_1("%5.1S", " N", Py_None); + CHECK_FORMAT_1("%5.1R", " N", Py_None); + CHECK_FORMAT_2("%5.1V", " N", unicode, "ignored"); + CHECK_FORMAT_2("%5.1V", " N", NULL, "None"); + + // Strings: width < length, precision > length + CHECK_FORMAT_1("%1.5s", "None", "None"); + CHECK_FORMAT_1("%1.5U", "None", unicode); + CHECK_FORMAT_1("%1.5A", "None", Py_None); + CHECK_FORMAT_1("%1.5S", "None", Py_None); + CHECK_FORMAT_1("%1.5R", "None", Py_None); + CHECK_FORMAT_2("%1.5V", "None", unicode, "ignored"); + CHECK_FORMAT_2("%1.5V", "None", NULL, "None"); + + Py_XDECREF(unicode); + Py_RETURN_NONE; + + Fail: + Py_XDECREF(result); + Py_XDECREF(unicode); + return NULL; + +#undef CHECK_FORMAT_2 +#undef CHECK_FORMAT_1 +#undef CHECK_FORMAT_0 +} + +static PyMethodDef TestMethods[] = { + {"codec_incrementalencoder", codec_incrementalencoder, METH_VARARGS}, + {"codec_incrementaldecoder", codec_incrementaldecoder, METH_VARARGS}, + {"test_unicode_compare_with_ascii", + test_unicode_compare_with_ascii, METH_NOARGS}, + {"test_string_from_format", test_string_from_format, METH_NOARGS}, + {"test_widechar", test_widechar, METH_NOARGS}, + {"unicode_aswidechar", unicode_aswidechar, METH_VARARGS}, + {"unicode_aswidecharstring", unicode_aswidecharstring, METH_VARARGS}, + {"unicode_asucs4", unicode_asucs4, METH_VARARGS}, + {"unicode_asutf8", unicode_asutf8, METH_VARARGS}, + {"unicode_asutf8andsize", unicode_asutf8andsize, METH_VARARGS}, + {"unicode_findchar", unicode_findchar, METH_VARARGS}, + {"unicode_copycharacters", unicode_copycharacters, METH_VARARGS}, + {NULL}, +}; + +int +_PyTestCapi_Init_Unicode(PyObject *m) { + _testcapimodule = PyModule_GetDef(m); + + if (PyModule_AddFunctions(m, TestMethods) < 0) { + return -1; + } + + return 0; +} diff --git a/Modules/_testcapi/vectorcall.c b/Modules/_testcapi/vectorcall.c index 9bc702905caf76..626706eafab5e8 100644 --- a/Modules/_testcapi/vectorcall.c +++ b/Modules/_testcapi/vectorcall.c @@ -1,5 +1,8 @@ #include "parts.h" -#include // offsetof +#include "clinic/vectorcall.c.h" + +#include "structmember.h" // PyMemberDef +#include // offsetof /* Test PEP 590 - Vectorcall */ @@ -122,11 +125,128 @@ test_pyvectorcall_call(PyObject *self, PyObject *args) return PyVectorcall_Call(func, argstuple, kwargs); } +PyObject * +VectorCallClass_tpcall(PyObject *self, PyObject *args, PyObject *kwargs) { + return PyUnicode_FromString("tp_call"); +} + +PyObject * +VectorCallClass_vectorcall(PyObject *callable, + PyObject *const *args, + size_t nargsf, + PyObject *kwnames) { + return PyUnicode_FromString("vectorcall"); +} + +/*[clinic input] +module _testcapi +class _testcapi.VectorCallClass "PyObject *" "&PyType_Type" +[clinic start generated code]*/ +/*[clinic end generated code: output=da39a3ee5e6b4b0d input=8423a8e919f2f0df]*/ + +/*[clinic input] +_testcapi.VectorCallClass.set_vectorcall + + type: object(subclass_of="&PyType_Type", type="PyTypeObject *") + / + +Set self's vectorcall function for `type` to one that returns "vectorcall" +[clinic start generated code]*/ + +static PyObject * +_testcapi_VectorCallClass_set_vectorcall_impl(PyObject *self, + PyTypeObject *type) +/*[clinic end generated code: output=b37f0466f15da903 input=840de66182c7d71a]*/ +{ + if (!PyObject_TypeCheck(self, type)) { + return PyErr_Format( + PyExc_TypeError, + "expected %s instance", + PyType_GetName(type)); + } + if (!type->tp_vectorcall_offset) { + return PyErr_Format( + PyExc_TypeError, + "type %s has no vectorcall offset", + PyType_GetName(type)); + } + *(vectorcallfunc*)((char*)self + type->tp_vectorcall_offset) = ( + VectorCallClass_vectorcall); + Py_RETURN_NONE; +} + +PyMethodDef VectorCallClass_methods[] = { + _TESTCAPI_VECTORCALLCLASS_SET_VECTORCALL_METHODDEF + {NULL, NULL} +}; + +PyMemberDef VectorCallClass_members[] = { + {"__vectorcalloffset__", T_PYSSIZET, 0/* set later */, READONLY}, + {NULL} +}; + +PyType_Slot VectorCallClass_slots[] = { + {Py_tp_call, VectorCallClass_tpcall}, + {Py_tp_members, VectorCallClass_members}, + {Py_tp_methods, VectorCallClass_methods}, + {0}, +}; + +/*[clinic input] +_testcapi.make_vectorcall_class + + base: object(subclass_of="&PyType_Type", type="PyTypeObject *") = NULL + / + +Create a class whose instances return "tpcall" when called. + +When the "set_vectorcall" method is called on an instance, a vectorcall +function that returns "vectorcall" will be installed. +[clinic start generated code]*/ + +static PyObject * +_testcapi_make_vectorcall_class_impl(PyObject *module, PyTypeObject *base) +/*[clinic end generated code: output=16dcfc3062ddf968 input=f72e01ccf52de2b4]*/ +{ + if (!base) { + base = (PyTypeObject *)&PyBaseObject_Type; + } + VectorCallClass_members[0].offset = base->tp_basicsize; + PyType_Spec spec = { + .name = "_testcapi.VectorcallClass", + .basicsize = (int)(base->tp_basicsize + sizeof(vectorcallfunc)), + .flags = Py_TPFLAGS_DEFAULT + | Py_TPFLAGS_HAVE_VECTORCALL + | Py_TPFLAGS_BASETYPE, + .slots = VectorCallClass_slots, + }; + + return PyType_FromSpecWithBases(&spec, (PyObject *)base); +} + +/*[clinic input] +_testcapi.has_vectorcall_flag -> bool + + type: object(subclass_of="&PyType_Type", type="PyTypeObject *") + / + +Return true iff Py_TPFLAGS_HAVE_VECTORCALL is set on the class. +[clinic start generated code]*/ + +static int +_testcapi_has_vectorcall_flag_impl(PyObject *module, PyTypeObject *type) +/*[clinic end generated code: output=3ae8d1374388c671 input=8eee492ac548749e]*/ +{ + return PyType_HasFeature(type, Py_TPFLAGS_HAVE_VECTORCALL); +} + static PyMethodDef TestMethods[] = { {"pyobject_fastcall", test_pyobject_fastcall, METH_VARARGS}, {"pyobject_fastcalldict", test_pyobject_fastcalldict, METH_VARARGS}, {"pyobject_vectorcall", test_pyobject_vectorcall, METH_VARARGS}, {"pyvectorcall_call", test_pyvectorcall_call, METH_VARARGS}, + _TESTCAPI_MAKE_VECTORCALL_CLASS_METHODDEF + _TESTCAPI_HAS_VECTORCALL_FLAG_METHODDEF {NULL}, }; diff --git a/Modules/_testcapi/vectorcall_limited.c b/Modules/_testcapi/vectorcall_limited.c new file mode 100644 index 00000000000000..c5184318e292e6 --- /dev/null +++ b/Modules/_testcapi/vectorcall_limited.c @@ -0,0 +1,92 @@ +#include "pyconfig.h" // Py_TRACE_REFS + +#ifdef Py_TRACE_REFS + +// Py_TRACE_REFS is incompatible with Limited API +#include "parts.h" +int +_PyTestCapi_Init_VectorcallLimited(PyObject *m) { + return 0; +} + +#else + +#define Py_LIMITED_API 0x030c0000 // 3.12 +#include "parts.h" +#include "structmember.h" // PyMemberDef + +/* Test Vectorcall in the limited API */ + +static PyObject * +LimitedVectorCallClass_tpcall(PyObject *self, PyObject *args, PyObject *kwargs) { + return PyUnicode_FromString("tp_call called"); +} + +static PyObject * +LimitedVectorCallClass_vectorcall(PyObject *callable, + PyObject *const *args, + size_t nargsf, + PyObject *kwnames) { + return PyUnicode_FromString("vectorcall called"); +} + +static PyObject * +LimitedVectorCallClass_new(PyTypeObject *tp, PyTypeObject *a, PyTypeObject *kw) +{ + PyObject *self = ((allocfunc)PyType_GetSlot(tp, Py_tp_alloc))(tp, 0); + if (!self) { + return NULL; + } + *(vectorcallfunc*)((char*)self + sizeof(PyObject)) = ( + LimitedVectorCallClass_vectorcall); + return self; +} + +static PyMemberDef LimitedVectorCallClass_members[] = { + {"__vectorcalloffset__", T_PYSSIZET, sizeof(PyObject), READONLY}, + {NULL} +}; + +static PyType_Slot LimitedVectorallClass_slots[] = { + {Py_tp_new, LimitedVectorCallClass_new}, + {Py_tp_call, LimitedVectorCallClass_tpcall}, + {Py_tp_members, LimitedVectorCallClass_members}, + {0}, +}; + +static PyType_Spec LimitedVectorCallClass_spec = { + .name = "_testcapi.LimitedVectorCallClass", + .basicsize = (int)(sizeof(PyObject) + sizeof(vectorcallfunc)), + .flags = Py_TPFLAGS_DEFAULT + | Py_TPFLAGS_HAVE_VECTORCALL + | Py_TPFLAGS_BASETYPE, + .slots = LimitedVectorallClass_slots, +}; + +static PyMethodDef TestMethods[] = { + /* Add module methods here. + * (Empty list left here as template/example, since using + * PyModule_AddFunctions isn't very common.) + */ + {NULL}, +}; + +int +_PyTestCapi_Init_VectorcallLimited(PyObject *m) { + if (PyModule_AddFunctions(m, TestMethods) < 0) { + return -1; + } + + PyObject *LimitedVectorCallClass = PyType_FromModuleAndSpec( + m, &LimitedVectorCallClass_spec, NULL); + if (!LimitedVectorCallClass) { + return -1; + } + if (PyModule_AddType(m, (PyTypeObject *)LimitedVectorCallClass) < 0) { + return -1; + } + + return 0; +} + +#endif // Py_TRACE_REFS diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index 517591465b4914..91bdeb8b6464df 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -1886,234 +1886,6 @@ parse_tuple_and_keywords(PyObject *self, PyObject *args) return return_value; } -static PyObject * -test_widechar(PyObject *self, PyObject *Py_UNUSED(ignored)) -{ -#if defined(SIZEOF_WCHAR_T) && (SIZEOF_WCHAR_T == 4) - const wchar_t wtext[2] = {(wchar_t)0x10ABCDu}; - size_t wtextlen = 1; - const wchar_t invalid[1] = {(wchar_t)0x110000u}; -#else - const wchar_t wtext[3] = {(wchar_t)0xDBEAu, (wchar_t)0xDFCDu}; - size_t wtextlen = 2; -#endif - PyObject *wide, *utf8; - - wide = PyUnicode_FromWideChar(wtext, wtextlen); - if (wide == NULL) - return NULL; - - utf8 = PyUnicode_FromString("\xf4\x8a\xaf\x8d"); - if (utf8 == NULL) { - Py_DECREF(wide); - return NULL; - } - - if (PyUnicode_GET_LENGTH(wide) != PyUnicode_GET_LENGTH(utf8)) { - Py_DECREF(wide); - Py_DECREF(utf8); - return raiseTestError("test_widechar", - "wide string and utf8 string " - "have different length"); - } - if (PyUnicode_Compare(wide, utf8)) { - Py_DECREF(wide); - Py_DECREF(utf8); - if (PyErr_Occurred()) - return NULL; - return raiseTestError("test_widechar", - "wide string and utf8 string " - "are different"); - } - - Py_DECREF(wide); - Py_DECREF(utf8); - -#if defined(SIZEOF_WCHAR_T) && (SIZEOF_WCHAR_T == 4) - wide = PyUnicode_FromWideChar(invalid, 1); - if (wide == NULL) - PyErr_Clear(); - else - return raiseTestError("test_widechar", - "PyUnicode_FromWideChar(L\"\\U00110000\", 1) didn't fail"); -#endif - Py_RETURN_NONE; -} - -static PyObject * -unicode_aswidechar(PyObject *self, PyObject *args) -{ - PyObject *unicode, *result; - Py_ssize_t buflen, size; - wchar_t *buffer; - - if (!PyArg_ParseTuple(args, "Un", &unicode, &buflen)) - return NULL; - buffer = PyMem_New(wchar_t, buflen); - if (buffer == NULL) - return PyErr_NoMemory(); - - size = PyUnicode_AsWideChar(unicode, buffer, buflen); - if (size == -1) { - PyMem_Free(buffer); - return NULL; - } - - if (size < buflen) - buflen = size + 1; - else - buflen = size; - result = PyUnicode_FromWideChar(buffer, buflen); - PyMem_Free(buffer); - if (result == NULL) - return NULL; - - return Py_BuildValue("(Nn)", result, size); -} - -static PyObject * -unicode_aswidecharstring(PyObject *self, PyObject *args) -{ - PyObject *unicode, *result; - Py_ssize_t size; - wchar_t *buffer; - - if (!PyArg_ParseTuple(args, "U", &unicode)) - return NULL; - - buffer = PyUnicode_AsWideCharString(unicode, &size); - if (buffer == NULL) - return NULL; - - result = PyUnicode_FromWideChar(buffer, size + 1); - PyMem_Free(buffer); - if (result == NULL) - return NULL; - return Py_BuildValue("(Nn)", result, size); -} - -static PyObject * -unicode_asucs4(PyObject *self, PyObject *args) -{ - PyObject *unicode, *result; - Py_UCS4 *buffer; - int copy_null; - Py_ssize_t str_len, buf_len; - - if (!PyArg_ParseTuple(args, "Unp:unicode_asucs4", &unicode, &str_len, ©_null)) { - return NULL; - } - - buf_len = str_len + 1; - buffer = PyMem_NEW(Py_UCS4, buf_len); - if (buffer == NULL) { - return PyErr_NoMemory(); - } - memset(buffer, 0, sizeof(Py_UCS4)*buf_len); - buffer[str_len] = 0xffffU; - - if (!PyUnicode_AsUCS4(unicode, buffer, buf_len, copy_null)) { - PyMem_Free(buffer); - return NULL; - } - - result = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND, buffer, buf_len); - PyMem_Free(buffer); - return result; -} - -static PyObject * -unicode_asutf8(PyObject *self, PyObject *args) -{ - PyObject *unicode; - const char *buffer; - - if (!PyArg_ParseTuple(args, "U", &unicode)) { - return NULL; - } - - buffer = PyUnicode_AsUTF8(unicode); - if (buffer == NULL) { - return NULL; - } - - return PyBytes_FromString(buffer); -} - -static PyObject * -unicode_asutf8andsize(PyObject *self, PyObject *args) -{ - PyObject *unicode, *result; - const char *buffer; - Py_ssize_t utf8_len; - - if(!PyArg_ParseTuple(args, "U", &unicode)) { - return NULL; - } - - buffer = PyUnicode_AsUTF8AndSize(unicode, &utf8_len); - if (buffer == NULL) { - return NULL; - } - - result = PyBytes_FromString(buffer); - if (result == NULL) { - return NULL; - } - - return Py_BuildValue("(Nn)", result, utf8_len); -} - -static PyObject * -unicode_findchar(PyObject *self, PyObject *args) -{ - PyObject *str; - int direction; - unsigned int ch; - Py_ssize_t result; - Py_ssize_t start, end; - - if (!PyArg_ParseTuple(args, "UInni:unicode_findchar", &str, &ch, - &start, &end, &direction)) { - return NULL; - } - - result = PyUnicode_FindChar(str, (Py_UCS4)ch, start, end, direction); - if (result == -2) - return NULL; - else - return PyLong_FromSsize_t(result); -} - -static PyObject * -unicode_copycharacters(PyObject *self, PyObject *args) -{ - PyObject *from, *to, *to_copy; - Py_ssize_t from_start, to_start, how_many, copied; - - if (!PyArg_ParseTuple(args, "UnOnn:unicode_copycharacters", &to, &to_start, - &from, &from_start, &how_many)) { - return NULL; - } - - if (!(to_copy = PyUnicode_New(PyUnicode_GET_LENGTH(to), - PyUnicode_MAX_CHAR_VALUE(to)))) { - return NULL; - } - if (PyUnicode_Fill(to_copy, 0, PyUnicode_GET_LENGTH(to_copy), 0U) < 0) { - Py_DECREF(to_copy); - return NULL; - } - - if ((copied = PyUnicode_CopyCharacters(to_copy, to_start, from, - from_start, how_many)) < 0) { - Py_DECREF(to_copy); - return NULL; - } - - return Py_BuildValue("(Nn)", to_copy, copied); -} - static PyObject * getargs_w_star(PyObject *self, PyObject *args) { @@ -2164,27 +1936,6 @@ test_empty_argparse(PyObject *self, PyObject *Py_UNUSED(ignored)) } } -static PyObject * -codec_incrementalencoder(PyObject *self, PyObject *args) -{ - const char *encoding, *errors = NULL; - if (!PyArg_ParseTuple(args, "s|s:test_incrementalencoder", - &encoding, &errors)) - return NULL; - return PyCodec_IncrementalEncoder(encoding, errors); -} - -static PyObject * -codec_incrementaldecoder(PyObject *self, PyObject *args) -{ - const char *encoding, *errors = NULL; - if (!PyArg_ParseTuple(args, "s|s:test_incrementaldecoder", - &encoding, &errors)) - return NULL; - return PyCodec_IncrementalDecoder(encoding, errors); -} - - /* Simple test of _PyLong_NumBits and _PyLong_Sign. */ static PyObject * test_long_numbits(PyObject *self, PyObject *Py_UNUSED(ignored)) @@ -2847,63 +2598,6 @@ pending_threadfunc(PyObject *self, PyObject *arg) Py_RETURN_TRUE; } -/* Some tests of PyUnicode_FromFormat(). This needs more tests. */ -static PyObject * -test_string_from_format(PyObject *self, PyObject *Py_UNUSED(ignored)) -{ - PyObject *result; - char *msg; - -#define CHECK_1_FORMAT(FORMAT, TYPE) \ - result = PyUnicode_FromFormat(FORMAT, (TYPE)1); \ - if (result == NULL) \ - return NULL; \ - if (!_PyUnicode_EqualToASCIIString(result, "1")) { \ - msg = FORMAT " failed at 1"; \ - goto Fail; \ - } \ - Py_DECREF(result) - - CHECK_1_FORMAT("%d", int); - CHECK_1_FORMAT("%ld", long); - /* The z width modifier was added in Python 2.5. */ - CHECK_1_FORMAT("%zd", Py_ssize_t); - - /* The u type code was added in Python 2.5. */ - CHECK_1_FORMAT("%u", unsigned int); - CHECK_1_FORMAT("%lu", unsigned long); - CHECK_1_FORMAT("%zu", size_t); - - /* "%lld" and "%llu" support added in Python 2.7. */ - CHECK_1_FORMAT("%llu", unsigned long long); - CHECK_1_FORMAT("%lld", long long); - - Py_RETURN_NONE; - - Fail: - Py_XDECREF(result); - return raiseTestError("test_string_from_format", msg); - -#undef CHECK_1_FORMAT -} - - -static PyObject * -test_unicode_compare_with_ascii(PyObject *self, PyObject *Py_UNUSED(ignored)) { - PyObject *py_s = PyUnicode_FromStringAndSize("str\0", 4); - int result; - if (py_s == NULL) - return NULL; - result = PyUnicode_CompareWithASCIIString(py_s, "str"); - Py_DECREF(py_s); - if (!result) { - PyErr_SetString(TestError, "Python string ending in NULL " - "should not compare equal to c string."); - return NULL; - } - Py_RETURN_NONE; -} - /* This is here to provide a docstring for test_descr. */ static PyObject * test_with_docstring(PyObject *self, PyObject *Py_UNUSED(ignored)) @@ -5823,12 +5517,9 @@ static PyMethodDef TestMethods[] = { {"pyobject_repr_from_null", pyobject_repr_from_null, METH_NOARGS}, {"pyobject_str_from_null", pyobject_str_from_null, METH_NOARGS}, {"pyobject_bytes_from_null", pyobject_bytes_from_null, METH_NOARGS}, - {"test_string_from_format", (PyCFunction)test_string_from_format, METH_NOARGS}, {"test_with_docstring", test_with_docstring, METH_NOARGS, PyDoc_STR("This is a pretty normal docstring.")}, {"test_string_to_double", test_string_to_double, METH_NOARGS}, - {"test_unicode_compare_with_ascii", test_unicode_compare_with_ascii, - METH_NOARGS}, {"test_capsule", (PyCFunction)test_capsule, METH_NOARGS}, {"test_from_contiguous", (PyCFunction)test_from_contiguous, METH_NOARGS}, #if (defined(__linux__) || defined(__FreeBSD__)) && defined(__GNUC__) @@ -5897,19 +5588,7 @@ static PyMethodDef TestMethods[] = { {"getargs_et", getargs_et, METH_VARARGS}, {"getargs_es_hash", getargs_es_hash, METH_VARARGS}, {"getargs_et_hash", getargs_et_hash, METH_VARARGS}, - {"codec_incrementalencoder", - (PyCFunction)codec_incrementalencoder, METH_VARARGS}, - {"codec_incrementaldecoder", - (PyCFunction)codec_incrementaldecoder, METH_VARARGS}, {"test_s_code", test_s_code, METH_NOARGS}, - {"test_widechar", test_widechar, METH_NOARGS}, - {"unicode_aswidechar", unicode_aswidechar, METH_VARARGS}, - {"unicode_aswidecharstring",unicode_aswidecharstring, METH_VARARGS}, - {"unicode_asucs4", unicode_asucs4, METH_VARARGS}, - {"unicode_asutf8", unicode_asutf8, METH_VARARGS}, - {"unicode_asutf8andsize", unicode_asutf8andsize, METH_VARARGS}, - {"unicode_findchar", unicode_findchar, METH_VARARGS}, - {"unicode_copycharacters", unicode_copycharacters, METH_VARARGS}, {"_test_thread_state", test_thread_state, METH_VARARGS}, {"_pending_threadfunc", pending_threadfunc, METH_VARARGS}, #ifdef HAVE_GETTIMEOFDAY @@ -6865,9 +6544,15 @@ PyInit__testcapi(void) if (_PyTestCapi_Init_Vectorcall(m) < 0) { return NULL; } + if (_PyTestCapi_Init_VectorcallLimited(m) < 0) { + return NULL; + } if (_PyTestCapi_Init_Heaptype(m) < 0) { return NULL; } + if (_PyTestCapi_Init_Unicode(m) < 0) { + return NULL; + } PyState_AddModule(m, &_testcapimodule); return m; diff --git a/Modules/gc_weakref.txt b/Modules/gc_weakref.txt index 6d07cce1236431..f53fb99dd6cdcb 100644 --- a/Modules/gc_weakref.txt +++ b/Modules/gc_weakref.txt @@ -47,7 +47,7 @@ soon as we execute Python code, threads other than the gc thread can run too, and they can do ordinary things with weakrefs that end up resurrecting CT while gc is running. - https://www.python.org/sf/1055820 + https://bugs.python.org/issue1055820 shows how innocent it can be, and also how nasty. Variants of the three focused test cases attached to that bug report are now part of Python's diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c index dcd46feff0cc48..97cb6e6e1efb1f 100644 --- a/Modules/gcmodule.c +++ b/Modules/gcmodule.c @@ -794,9 +794,12 @@ handle_weakrefs(PyGC_Head *unreachable, PyGC_Head *old) if (! _PyType_SUPPORTS_WEAKREFS(Py_TYPE(op))) continue; - /* It supports weakrefs. Does it have any? */ - wrlist = (PyWeakReference **) - _PyObject_GET_WEAKREFS_LISTPTR(op); + /* It supports weakrefs. Does it have any? + * + * This is never triggered for static types so we can avoid the + * (slightly) more costly _PyObject_GET_WEAKREFS_LISTPTR(). + */ + wrlist = _PyObject_GET_WEAKREFS_LISTPTR_FROM_OFFSET(op); /* `op` may have some weakrefs. March over the list, clear * all the weakrefs, and move the weakrefs with callbacks diff --git a/Modules/getpath.py b/Modules/getpath.py index dceeed7702c0be..a50313aea78b4f 100644 --- a/Modules/getpath.py +++ b/Modules/getpath.py @@ -422,7 +422,7 @@ def search_up(prefix, *landmarks, test=isfile): # ****************************************************************************** # The contents of an optional ._pth file are used to totally override -# sys.path calcualation. Its presence also implies isolated mode and +# sys.path calculation. Its presence also implies isolated mode and # no-site (unless explicitly requested) pth = None pth_dir = None diff --git a/Objects/call.c b/Objects/call.c index ed168c9c4796e2..c2509db2a9a263 100644 --- a/Objects/call.c +++ b/Objects/call.c @@ -1047,3 +1047,11 @@ _PyStack_UnpackDict_Free(PyObject *const *stack, Py_ssize_t nargs, PyMem_Free((PyObject **)stack - 1); Py_DECREF(kwnames); } + +// Export for the stable ABI +#undef PyVectorcall_NARGS +Py_ssize_t +PyVectorcall_NARGS(size_t n) +{ + return _PyVectorcall_NARGS(n); +} diff --git a/Objects/object.c b/Objects/object.c index f0c0434fab3d1e..a90c6faf99db48 100644 --- a/Objects/object.c +++ b/Objects/object.c @@ -1079,7 +1079,11 @@ _PyObject_ComputedDictPointer(PyObject *obj) /* Helper to get a pointer to an object's __dict__ slot, if any. * Creates the dict from inline attributes if necessary. - * Does not set an exception. */ + * Does not set an exception. + * + * Note that the tp_dictoffset docs used to recommend this function, + * so it should be treated as part of the public API. + */ PyObject ** _PyObject_GetDictPtr(PyObject *obj) { diff --git a/Objects/structseq.c b/Objects/structseq.c index 24cd0e705969b2..9a7013372e688c 100644 --- a/Objects/structseq.c +++ b/Objects/structseq.c @@ -580,7 +580,7 @@ _PyStructSequence_FiniType(PyTypeObject *type) assert(type->tp_base == &PyTuple_Type); // Cannot delete a type if it still has subclasses - if (type->tp_subclasses != NULL) { + if (_PyType_HasSubclasses(type)) { return; } diff --git a/Objects/typeobject.c b/Objects/typeobject.c index 1980fcbf9323fd..da02e86f94ed2c 100644 --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -73,7 +73,7 @@ static inline PyTypeObject * subclass_from_ref(PyObject *ref); static inline int static_builtin_index_is_set(PyTypeObject *self) { - return self->tp_static_builtin_index > 0; + return self->tp_subclasses != NULL; } static inline size_t @@ -81,7 +81,7 @@ static_builtin_index_get(PyTypeObject *self) { assert(static_builtin_index_is_set(self)); /* We store a 1-based index so 0 can mean "not initialized". */ - return self->tp_static_builtin_index - 1; + return (size_t)self->tp_subclasses - 1; } static inline void @@ -89,13 +89,13 @@ static_builtin_index_set(PyTypeObject *self, size_t index) { assert(index < _Py_MAX_STATIC_BUILTIN_TYPES); /* We store a 1-based index so 0 can mean "not initialized". */ - self->tp_static_builtin_index = index + 1; + self->tp_subclasses = (PyObject *)(index + 1); } static inline void static_builtin_index_clear(PyTypeObject *self) { - self->tp_static_builtin_index = 0; + self->tp_subclasses = NULL; } static inline static_builtin_state * @@ -127,6 +127,7 @@ static_builtin_state_init(PyTypeObject *self) static_builtin_state *state = static_builtin_state_get(interp, self); state->type = self; + /* state->tp_subclasses is left NULL until init_subclasses() sets it. */ /* state->tp_weaklist is left NULL until insert_head() or insert_after() (in weakrefobject.c) sets it. */ } @@ -373,6 +374,8 @@ _PyTypes_Fini(PyInterpreterState *interp) } +static PyObject * lookup_subclasses(PyTypeObject *); + void PyType_Modified(PyTypeObject *type) { @@ -395,7 +398,7 @@ PyType_Modified(PyTypeObject *type) return; } - PyObject *subclasses = type->tp_subclasses; + PyObject *subclasses = lookup_subclasses(type); if (subclasses != NULL) { assert(PyDict_CheckExact(subclasses)); @@ -783,7 +786,7 @@ mro_hierarchy(PyTypeObject *type, PyObject *temp) Py_XDECREF(old_mro); // Avoid creating an empty list if there is no subclass - if (type->tp_subclasses != NULL) { + if (_PyType_HasSubclasses(type)) { /* Obtain a copy of subclasses list to iterate over. Otherwise type->tp_subclasses might be altered @@ -1505,11 +1508,15 @@ subtype_dealloc(PyObject *self) finalizers since they might rely on part of the object being finalized that has already been destroyed. */ if (type->tp_weaklistoffset && !base->tp_weaklistoffset) { - /* Modeled after GET_WEAKREFS_LISTPTR() */ - PyWeakReference **list = (PyWeakReference **) \ - _PyObject_GET_WEAKREFS_LISTPTR(self); - while (*list) + /* Modeled after GET_WEAKREFS_LISTPTR(). + + This is never triggered for static types so we can avoid the + (slightly) more costly _PyObject_GET_WEAKREFS_LISTPTR(). */ + PyWeakReference **list = \ + _PyObject_GET_WEAKREFS_LISTPTR_FROM_OFFSET(self); + while (*list) { _PyWeakref_ClearRef(*list); + } } } @@ -3676,6 +3683,32 @@ PyType_FromMetaclass(PyTypeObject *metaclass, PyObject *module, goto finally; } + /* If this is an immutable type, check if all bases are also immutable, + * and (for now) fire a deprecation warning if not. + * (This isn't necessary for static types: those can't have heap bases, + * and only heap types can be mutable.) + */ + if (spec->flags & Py_TPFLAGS_IMMUTABLETYPE) { + for (int i=0; iname, + b->tp_name)) + { + goto finally; + } + } + } + } + /* Calculate the metaclass */ if (!metaclass) { @@ -4315,10 +4348,13 @@ type_dealloc_common(PyTypeObject *type) } +static void clear_subclasses(PyTypeObject *self); + static void clear_static_tp_subclasses(PyTypeObject *type) { - if (type->tp_subclasses == NULL) { + PyObject *subclasses = lookup_subclasses(type); + if (subclasses == NULL) { return; } @@ -4342,9 +4378,19 @@ clear_static_tp_subclasses(PyTypeObject *type) going to leak. This mostly only affects embedding scenarios. */ - // For now we just clear tp_subclasses. + // For now we just do a sanity check and then clear tp_subclasses. + Py_ssize_t i = 0; + PyObject *key, *ref; // borrowed ref + while (PyDict_Next(subclasses, &i, &key, &ref)) { + PyTypeObject *subclass = subclass_from_ref(ref); // borrowed + if (subclass == NULL) { + continue; + } + // All static builtin subtypes should have been finalized already. + assert(!(subclass->tp_flags & _Py_TPFLAGS_STATIC_BUILTIN)); + } - Py_CLEAR(type->tp_subclasses); + clear_subclasses(type); } void @@ -4394,7 +4440,7 @@ type_dealloc(PyTypeObject *type) Py_XDECREF(type->tp_bases); Py_XDECREF(type->tp_mro); Py_XDECREF(type->tp_cache); - Py_XDECREF(type->tp_subclasses); + clear_subclasses(type); /* A type's tp_doc is heap allocated, unlike the tp_doc slots * of most other objects. It's okay to cast it to char *. @@ -4414,6 +4460,30 @@ type_dealloc(PyTypeObject *type) } +static PyObject * +lookup_subclasses(PyTypeObject *self) +{ + if (self->tp_flags & _Py_TPFLAGS_STATIC_BUILTIN) { + static_builtin_state *state = _PyStaticType_GetState(self); + assert(state != NULL); + return state->tp_subclasses; + } + return (PyObject *)self->tp_subclasses; +} + +int +_PyType_HasSubclasses(PyTypeObject *self) +{ + if (self->tp_flags & _Py_TPFLAGS_STATIC_BUILTIN && + _PyStaticType_GetState(self) == NULL) { + return 0; + } + if (lookup_subclasses(self) == NULL) { + return 0; + } + return 1; +} + PyObject* _PyType_GetSubclasses(PyTypeObject *self) { @@ -4422,7 +4492,7 @@ _PyType_GetSubclasses(PyTypeObject *self) return NULL; } - PyObject *subclasses = self->tp_subclasses; // borrowed ref + PyObject *subclasses = lookup_subclasses(self); // borrowed ref if (subclasses == NULL) { return list; } @@ -5390,7 +5460,7 @@ object_getstate(PyObject *obj, int required) PyCFunction_GET_SELF(getstate) == obj && PyCFunction_GET_FUNCTION(getstate) == object___getstate__) { - /* If __getstate__ is not overriden pass the required argument. */ + /* If __getstate__ is not overridden pass the required argument. */ state = object_getstate_default(obj, required); } else { @@ -6264,11 +6334,9 @@ inherit_slots(PyTypeObject *type, PyTypeObject *base) * won't be used automatically. */ COPYSLOT(tp_vectorcall_offset); - /* Inherit Py_TPFLAGS_HAVE_VECTORCALL for non-heap types - * if tp_call is not overridden */ + /* Inherit Py_TPFLAGS_HAVE_VECTORCALL if tp_call is not overridden */ if (!type->tp_call && - _PyType_HasFeature(base, Py_TPFLAGS_HAVE_VECTORCALL) && - _PyType_HasFeature(type, Py_TPFLAGS_IMMUTABLETYPE)) + _PyType_HasFeature(base, Py_TPFLAGS_HAVE_VECTORCALL)) { type->tp_flags |= Py_TPFLAGS_HAVE_VECTORCALL; } @@ -6802,6 +6870,36 @@ _PyStaticType_InitBuiltin(PyTypeObject *self) } +static PyObject * +init_subclasses(PyTypeObject *self) +{ + PyObject *subclasses = PyDict_New(); + if (subclasses == NULL) { + return NULL; + } + if (self->tp_flags & _Py_TPFLAGS_STATIC_BUILTIN) { + static_builtin_state *state = _PyStaticType_GetState(self); + state->tp_subclasses = subclasses; + return subclasses; + } + self->tp_subclasses = (void *)subclasses; + return subclasses; +} + +static void +clear_subclasses(PyTypeObject *self) +{ + /* Delete the dictionary to save memory. _PyStaticType_Dealloc() + callers also test if tp_subclasses is NULL to check if a static type + has no subclass. */ + if (self->tp_flags & _Py_TPFLAGS_STATIC_BUILTIN) { + static_builtin_state *state = _PyStaticType_GetState(self); + Py_CLEAR(state->tp_subclasses); + return; + } + Py_CLEAR(self->tp_subclasses); +} + static int add_subclass(PyTypeObject *base, PyTypeObject *type) { @@ -6818,9 +6916,9 @@ add_subclass(PyTypeObject *base, PyTypeObject *type) // Only get tp_subclasses after creating the key and value. // PyWeakref_NewRef() can trigger a garbage collection which can execute // arbitrary Python code and so modify base->tp_subclasses. - PyObject *subclasses = base->tp_subclasses; + PyObject *subclasses = lookup_subclasses(base); if (subclasses == NULL) { - base->tp_subclasses = subclasses = PyDict_New(); + subclasses = init_subclasses(base); if (subclasses == NULL) { Py_DECREF(key); Py_DECREF(ref); @@ -6877,10 +6975,13 @@ get_subclasses_key(PyTypeObject *type, PyTypeObject *base) We fall back to manually traversing the values. */ Py_ssize_t i = 0; PyObject *ref; // borrowed ref - while (PyDict_Next((PyObject *)base->tp_subclasses, &i, &key, &ref)) { - PyTypeObject *subclass = subclass_from_ref(ref); // borrowed - if (subclass == type) { - return Py_NewRef(key); + PyObject *subclasses = lookup_subclasses(base); + if (subclasses != NULL) { + while (PyDict_Next(subclasses, &i, &key, &ref)) { + PyTypeObject *subclass = subclass_from_ref(ref); // borrowed + if (subclass == type) { + return Py_NewRef(key); + } } } /* It wasn't found. */ @@ -6890,7 +6991,7 @@ get_subclasses_key(PyTypeObject *type, PyTypeObject *base) static void remove_subclass(PyTypeObject *base, PyTypeObject *type) { - PyObject *subclasses = base->tp_subclasses; // borrowed ref + PyObject *subclasses = lookup_subclasses(base); // borrowed ref if (subclasses == NULL) { return; } @@ -6906,10 +7007,7 @@ remove_subclass(PyTypeObject *base, PyTypeObject *type) Py_XDECREF(key); if (PyDict_Size(subclasses) == 0) { - // Delete the dictionary to save memory. _PyStaticType_Dealloc() - // callers also test if tp_subclasses is NULL to check if a static type - // has no subclass. - Py_CLEAR(base->tp_subclasses); + clear_subclasses(base); } } @@ -8687,8 +8785,17 @@ update_one_slot(PyTypeObject *type, slotdef *p) { PyObject *descr; PyWrapperDescrObject *d; - void *generic = NULL, *specific = NULL; + + // The correct specialized C function, like "tp_repr of str" in the + // example above + void *specific = NULL; + + // A generic wrapper that uses method lookup (safe but slow) + void *generic = NULL; + + // Set to 1 if the generic wrapper is necessary int use_generic = 0; + int offset = p->offset; int error; void **ptr = slotptr(type, offset); @@ -8771,6 +8878,10 @@ update_one_slot(PyTypeObject *type, slotdef *p) else { use_generic = 1; generic = p->function; + if (p->function == slot_tp_call) { + /* A generic __call__ is incompatible with vectorcall */ + type->tp_flags &= ~Py_TPFLAGS_HAVE_VECTORCALL; + } } } while ((++p)->offset == offset); if (specific && !use_generic) @@ -8981,7 +9092,7 @@ recurse_down_subclasses(PyTypeObject *type, PyObject *attr_name, // It is safe to use a borrowed reference because update_subclasses() is // only used with update_slots_callback() which doesn't modify // tp_subclasses. - PyObject *subclasses = type->tp_subclasses; // borrowed ref + PyObject *subclasses = lookup_subclasses(type); // borrowed ref if (subclasses == NULL) { return 0; } diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index 7ff79953257ee6..184a2bfd5dd869 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -2355,6 +2355,13 @@ unicode_fromformat_arg(_PyUnicodeWriter *writer, p = f; f++; + if (*f == '%') { + if (_PyUnicodeWriter_WriteCharInline(writer, '%') < 0) + return NULL; + f++; + return f; + } + zeropad = 0; if (*f == '0') { zeropad = 1; @@ -2392,14 +2399,6 @@ unicode_fromformat_arg(_PyUnicodeWriter *writer, f++; } } - if (*f == '%') { - /* "%.3%s" => f points to "3" */ - f--; - } - } - if (*f == '\0') { - /* bogus format "%.123" => go backward, f points to "3" */ - f--; } /* Handle %ld, %lu, %lld and %llu. */ @@ -2423,7 +2422,7 @@ unicode_fromformat_arg(_PyUnicodeWriter *writer, ++f; } - if (f[1] == '\0') + if (f[0] != '\0' && f[1] == '\0') writer->overallocate = 0; switch (*f) { @@ -2616,21 +2615,9 @@ unicode_fromformat_arg(_PyUnicodeWriter *writer, break; } - case '%': - if (_PyUnicodeWriter_WriteCharInline(writer, '%') < 0) - return NULL; - break; - default: - /* if we stumble upon an unknown formatting code, copy the rest - of the format string to the output string. (we cannot just - skip the code, since there's no way to know what's in the - argument list) */ - len = strlen(p); - if (_PyUnicodeWriter_WriteLatin1String(writer, p, len) == -1) - return NULL; - f = p+len; - return f; + PyErr_Format(PyExc_SystemError, "invalid format string: %s", p); + return NULL; } f++; diff --git a/PC/python3dll.c b/PC/python3dll.c index 024ec49d68d797..89bbd05932b853 100755 --- a/PC/python3dll.c +++ b/PC/python3dll.c @@ -723,6 +723,8 @@ EXPORT_FUNC(PyUnicodeTranslateError_GetStart) EXPORT_FUNC(PyUnicodeTranslateError_SetEnd) EXPORT_FUNC(PyUnicodeTranslateError_SetReason) EXPORT_FUNC(PyUnicodeTranslateError_SetStart) +EXPORT_FUNC(PyVectorcall_Call) +EXPORT_FUNC(PyVectorcall_NARGS) EXPORT_FUNC(PyWeakref_GetObject) EXPORT_FUNC(PyWeakref_NewProxy) EXPORT_FUNC(PyWeakref_NewRef) diff --git a/PCbuild/_sqlite3.vcxproj b/PCbuild/_sqlite3.vcxproj index 804aa07367a024..57c7413671e54e 100644 --- a/PCbuild/_sqlite3.vcxproj +++ b/PCbuild/_sqlite3.vcxproj @@ -94,7 +94,7 @@ $(sqlite3Dir);%(AdditionalIncludeDirectories) - PY_SQLITE_HAVE_SERIALIZE;%(PreprocessorDefinitions) + PY_SQLITE_HAVE_SERIALIZE;PY_SQLITE_ENABLE_LOAD_EXTENSION;%(PreprocessorDefinitions) diff --git a/PCbuild/_testcapi.vcxproj b/PCbuild/_testcapi.vcxproj index a88540cab19f9a..23bb5ec85274ae 100644 --- a/PCbuild/_testcapi.vcxproj +++ b/PCbuild/_testcapi.vcxproj @@ -95,7 +95,9 @@ + + diff --git a/PCbuild/_testcapi.vcxproj.filters b/PCbuild/_testcapi.vcxproj.filters index a43ab5ea0ff941..fc2c4345fe142e 100644 --- a/PCbuild/_testcapi.vcxproj.filters +++ b/PCbuild/_testcapi.vcxproj.filters @@ -15,9 +15,15 @@ Source Files + + Source Files + Source Files + + Source Files + diff --git a/PCbuild/lib.pyproj b/PCbuild/lib.pyproj index 0556efe1a77d2b..e8c99f6b246c82 100644 --- a/PCbuild/lib.pyproj +++ b/PCbuild/lib.pyproj @@ -642,7 +642,6 @@ - @@ -760,6 +759,7 @@ + @@ -1261,7 +1261,6 @@ - diff --git a/Python/stdlib_module_names.h b/Python/stdlib_module_names.h index ff422e72301847..b28156608d1b80 100644 --- a/Python/stdlib_module_names.h +++ b/Python/stdlib_module_names.h @@ -241,7 +241,6 @@ static const char* _Py_stdlib_module_names[] = { "shutil", "signal", "site", -"smtpd", "smtplib", "sndhdr", "socket", diff --git a/Tools/c-analyzer/c_parser/parser/__init__.py b/Tools/c-analyzer/c_parser/parser/__init__.py index 0343c2d68aac9f..4227e938d7f8da 100644 --- a/Tools/c-analyzer/c_parser/parser/__init__.py +++ b/Tools/c-analyzer/c_parser/parser/__init__.py @@ -12,7 +12,7 @@ * ... -(see: http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf) +(see: https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf) We have taken advantage of the elements of the C grammar that are used only in a few limited contexts, mostly as delimiters. They allow us to diff --git a/Tools/c-analyzer/cpython/globals-to-fix.tsv b/Tools/c-analyzer/cpython/globals-to-fix.tsv index 496bc9a264b1da..c8d23e9db0e127 100644 --- a/Tools/c-analyzer/cpython/globals-to-fix.tsv +++ b/Tools/c-analyzer/cpython/globals-to-fix.tsv @@ -539,32 +539,6 @@ Python/specialize.c - _Py_QuickenedCount - ################################## # global objects to fix in builtin modules -#----------------------- -# modules - -Modules/_abc.c - _abcmodule - -Modules/_codecsmodule.c - codecsmodule - -Modules/_collectionsmodule.c - _collectionsmodule - -Modules/_functoolsmodule.c - _functools_module - -Modules/_io/_iomodule.c - _PyIO_Module - -Modules/_io/_iomodule.h - _PyIO_Module - -Modules/_localemodule.c - _localemodule - -Modules/_sre.c - sremodule - -Modules/_stat.c - statmodule - -Modules/_threadmodule.c - threadmodule - -Modules/_tracemalloc.c - module_def - -Modules/_weakref.c - weakrefmodule - -Modules/atexitmodule.c - atexitmodule - -Modules/errnomodule.c - errnomodule - -Modules/faulthandler.c - module_def - -Modules/gcmodule.c - gcmodule - -Modules/itertoolsmodule.c - itertoolsmodule - -Modules/posixmodule.c - posixmodule - -Modules/pwdmodule.c - pwdmodule - -Modules/signalmodule.c - signalmodule - -Modules/symtablemodule.c - symtablemodule - -Modules/timemodule.c - timemodule - - #----------------------- # static types diff --git a/Tools/c-analyzer/cpython/ignored.tsv b/Tools/c-analyzer/cpython/ignored.tsv index ba48ef4933a465..a395775e8f1b2f 100644 --- a/Tools/c-analyzer/cpython/ignored.tsv +++ b/Tools/c-analyzer/cpython/ignored.tsv @@ -30,6 +30,7 @@ Modules/_io/_iomodule.h - PyBufferedRandom_Type - Modules/_io/_iomodule.h - PyTextIOWrapper_Type - Modules/_io/_iomodule.h - PyIncrementalNewlineDecoder_Type - Modules/_io/_iomodule.h - _PyBytesIOBuffer_Type - +Modules/_io/_iomodule.h - _PyIO_Module - Modules/_io/_iomodule.h - _PyIO_str_close - Modules/_io/_iomodule.h - _PyIO_str_closed - Modules/_io/_iomodule.h - _PyIO_str_decode - @@ -405,16 +406,35 @@ Python/sysmodule.c sys_set_asyncgen_hooks keywords - #----------------------- # PyModuleDef + +Modules/_abc.c - _abcmodule - +Modules/_codecsmodule.c - codecsmodule - +Modules/_collectionsmodule.c - _collectionsmodule - +Modules/_functoolsmodule.c - _functools_module - +Modules/_io/_iomodule.c - _PyIO_Module - +Modules/_localemodule.c - _localemodule - Modules/_multiprocessing/posixshmem.c - _posixshmemmodule - Modules/_sqlite/module.h - _sqlite3module - -Modules/_sre/sre.c - sremodule static struct - +Modules/_sre/sre.c - sremodule - Modules/_ssl.c - _sslmodule_def - Modules/_ssl.h - _sslmodule_def - -Modules/_testcapi/heaptype.c - _testcapimodule static - +Modules/_stat.c - statmodule - +Modules/_testcapi/heaptype.c - _testcapimodule - Modules/_testmultiphase.c - def_module_state_shared - Modules/_threadmodule.c - thread_module - +Modules/_tracemalloc.c - module_def - Modules/_typingmodule.c - typingmodule - +Modules/_weakref.c - weakrefmodule - +Modules/atexitmodule.c - atexitmodule - +Modules/errnomodule.c - errnomodule - +Modules/faulthandler.c - module_def - +Modules/gcmodule.c - gcmodule - +Modules/itertoolsmodule.c - itertoolsmodule - +Modules/posixmodule.c - posixmodule - +Modules/pwdmodule.c - pwdmodule - Modules/signalmodule.c - signal_module - +Modules/symtablemodule.c - symtablemodule - +Modules/timemodule.c - timemodule - Modules/xxlimited_35.c - xxmodule - Python/Python-ast.c - _astmodule - Python/Python-tokenize.c - _tokenizemodule - @@ -1371,8 +1391,10 @@ Modules/_testcapi/heaptype.c - HeapCTypeSubclassWithFinalizer_spec - Modules/_testcapi/heaptype.c - HeapCTypeMetaclass_spec - Modules/_testcapi/heaptype.c - HeapCTypeMetaclassCustomNew_spec - Modules/_testcapi/heaptype.c - HeapCTypeWithDict_spec - +Modules/_testcapi/heaptype.c - HeapCTypeWithDict2_spec - Modules/_testcapi/heaptype.c - HeapCTypeWithNegativeDict_spec - Modules/_testcapi/heaptype.c - HeapCTypeWithWeakref_spec - +Modules/_testcapi/heaptype.c - HeapCTypeWithWeakref2_spec - Modules/_testcapi/heaptype.c - HeapCTypeSetattr_spec - Modules/_testcapimodule.c - HeapTypeNameType_Spec - Modules/_testcapimodule.c - NullTpDocType_spec - diff --git a/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp b/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp index fdc2a21d83d5f3..3a17ffbaa0b655 100644 --- a/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp +++ b/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp @@ -724,6 +724,8 @@ class PythonBootstrapperApplication : public CBalBaseBootstrapperApplication { auto hr = LoadAssociateFilesStateFromKey(_engine, fPerMachine ? HKEY_LOCAL_MACHINE : HKEY_CURRENT_USER); if (hr == S_OK) { _engine->SetVariableNumeric(L"AssociateFiles", 1); + } else if (hr == S_FALSE) { + _engine->SetVariableNumeric(L"AssociateFiles", 0); } else if (FAILED(hr)) { BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "Failed to load AssociateFiles state: error code 0x%08X", hr); } @@ -817,6 +819,8 @@ class PythonBootstrapperApplication : public CBalBaseBootstrapperApplication { auto hr = LoadAssociateFilesStateFromKey(_engine, hkey); if (hr == S_OK) { _engine->SetVariableNumeric(L"AssociateFiles", 1); + } else if (hr == S_FALSE) { + _engine->SetVariableNumeric(L"AssociateFiles", 0); } else if (FAILED(hr)) { BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "Failed to load AssociateFiles state: error code 0x%08X", hr); } @@ -834,7 +838,17 @@ class PythonBootstrapperApplication : public CBalBaseBootstrapperApplication { LONGLONG includeLauncher; if (SUCCEEDED(BalGetNumericVariable(L"Include_launcher", &includeLauncher)) && includeLauncher == -1) { - _engine->SetVariableNumeric(L"Include_launcher", 1); + if (BOOTSTRAPPER_ACTION_LAYOUT == _command.action || + (BOOTSTRAPPER_ACTION_INSTALL == _command.action && !_upgrading)) { + // When installing/downloading, we want to include the launcher + // by default. + _engine->SetVariableNumeric(L"Include_launcher", 1); + } else { + // Any other action, if we didn't detect the MSI then we want to + // keep it excluded + _engine->SetVariableNumeric(L"Include_launcher", 0); + _engine->SetVariableNumeric(L"AssociateFiles", 0); + } } } @@ -2812,6 +2826,17 @@ class PythonBootstrapperApplication : public CBalBaseBootstrapperApplication { return ::CompareStringW(LOCALE_NEUTRAL, 0, platform, -1, L"x64", -1) == CSTR_EQUAL; } + static bool IsTargetPlatformARM64(__in IBootstrapperEngine* pEngine) { + WCHAR platform[8]; + DWORD platformLen = 8; + + if (FAILED(pEngine->GetVariableString(L"TargetPlatform", platform, &platformLen))) { + return S_FALSE; + } + + return ::CompareStringW(LOCALE_NEUTRAL, 0, platform, -1, L"ARM64", -1) == CSTR_EQUAL; + } + static HRESULT LoadOptionalFeatureStatesFromKey( __in IBootstrapperEngine* pEngine, __in HKEY hkHive, @@ -2820,7 +2845,7 @@ class PythonBootstrapperApplication : public CBalBaseBootstrapperApplication { HKEY hKey; LRESULT res; - if (IsTargetPlatformx64(pEngine)) { + if (IsTargetPlatformx64(pEngine) || IsTargetPlatformARM64(pEngine)) { res = RegOpenKeyExW(hkHive, subkey, 0, KEY_READ | KEY_WOW64_64KEY, &hKey); } else { res = RegOpenKeyExW(hkHive, subkey, 0, KEY_READ | KEY_WOW64_32KEY, &hKey); @@ -2859,7 +2884,7 @@ class PythonBootstrapperApplication : public CBalBaseBootstrapperApplication { BYTE buffer[1024]; DWORD bufferLen = sizeof(buffer); - if (IsTargetPlatformx64(pEngine)) { + if (IsTargetPlatformx64(pEngine) || IsTargetPlatformARM64(pEngine)) { res = RegOpenKeyExW(hkHive, subkey, 0, KEY_READ | KEY_WOW64_64KEY, &hKey); } else { res = RegOpenKeyExW(hkHive, subkey, 0, KEY_READ | KEY_WOW64_32KEY, &hKey); @@ -2917,12 +2942,7 @@ class PythonBootstrapperApplication : public CBalBaseBootstrapperApplication { HRESULT hr; HKEY hkHive; - // The launcher installation is separate from the Python install, so we - // check its state later. For now, assume we don't want the launcher or - // file associations, and if they have already been installed then - // loading the state will reactivate these settings. - pEngine->SetVariableNumeric(L"Include_launcher", 0); - pEngine->SetVariableNumeric(L"AssociateFiles", 0); + BalLog(BOOTSTRAPPER_LOG_LEVEL_STANDARD, "Loading state of optional features"); // Get the registry key from the bundle, to save having to duplicate it // in multiple places. diff --git a/Tools/wasm/wasm_assets.py b/Tools/wasm/wasm_assets.py index 07450ac928dcd6..695e6ffc31fabb 100755 --- a/Tools/wasm/wasm_assets.py +++ b/Tools/wasm/wasm_assets.py @@ -77,7 +77,6 @@ "mailcap.py", "nntplib.py", "poplib.py", - "smtpd.py", "smtplib.py", "socketserver.py", "telnetlib.py", diff --git a/configure b/configure index 5df9f83290da06..3f25d43dde6f0c 100755 --- a/configure +++ b/configure @@ -927,7 +927,6 @@ MULTIARCH_CPPFLAGS PLATFORM_TRIPLET MULTIARCH ac_ct_CXX -MAINCC CXX EGREP SED @@ -1036,7 +1035,6 @@ enable_universalsdk with_universal_archs with_framework_name enable_framework -with_cxx_main with_emscripten_target enable_wasm_dynamic_linking enable_wasm_pthreads @@ -1805,9 +1803,6 @@ Optional Packages: specify the name for the python framework on macOS only valid when --enable-framework is set. see Mac/README.rst (default is 'Python') - --with-cxx-main[=COMPILER] - compile main() and link Python executable with C++ - compiler specified in COMPILER (default is $CXX) --with-emscripten-target=[browser|node] Emscripten platform --with-suffix=SUFFIX set executable suffix to SUFFIX (default is empty, @@ -5550,35 +5545,6 @@ $as_echo "$ac_cv_safe_to_define___extensions__" >&6; } -{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for --with-cxx-main=" >&5 -$as_echo_n "checking for --with-cxx-main=... " >&6; } - -# Check whether --with-cxx_main was given. -if test "${with_cxx_main+set}" = set; then : - withval=$with_cxx_main; - - case $withval in - no) with_cxx_main=no - MAINCC='$(CC)';; - yes) with_cxx_main=yes - MAINCC='$(CXX)';; - *) with_cxx_main=yes - MAINCC=$withval - if test -z "$CXX" - then - CXX=$withval - fi;; - esac -else - - with_cxx_main=no - MAINCC='$(CC)' - -fi - -{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_cxx_main" >&5 -$as_echo "$with_cxx_main" >&6; } - preset_cxx="$CXX" if test -z "$CXX" then @@ -6661,7 +6627,7 @@ LDVERSION="$VERSION" $as_echo_n "checking LINKCC... " >&6; } if test -z "$LINKCC" then - LINKCC='$(PURIFY) $(MAINCC)' + LINKCC='$(PURIFY) $(CC)' case $ac_sys_system in QNX*) # qcc must be used because the other compilers do not diff --git a/configure.ac b/configure.ac index 38880fcc8cc15b..8decd9ebae84cf 100644 --- a/configure.ac +++ b/configure.ac @@ -865,29 +865,6 @@ rm -f conftest.c conftest.out AC_USE_SYSTEM_EXTENSIONS AC_SUBST(CXX) -AC_SUBST(MAINCC) -AC_MSG_CHECKING(for --with-cxx-main=) -AC_ARG_WITH(cxx_main, - AS_HELP_STRING([--with-cxx-main@<:@=COMPILER@:>@], - [compile main() and link Python executable with C++ compiler specified in COMPILER (default is $CXX)]), -[ - - case $withval in - no) with_cxx_main=no - MAINCC='$(CC)';; - yes) with_cxx_main=yes - MAINCC='$(CXX)';; - *) with_cxx_main=yes - MAINCC=$withval - if test -z "$CXX" - then - CXX=$withval - fi;; - esac], [ - with_cxx_main=no - MAINCC='$(CC)' -]) -AC_MSG_RESULT($with_cxx_main) preset_cxx="$CXX" if test -z "$CXX" @@ -1358,7 +1335,7 @@ AC_SUBST(LINKCC) AC_MSG_CHECKING(LINKCC) if test -z "$LINKCC" then - LINKCC='$(PURIFY) $(MAINCC)' + LINKCC='$(PURIFY) $(CC)' case $ac_sys_system in QNX*) # qcc must be used because the other compilers do not