3.8.0b1.rst

:cve:`2019-9948`: Avoid file reading by disallowing local-file:// and local_file:// URL schemes in URLopener().open() and URLopener().retrieve() of :mod:`urllib.request`.

Prevent fold function used in email header encoding from entering infinite loop when there are too many non-ASCII characters in a header.

Updated blake2 implementation which uses secure memset implementation provided by platform.

Allow unpacking in the right hand side of annotated assignments. In particular, t: Tuple[int, ...] = x, y, *z is now allowed.

All structseq objects are now tracked by the garbage collector. Patch by Pablo Galindo.

Make the co_argcount attribute of code objects represent the total number of positional arguments (including positional-only arguments). The value of co_posonlyargcount can be used to distinguish which arguments are positional only, and the difference (co_argcount - co_posonlyargcount) is the number of positional-or-keyword arguments. Patch by Pablo Galindo.

Constructors of :class:`int`, :class:`float` and :class:`complex` will now use the :meth:`~object.__index__` special method, if available and the corresponding method :meth:`~object.__int__`, :meth:`~object.__float__` or :meth:`~object.__complex__` is not available.

Add native thread ID (TID) support to OpenBSD.

Implemented per opcode cache mechanism and LOAD_GLOBAL instruction use it. LOAD_GLOBAL is now about 40% faster. Contributed by Yury Selivanov, and Inada Naoki.

Fix crash in PyAST_FromNodeObject() when flags is NULL.

Freeing a great many small objects could take time quadratic in the number of arenas, due to using linear search to keep obmalloc.c's list of usable arenas sorted by order of number of free memory pools. This is accomplished without search now, leaving the worst-case time linear in the number of arenas. For programs where this quite visibly matters (typically with more than 100 thousand small objects alive simultaneously), this can greatly reduce the time needed to release their memory.

Fix possible overflow in wrap_lenfunc() when sizeof(long) < sizeof(Py_ssize_t) (e.g., 64-bit Windows).

Improve the AST for "debug" f-strings, which use '=' to print out the source of the expression being evaluated. Delete expr_text from the FormattedValue node, and instead use a Constant string node (possibly merged with adjacent constant expressions inside the f-string).

The bytes.hex, bytearray.hex, and memoryview.hex methods as well as the binascii.hexlify and b2a_hex functions now have the ability to include an optional separator between hex bytes. This functionality was inspired by MicroPython's hexlify implementation.

Add :func:`os.memfd_create`.

Added new replace() method to the code type (:class:`types.CodeType`).

Implement :func:`socket.if_nameindex`, :func:`socket.if_nametoindex`, and :func:`socket.if_indextoname` on Windows.

:c:func:`PyErr_WriteUnraisable` now creates a traceback object if there is no current traceback. Moreover, call :c:func:`PyErr_NormalizeException` and :c:func:`PyException_SetTraceback` to normalize the exception value. Ignore any error.

Only accept text after # type: ignore if the first character is ASCII. This is to disallow things like # type: ignoreé.

Store text appearing after a # type: ignore comment in the AST. For example a type ignore like # type: ignore[E1000] will have the string "[E1000]" stored in its AST node.

Treat line continuation at EOF as a SyntaxError by Anthony Sottile.

Fix a crash when calling a C function with a keyword dict (f(**kwargs)) and changing the dict kwargs while that function is running.

Fix possible signed integer overflow when handling slices.

Add NamedExpression kind support to ast_unparse.c

A :exc:`SyntaxError` is now raised if a code blocks that will be optimized away (e.g. if conditions that are always false) contains syntax errors. Patch by Pablo Galindo.

Allow computation of modular inverses via three-argument pow: the second argument is now permitted to be negative in the case where the first and third arguments are relatively prime.

Update the Unicode database to version 12.1.0.

Avoid caching attributes of classes which type defines mro() to avoid a hard cache invalidation problem.

The FrameType stack is now correctly cleaned up if the execution ends with a return and the stack is not empty.

The compile() builtin functions now support the ast.PyCF_ALLOW_TOP_LEVEL_AWAIT flag, which allow to compile sources that contains top-level await, async with or async for. This is useful to evaluate async-code from with an already async functions; for example in a custom REPL.

Implement PEP 578, adding sys.audit, io.open_code and related APIs.

Correct return type for UserList slicing operations. Patch by Michael Blahay, Erick Cervantes, and vaultah

Move PyRuntimeState.warnings into per-interpreter state (via "module state").

Removed __str__ implementations from builtin types :class:`bool`, :class:`int`, :class:`float`, :class:`complex` and few classes from the standard library. They now inherit __str__() from :class:`object`.

Add a = feature f-strings for debugging. This can precede !s, !r, or !a. It produces the text of the expression, followed by an equal sign, followed by the repr of the value of the expression. So f'{3*9+15=}' would be equal to the string '3*9+15=42'. If = is specified, the default conversion is set to !r, unless a format spec is given, in which case the formatting behavior is unchanged, and __format__ will be used.

Save the live exception during import.c's remove_module().

pymalloc returns memory blocks aligned by 16 bytes, instead of 8 bytes, on 64-bit platforms to conform x86-64 ABI. Recent compilers assume this alignment more often. Patch by Inada Naoki.

A long-since-meaningless check for getpid() == main_pid was removed from Python's internal C signal handler.

Fix incorrect use of %p in format strings. Patch by Zackery Spytz.

builtins.help() now prefixes async for async functions.

Add native thread ID (TID) to threading.Thread objects (supported platforms: Windows, FreeBSD, Linux, macOS)

Added fix for broken symlinks in combination with pathlib

Added new trashcan macros to deal with a double deallocation that could occur when the tp_dealloc of a subclass calls the tp_dealloc of a base class and that base class uses the trashcan mechanism. Patch by Jeroen Demeyer.

Do not clear :data:`sys.flags` and :data:`sys.float_info` during shutdown. Patch by Zackery Spytz.

Expose :func:`copy_file_range` as a low level API in the :mod:`os` module.

Remove cross-version binary compatibility requirement in tp_flags.

Port binascii to PEP 489 multiphase initialization. Patch by Marcel Plch.

Added :func:`math.perm`.

Add SSLContext.num_tickets to control the number of TLSv1.3 session tickets.

Fix the error handling in :meth:`!msilib.SummaryInformation.GetProperty`. Patch by Zackery Spytz.

The fcntl module now contains file sealing constants for sealing of memfds.

Add get_origin() and get_args() introspection helpers to typing module.

:meth:`!msilib.Directory.start_component` no longer fails if keyfile is not None.

Add the asyncio.Task.get_coro() method to publicly expose the tasks's coroutine object.

Make :func:`asyncio.create_subprocess_exec` accept path-like arguments.

Change default max_workers of ThreadPoolExecutor from cpu_count() * 5 to min(32, cpu_count() + 4). Previous value was unreasonably large on many cores machines.

:func:`_thread.start_new_thread` now logs uncaught exception raised by the function using :func:`sys.unraisablehook`, rather than :func:`sys.excepthook`, so the hook gets access to the function which raised the exception.

On macOS, the :mod:`multiprocessing` module now uses spawn start method by default.

Fix destructor :class:`!_pyio.BytesIO` and :class:`!_pyio.TextIOWrapper`: initialize their _buffer attribute as soon as possible (in the class body), because it's used by __del__() which calls close().

PEP 544: Add Protocol and @runtime_checkable to the typing module.

The functions sys.set_coroutine_wrapper and sys.get_coroutine_wrapper that were deprecated and marked for removal in 3.8 have been removed.

Handle late binding and attribute access in :class:`unittest.mock.AsyncMock` setup for autospeccing. Document newly implemented async methods in :class:`unittest.mock.MagicMock`.

PEP 589: Add TypedDict to the typing module.

PEP 586: Add Literal to the typing module.

PEP 591: Add Final qualifier and @final decorator to the typing module.

Don't log OSError based exceptions if a fatal error has occurred in asyncio transport. Peer can generate almost any OSError, user cannot avoid these exceptions by fixing own code. Errors are still propagated to user code, it's just logging them is pointless and pollute asyncio logs.

:func:`symtable.symtable` now accepts the same input types for source code as the built-in :func:`compile` function. Patch by Dino Viehland.

Implement asyncio REPL

Return safe to use proxy socket object from transport.get_extra_info('socket')

Make asyncio.CancelledError a BaseException.

This will address the common mistake many asyncio users make: an "except Exception" clause breaking Tasks cancellation.

In addition to this change, we stop inheriting asyncio.TimeoutError and asyncio.InvalidStateError from their concurrent.futures.* counterparts. There's no point for these exceptions to share the inheritance chain.

Add a new :func:`threading.excepthook` function which handles uncaught :meth:`threading.Thread.run` exception. It can be overridden to control how uncaught :meth:`threading.Thread.run` exceptions are handled.

Handle :func:`unittest.mock.patch` used as a decorator on async functions.

Add support for calling :func:`next` with the mock resulting from :func:`unittest.mock.mock_open`

Allow whitespace only header encoding in email.header - by Batuhan Taskaya

PDB command args now display positional only arguments. Patch contributed by Rémi Lapeyre.

PDB command args now display keyword only arguments. Patch contributed by Rémi Lapeyre.

Add missing names to typing.__all__: ChainMap, ForwardRef, OrderedDict - by Anthony Sottile.

Add SupportsIndex protocol to the typing module to allow type checking to detect classes that can be passed to hex(), oct() and bin().

Implement unittest.IsolatedAsyncioTestCase to help testing asyncio-based code.

:func:`fileinput.input` and :class:`fileinput.FileInput` bufsize argument has been removed (was deprecated and ignored since Python 3.6), and as a result the mode and openhook arguments have been made keyword-only.

Starting with Python 3.3, importing ABCs from :mod:`collections` is deprecated, and import should be done from :mod:`collections.abc`. Still being able to import from :mod:`collections` was marked for removal in 3.8, but has been delayed to 3.9; documentation and DeprecationWarning clarified.

Implement __repr__ for WeakSet objects.

Fix :exc:`NameError` in :meth:`urllib.request.URLopener.retrieve`. Patch by Karthikeyan Singaravelan.

Fix the folding of email header when the max_line_length is 0 or None and the header contains non-ascii characters. Contributed by Licht Takeuchi (@Licht-T).

:func:`shutil.copystat` now ignores :const:`errno.EINVAL` on :func:`os.setxattr` which may occur when copying files on filesystems without extended attributes support.

Original patch by Giampaolo Rodola, updated by Ying Wang.

Python child processes can now access the status of their parent process using multiprocessing.process.parent_process

Deprecate @coroutine for sake of async def.

Fix bug in __rmod__ of UserString - by Batuhan Taskaya.

Remove a message about an unhandled exception in a task when writer.write() is used without await and writer.drain() fails with an exception.

Introduce :class:`asyncio.Stream` class that merges :class:`asyncio.StreamReader` and :class:`asyncio.StreamWriter` functionality. :class:`asyncio.Stream` can work in readonly, writeonly and readwrite modes. Provide :func:`asyncio.connect`, :func:`asyncio.connect_unix`, :func:`asyncio.connect_read_pipe` and :func:`asyncio.connect_write_pipe` factories to open :class:`asyncio.Stream` connections. Provide :class:`asyncio.StreamServer` and :class:`UnixStreamServer` to serve servers with asyncio.Stream API. Modify :func:`asyncio.create_subprocess_shell` and :func:`asyncio.create_subprocess_exec` to use :class:`asyncio.Stream` instead of deprecated :class:`StreamReader` and :class:`StreamWriter`. Deprecate :class:`asyncio.StreamReader` and :class:`asyncio.StreamWriter`. Deprecate usage of private classes, e.g. :class:`asyncio.FlowControlMixing` and :class:`asyncio.StreamReaderProtocol` outside of asyncio package.

Added validation of integer prefixes to the construction of IP networks and interfaces in the ipaddress module.

Add an extend action to argparser.

Fix a bug making a SharedMemoryManager instance and its parent process use two separate resource_tracker processes.

Adds a grammar to lib2to3.pygram that contains exec as a function not as statement.

The function time.clock() was deprecated in 3.3 in favor of time.perf_counter() and marked for removal in 3.8, it has removed.

Fix asyncio discarding IPv6 scopes when ensuring hostname resolutions internally

Add new function :func:`math.isqrt` to compute integer square roots.

Introduce the importlib.metadata module with (provisional) support for reading metadata from third-party packages.

When using type_comments=True in ast.parse, treat # type: ignore followed by a non-alphanumeric character and then arbitrary text as a type ignore, instead of requiring nothing but whitespace or another comment. This is to permit formations such as # type: ignore[E1000].

cp65001 encoding (Windows code page 65001) becomes an alias to utf_8 encoding.

The multiprocessing.resource_tracker replaces the multiprocessing.semaphore_tracker module. Other than semaphores, resource_tracker also tracks shared_memory segments.

The Cache and Statement objects of the :mod:`sqlite3` module are not exposed to the user. Patch by Aviv Palivoda.

In shutil.copystat(), first copy extended file attributes and then file permissions, since extended attributes can only be set on the destination while it is still writeable.

Add new :func:`sys.unraisablehook` function which can be overridden to control how "unraisable exceptions" are handled. It is called when an exception has occurred but there is no way for Python to handle it. For example, when a destructor raises an exception or during garbage collection (:func:`gc.collect`).

Introducing zipfile.Path, a pathlib-compatible wrapper for traversing zip files.

Fix an issue where os.posix_spawnp() would incorrectly raise a TypeError when file_actions is None.

Handle exceptions raised by functions added by concurrent.futures add_done_callback correctly when the Future has already completed.

Limit max_workers in ProcessPoolExecutor to 61 to work around a WaitForMultipleObjects limitation.

Fix :class:`~logging.handlers.QueueListener` to call queue.task_done() upon stopping. Patch by Bar Harel.

Forbid creation of asyncio stream objects like StreamReader, StreamWriter, Process, and their protocols outside of asyncio package.

Provide both sync and async calls for StreamWriter.write() and StreamWriter.close()

Properly handle SSL connection closing in asyncio StreamWriter.drain() call.

Implement PEP 574 (pickle protocol 5 with out-of-band buffers).

functools.lru_cache() can now be used as a straight decorator in addition to its existing usage as a function that returns a decorator.

Add a :exc:`~gzip.BadGzipFile` exception to the :mod:`gzip` module.

Optimized write buffering in C implementation of TextIOWrapper. Writing ASCII string to TextIOWrapper with ascii, latin1, or utf-8 encoding is about 20% faster. Patch by Inada Naoki.

Don't mark wsgiref.simple_server.SimpleServer as multi-threaded since wsgiref.simple_server.WSGIServer is single-threaded.

:func:`py_compile.compile` now supports silent mode. Patch by Joannah Nanjekye

Fix double exceptions in :class:`wsgiref.handlers.BaseHandler` by calling its :meth:`~wsgiref.handlers.BaseHandler.close` method only when no exception is raised.

Improved the repr of regular expression flags.

The signature of Python functions can now be overridden by specifying the __text_signature__ attribute.

Reinitialize logging.Handler locks in forked child processes instead of attempting to acquire them all in the parent before forking only to be released in the child process. The acquire/release pattern was leading to deadlocks in code that has implemented any form of chained logging handlers that depend upon one another as the lock acquisition order cannot be guaranteed.

Throw a TypeError instead of an AssertionError when using an invalid type annotation with singledispatch.

Allow reduction methods to return a 6-item tuple where the 6th item specifies a custom state-setting method that's called instead of the regular __setstate__ method.

enable custom reduction callback registration for functions and classes in _pickle.c, using the new Pickler's attribute reducer_override

Fix a bug crashing SharedMemoryManager instances in interactive sessions after a ctrl-c (KeyboardInterrupt) was sent

Fix mmap fail for VxWorks

:meth:`csv.DictWriter.writeheader` now returns the return value of the underlying :meth:`csv.Writer.writerow` method. Patch contributed by Ashish Nitin Patil.

Parsing .mo files now ignores comments starting and ending with #-#-#-#-#.

Enable plistlib to read and write binary plist files that were created as a KeyedArchive file. Specifically, this allows the plistlib to process 0x80 tokens as UID objects.

Add posix module support for VxWorks.

Asyncio: Remove inner callback on outer cancellation in shield

Fix :meth:`asyncio.SelectorEventLoop.subprocess_exec` leaks file descriptors if Popen fails and called with stdin=subprocess.PIPE. Patch by Niklas Fiekas.

:func:`unittest.mock.mock_open` results now respects the argument of read([size]). Patch contributed by Rémi Lapeyre.

Implement :func:`math.comb` that returns binomial coefficient, that computes the number of ways to choose k items from n items without repetition and without order. Patch by Yash Aggarwal and Keller Fuchs.

Fixed permission errors in :class:`~tempfile.TemporaryDirectory` clean up. Previously TemporaryDirectory.cleanup() failed when non-writeable or non-searchable files or directories were created inside a temporary directory.

Add debugging helpers to ssl module. It's now possible to dump key material and to trace TLS protocol. The default and stdlib contexts also support SSLKEYLOGFILE env var.

Added AsyncMock to support using unittest to mock asyncio coroutines. Patch by Lisa Roach.

dataclasses.InitVar: Exposes the type used to create the init var.

Fix serialization of messages containing encoded strings when the policy.linesep is set to a multi-character string. Patch by Jens Troeger.

Performance of :func:`functools.reduce` is slightly improved. Patch by Sergey Fedoseev.

Fix a bug in :class:`codecs.StreamRecoder` where seeking might leave old data in a buffer and break subsequent read calls. Patch by Ammar Askar.

The :mod:`shlex` module now exposes :func:`shlex.join`, the inverse of :func:`shlex.split`. Patch by Bo Bayles.

:meth:`asyncio.AbstractEventLoop.create_datagram_endpoint`: Do not connect UDP socket when broadcast is allowed. This allows to receive replies after a UDP broadcast.

Change ThreadPoolExecutor to use existing idle threads before spinning up new ones.

Added support for bytes and path-like objects in :func:`subprocess.Popen` on Windows. The args parameter now accepts a :term:`path-like object` if shell is False and a sequence containing bytes and path-like objects. The executable parameter now accepts a bytes and :term:`path-like object`. The cwd parameter now accepts a bytes object. Based on patch by Anders Lorentsen.

:class:`pathlib.Path.unlink` now accepts a missing_ok parameter to avoid a :exc:`FileNotFoundError` from being raised. Patch by Robert Buchholz.

Allow :class:`mmap.mmap` objects to access the madvise() system call (through :meth:`mmap.mmap.madvise`).

Added support for ZIP files with disks set to 0. Such files are commonly created by builtin tools on Windows when use ZIP64 extension. Patch by Francisco Facioni.

trace.py can now run modules via python3 -m trace -t --module module_name

Changed :func:`unittest.mock.patch.dict` to return the patched dictionary when used as context manager. Patch by Vadim Tsander.

Added a __copy__() to collections.UserList and collections.UserDict in order to correctly implement shallow copying of the objects. Patch by Bar Harel.

\r, \0 and \x1a (end-of-file on Windows) are now escaped in protocol 0 pickles of Unicode strings. This allows to load them without loss from files open in text mode in Python 2.

_thread.interrupt_main() now avoids setting the Python error status if the SIGINT signal is ignored or not handled by Python.

Clarify that some types have unstable constructor signature between Python versions.

Improve documentation of the stdin, stdout, and stderr arguments of the asyncio.subprocess_exec function to specify which values are supported. Also mention that decoding as text is not supported.

Add a few tests to verify that the various values passed to the std* arguments actually work.

Improve version added references in typing module - by Anthony Sottile.

What's new now mentions SSLContext.hostname_checks_common_name instead of SSLContext.host_flags.

Add a note to the curses.addstr() documentation to warn that multiline strings can cause segfaults because of an ncurses bug.

Added C API Documentation for Time_FromTimeAndFold and PyDateTime_FromDateAndTimeAndFold as per PEP 495. Patch by Edison Abahurire.

More of the legacy distutils documentation has been either pruned, or else more clearly marked as being retained solely until the setuptools documentation covers it independently.

Add detail to the documentation on the pty.spawn function.

Remove deprecation and document urllib.parse.unwrap(). Patch contributed by Rémi Lapeyre.

Added the context variable in glossary.

Clarify that copy() is not part of the MutableSequence ABC.

Make codecs.StreamRecoder.writelines take a list of bytes.

Added documentation for func factorial to indicate that returns integer values

Expand object.__doc__ (docstring) to make it clearer. Modify pydoc.py so that help(object) lists object methods (for other classes, help omits methods of the object base class.)

Modify test_coroutines, test_cprofile, test_generators, test_raise, test_ssl and test_yield_from to use :func:`test.support.catch_unraisable_exception` rather than :func:`test.support.captured_stderr`.

Fix test_memfd_create on older Linux Kernels.

Test with OpenSSL 1.1.1c

Add :func:`test.support.catch_unraisable_exception`: context manager catching unraisable exception using :func:`sys.unraisablehook`.

The main regrtest process now always removes all temporary directories of worker processes even if they crash or if they are killed on KeyboardInterrupt (CTRL+c).

"python3 -m test -jN ..." now continues the execution of next tests when a worker process crash (CHILD_ERROR state). Previously, the test suite stopped immediately. Use --failfast to stop at the first error.

Update Lib/test/selfsigned_pythontestdotnet.pem to match self-signed.pythontest.net's new TLS certificate.

Skip httplib and nntplib networking tests when they would otherwise fail due to a modern OS or distro with a default OpenSSL policy of rejecting connections to servers with weak certificates.

Add tests for several C API functions in the :mod:`datetime` module. Patch by Edison Abahurire.

Fix test_multiprocessing in test_venv if platform lacks functioning sem_open.

To embed Python into an application, a new --embed option must be passed to python3-config --libs --embed to get -lpython3.8 (link the application to libpython). To support both 3.8 and older, try python3-config --libs --embed first and fallback to python3-config --libs (without --embed) if the previous command fails.

Add a pkg-config python-3.8-embed module to embed Python into an application: pkg-config python-3.8-embed --libs includes -lpython3.8. To support both 3.8 and older, try pkg-config python-X.Y-embed --libs first and fallback to pkg-config python-X.Y --libs (without --embed) if the previous command fails (replace X.Y with the Python version).

On the other hand, pkg-config python3.8 --libs no longer contains -lpython3.8. C extensions must not be linked to libpython (except on Android, case handled by the script); this change is backward incompatible on purpose.

"make install" now runs compileall in parallel.

include of STATUS_CONTROL_C_EXIT without depending on MSC compiler

Update to OpenSSL 1.1.1b for Windows.

Add Windows support for UDP transports for the Proactor Event Loop. Patch by Adam Meily.

The :c:macro:`Py_DEPRECATED()` macro has been implemented for MSVC.

Support building Python on macOS without /usr/include installed. As of macOS 10.14, system header files are only available within an SDK provided by either the Command Line Tools or the Xcode app.

Replace now redundant .context_use_ps1 with .prompt_last_line. This finishes change started in bpo-31858.

Make idlelib.run runnable; add test clause.

Print any argument other than None or int passed to SystemExit or sys.exit().

When saving a file, call os.fsync() so bits are flushed to e.g. USB drive.

In browser.py, remove extraneous sorting by line number since dictionary was created in line number order.

Handle strings like u"bar" correctly in Tools/parser/unparse.py. Patch by Chih-Hsuan Yen.

Implement the PEP 587 "Python Initialization Configuration".

Fix crashes when attempting to use the modulo parameter when __ipow__ is implemented in C.

Update :c:func:`PyObject_CallMethodObjArgs` and _PyObject_CallMethodIdObjArgs to use _PyObject_GetMethod to avoid creating a bound method object in many cases. Patch by Michael J. Sullivan.

Implement PEP 590: Vectorcall: a fast calling protocol for CPython. This is a new protocol to optimize calls of custom callable objects.

Py_Main() now returns the exitcode rather than calling Py_Exit(exitcode) when calling PyErr_Print() if the current exception type is SystemExit.

Add new type flag Py_TPFLAGS_METHOD_DESCRIPTOR for objects behaving like unbound methods. These are objects supporting the optimization given by the LOAD_METHOD/CALL_METHOD opcodes. See PEP 590.

The :c:func:`!PyEval_ReInitThreads` function has been removed from the C API. It should not be called explicitly: use :c:func:`PyOS_AfterFork_Child` instead.