Implement PEP 784 - Adding Zstandard to the Python standard library

[emmatyping]

Feature or enhancement

This is a tracking issue for implementing PEP 784. See the PEP text for more details.

Since the diff is significant (~10k lines) I wanted to split up the PRs a bit.

Implementation Plan:

• Add compression module just re-exporting existing compression modules. Move the _compression module.
• Add _zstd native module with Unix build config
• Add Windows build config for _zstd (blocked on adding libzstd to cpython-source-deps) and SBOM config.
• Add zstd Python module with tests
• add NEWS/What's New section
• Add documentation for zstd
• Decide on whether or not to check parameter types for options
• Verify style conformance of C and Python code
• Refactor _train_dict and _finalize_dict to share common code
• Refactor critical sections to follow style of having lock_held functions
• Improve error message when options value can't be converted to int

Linked PRs

• gh-132983: Introduce compression package and move _compression module #133018
• gh-132983: Introduce _zstd bindings module #133027
• gh-132983: Install compression package contents #133063
• gh-132983: make compression._common a full-fledged package #133076
• gh-132983: Convert compression._common to a package #133086
• gh-132983: [DRAFT] Test _zstd code with Python test suite #133185
• gh-132983: [Draft] Run refleak tests against _zstd with tests #133282
• gh-132983: Add compression.zstd and Python tests #133365
• GH-132983: Build _zstd on Windows #133366
• GH-132983: Use a standard autoconf version check for zstd #133479
• gh-132983: Add What's New entry for PEP 784 implementation #133495
• GH-132983: Remove zstd version check in the header file #133502
• gh-132983: Remove pyzstd in identifiers #133535
• gh-132983: Style improvements for compression.zstd #133547
• GH-132983: Support finding libzstd without pkg-config #133550
• GH-132983: Restore libzstd fallback detection #133565
• [3.14] gh-132983: Remove pyzstd in identifiers (GH-133535) #133629
• gh-132983: Clean-ups for _zstd #133670
• gh-103092: Support subinterpreters in _zstd #133674
• GH-132983: Remove subclassing support from zstd types #133694
• [3.14] gh-103092: Support subinterpreters in _zstd (GH-133674) #133695
• gh-132983: Fix small issues with zstd support in zipfile #133723
• gh-132983: Don't allow trailer data in ZstdFile #133736
• [3.14] gh-132983: Clean-ups for _zstd (GH-133670) #133756
• [3.14] GH-132983: Restore libzstd fallback detection (GH-133565) #133757
• [3.14] GH-132983: Remove subclassing support from zstd types (GH-133694) #133762
• gh-132983: Simplify _zstd_exec() #133775
• gh-132983: Make zstd types immutable #133784
• GH-132983: remove empty_bytes from _zstd module state #133785
• [3.14] gh-132983: Simplify _zstd_exec() (GH-133775) #133786
• [3.14] GH-132983: remove empty_bytes from _zstd module state (GH-133785) #133788
• GH-132983: PEP 7 and Argument Clinic changes for zstd #133791
• [3.14] GH-132983: PEP 7 and Argument Clinic changes for zstd (GH-133791) #133792
• gh-132983: Reduce the size of _zstdmodule.h #133793
• [3.14] gh-132983: Don't allow trailer data in ZstdFile (GH-133736) #133799
• [3.14] gh-132983: Reduce the size of _zstdmodule.h (GH-133793) #133854
• gh-132983: Remove EndlessZstdDecompressor remains #133856
• [3.14] gh-132983: Make zstd types immutable (GH-133784) #133857
• [3.14] gh-132983: Remove leftovers from EndlessZstdDecompressor (GH-133856) #133859
• gh-132983: Convert zstd __new__ methods to Argument Clinic #133860
• gh-132983: Add documentation for compression.zstd #133911
• [3.14] gh-132983: Convert zstd __new__ methods to Argument Clinic (GH-133860) #133915
• gh-132983: Split _zstd_set_c_parameters #133921
• gh-132983: Convert dict_content to take Py_buffer #133924
• gh-132983: Fix zstd compiler warning about unused function mt_continue_should_break() #133947
• [3.14] gh-132983: Fix compiler warning about unused function mt_continue_should_break() (GH-133947) #133950
• gh-132983: Call Py_XDECREF rather than PyObject_GC_Del in failed __new__ #133962
• [3.14] gh-132983: Fix small issues with zstd support in zipfile (GH-133723) #133974
• [3.14] gh-132983: Style improvements for compression.zstd (GH-133547) #134001

[picnixz]

It's something I should have asked on discourse, but is compression the final name of the public module? I see in the PEP that we want to mimic hashlib.algorithms_guaranteed as well and one way to mitigate the name shadowing is to actually use compressionlib as the name (we have hashlib, annotationlib, and we could have compressionlib as well; I also think compressionlib is a less likely name to be chosen but it's not the best one as well). Feel free to ignore this question as the PEP has been accepted.

[emmatyping]

It's something I should have asked on discourse, but is compression the final name of the public module?

Yes, the decision was to use compression. A search of Github code showed that it was used very infrequently. Literally one in a million files https://discuss.python.org/t/pep-784-adding-zstandard-to-the-standard-library/87377/66