Introduction of a compression package

Both the zstd and zstandard import names are claimed by projects on PyPI. To avoid breaking users of one of the existing bindings, this PEP proposes introducing a new namespace for compression libraries, compression. This name is already reserved on PyPI for use in the standard library. The new Zstandard module will be compression.zstd. Other compression modules will be re-exported to the compression namespace and their current import names will be deprecated.

Providing a common namespace for compression modules has several advantages. First, it reduces user confusion about where to find compression modules. Second, the top level compression module could provide information on which compression formats are available, similar to hashlib’s algorithms_available. If PEP 775 is accepted, a compression.algorithms_guaranteed could be provided as well, listing zlib. Finally, a compression namespace prevents future issues with merging other compression formats into the standard library. New compression formats will likely be published to PyPI prior to integration into CPython. Therefore, any new compression format import name will likely already be claimed by the time a module would be considered for inclusion in CPython. Putting compression modules under a package prefix prevents issues with potential future name clashes.

Code that would like to remain compatible across Python versions may use the following pattern to ensure compatibility:

try:
    from compression.lzma import LZMAFile
except ImportError:
    from lzma import LZMAFile

This will use the newer import name when available and fall back to the old name otherwise.

Implementation based on pyzstd

The implementation for this PEP is based on the pyzstd project. This project was chosen as the code was originally written to be upstreamed to CPython by Ma Lin, who also wrote the output buffer implementation used in the standard library today. The project has since been taken over by Rogdham and is published to PyPI. The APIs in pyzstd are similar to the APIs for other compression modules in the standard library such as bz2 and lzma.

Minimum supported Zstandard version

The minimum supported Zstandard was chosen as v1.4.5, released in May of 2020. This version was chosen as a minimum based on reviewing the versions of Zstandard available in a number of Linux distribution package repositories, including LTS releases. This version choice is rather conservative to maximize compatibility with existing LTS Linux distributions, but a newer Zstandard version could likely be chosen given that newer Python releases are generally packaged as part of newer distribution releases.

The compression namespace

A new namespace for compression modules will be introduced named compression. The top-level module for this package will be empty to begin with, but a standard API for interacting with compression routines may be added in the future to the toplevel.

The compression.zstd module

A new module, compression.zstd will be introduced with Zstandard compression APIs that match other compression modules in the standard library, namely

• compress() / decompress() - APIs for one-shot compression or decompression
• ZstdFile / open() - APIs for interacting with streams and file-like objects
• ZstdCompressor / ZstdDecompressor - APIs for incremental compression or decompression

It will also contain some Zstandard-specific functionality:

• ZstdDict / train_dict() / finalize_dict() - APIs for interacting with Zstandard dictionaries, which are useful for compressing many small chunks of similar data

libzstd optional dependency

The libzstd library will become an optional dependency of CPython. If the library is not available, the compression.zstd module will be unavailable. This is handled automatically on Unix platforms as part of the normal build environment detection.

On Windows, libzstd will be added to the source dependencies used to build libraries CPython depends on for Windows.

Other compression modules

New import names compression.lzma, compression.bz2, and compression.zlib will be introduced in Python 3.14 re-exporting the contents of the existing lzma, bz2, and zlib modules respectively.

The _compression module, given that it is marked private, will be immediately renamed to compression._common.streams. The new name was selected due to the current contents of the module being I/O related helpers for stream APIs (e.g. LZMAFile) in standard library compression modules.

Compression module migration timeline

Existing modules will emit a DeprecationWarning in the Python release following the last Python without the compression module leaving support. For example, if the compression namespace is introduced in 3.14, then the DeprecationWarnings would be emitted in 3.19, the next release after 3.13 reaches end of life. These warnings would begin five years after the introduction of compression namespace. In accordance with PEP 387, in Python 3.24, five years after the DeprecationWarnings are added and ten years after the new compression namespace is introduced, the existing modules will be removed and code must use the compression sub-modules. The documentation for these modules will be updated to discuss the planned deprecation and removal timelines.

Name the module libzstd and do not make a new compression namespace

One option instead of making a new compression namespace would be to find a different name, such as libzstd, as the import name. However, the issue of existing import names is likely to persist for future compression formats added to the standard library. LZ4, a common high speed compression format, has a package on PyPI, lz4, with the import name lz4. Instead of solving this issue for each compression format, it is better to solve it once and for all by using the already-claimed compression namespace.