Adding sys.abi_features to the Python Standard Library

The Python standard library already has mechanisms to provide information about the ABI of the interpreter, like the sys.abiflags attribute and the sysconfig module, but these do not provide all the ABI Features that are relevant for environment markers. The new attribute sys.abi_features is a natural extension to the existing mechanisms, and it is a simple and intuitive way to provide this information, including the information from sys.abiflags.

A Forward Looking View on Free-Threaded Python

PEP 703, the accepted proposal for free threading, states that the rollout of free-threaded Python should be gradual, which has been clarified by the Python Steering Council in the PEP 703 acceptance post to mean a three stage process over multiple releases. It is therefore important to make sure that the mechanisms in this PEP are useable for Python interpreters where either free-threading or non-free-threading could be the default or the only option.

At the time of writing, free-threaded Python is in Phase I: experimental phase. In this phase, there is an acute need for the proposed environment markers to help with the transition to free-threaded Python as package authors gradually add support.

As the number of packages with support increases, and particularly during Phase II: Supported-but-not-default phase, we still anticipate a strong need for the environment markers to help with the transition.

As free-threaded Python enters into Phase III: Default phase, the need for the environment markers will decrease, though at this point it is not clear that the GIL-enabled Python will be completely phased out (it may remain available as a non standard build option). If it persists, the inverse need for the ABI feature detection may arise.

Indeed, in all three phases it may be necessary for package authors to choose specific versions of their dependencies based on the ABI features, with a shift from GIL-enabled as default to free-threading as default over time.

The ABI features are designed with this in mind to guarantee usefulness and simplicity for the foreseeable future in a changing Python ecosystem.

Relation to Other PEPs

This PEP extends environment markers with set semantics for ABI features. PEP 751 includes a similar extension for lock file specific environment markers; although the two have been developed indepedently, they are compatible where they overlap in terms of the new set semantics.

ABI Features

ABI features are intrinsic properties of the Python interpreter, expressed as simple, understandable strings. However, not all features are equally applicable to all Python interpreters or Python versions. For example, the distinction between free-threaded and GIL-enabled interpreters is only relevant for CPython 3.13 onwards, but the bitness of the interpreter is relevant for all interpreters.

All interpreters MUST handle the following ABI features as stated. ABI features that are restricted to particular interpreters MUST NOT be provided by other interpreters. The features are subdivided into groups and for each group there MUST be exactly one feature present, except when the group is marked as optional, in which case there MUST be at most one feature present.

free-threading or gil-enabled (only CPython)

If the Python interpreter is free-threaded, the free-threading feature MUST be present and the gil-enabled feature MUST NOT be present. Otherwise, the gil-enabled feature MUST be present and the free-threading feature MUST NOT be present.

debug (only CPython, optional)

This ABI feature is reserved for the --with-pydebug build of CPython. If the interpreter is a CPython interpreter with Py_DEBUG capabilities, the debug feature MUST be present. On POSIX systems, this corresponds to "d" in sys.abiflags.

32-bit or 64-bit (optional)

The bitness of the interpreter, that is, whether it is a 32-bit or 64-bit build [2]. If the bitness is unknown or neither 32-bit nor 64-bit, this feature MUST NOT be present.

The sys.abi_features Attribute

Making the ABI features available in an easily accessible, expressive, standardized way is useful beyond the scope of environment markers. For example, "32-bit" in sys.abi_features is much more expressive than the current standard test of comparing sys.maxsize with 2**32, which can be found more than ten thousand times on GitHub. If one wants to determine whether the interpreter is a debug build, there is currently no standardized, cross platform way to do so, and sys.abiflags is not available on Windows. Hence, the ABI Features listed above are added to the Python standard library.

Since they are all the result of compile time choices describing basic features of the interpreter, the most intuitive place to put them is in sys. Since there is no intrinsic order, nor a possibility for duplication, they are added as a frozenset of strings.

All Python interpreters MUST provide the sys.abi_features attribute as a frozenset of strings, which MUST contain only the ABI Features that are defined in this PEP or in a subsequent PEP.

An example value would be sys.abi_features == frozenset({"free-threading", "debug", "32-bit"}) on a free-threaded debug build for win32.

The sys_abi_features Environment Marker

To make ABI features available in dependency specifications, a new environment marker variable, sys_abi_features, is added to the format of dependency specifiers with the same semantics as the sys.abi_features attribute proposed above.

To do this, we need to extend the grammar laid out in PEP 508 and maintained in the Dependency specifiers and document the possible values.

The grammar is extended to include the sys_abi_features marker variable by augmenting the definition of env_var as follows:

env_var       = ('python_version' | 'python_full_version' |
                 'os_name' | 'sys_platform' | 'platform_release' |
                 'platform_system' | 'platform_version' |
                 'platform_machine' | 'platform_python_implementation' |
                 'implementation_name' | 'implementation_version' |
                 'sys_abi_features' |
                 'extra' # ONLY when defined by a containing layer
                 )

Like the grammar, also the overview table of environment markers in Dependency specifiers is augmented to add the following row:

With these additions, ABI features can be used in dependency specifications via the in operator to test for the presence of a feature, or the not in operator to test for the absence of a feature.

Note that the presence of sys.abi_features in the Python standard library makes implementation particularly easy for new Python versions, but its absence in older versions does not prevent the implementation of the new environment markers, as demonstrated in the Reference Implementation.

Require Cython Pre-release for Free-Threaded Python

To require a pre-release of Cython only for a free-threaded Python interpreter, the following dependency specification can be used:

cython >3.1.0a1; "free-threading" in sys_abi_features
cython ==3.0.*; "free-threading" not in sys_abi_features

Require SciPy Unless on 32-bit win32

To require SciPy unless on a 32-bit interpreter on Windows, the following dependency specification can be used:

scipy; platform_system != "Windows" or "32-bit" not in sys_abi_features

Require NumPy for a Free-Threaded Interpreter With Debugging Capabilities

To require NumPy only for a free-threaded interpreter with debugging capabilities, the following dependency can be used:

numpy; "free-threading" in sys_abi_features and "debug" in sys_abi_features

Audit and Update Tools

A wide range of tools understand Python dependency data as expressed in requirements.txt files. (e.g., Dependabot, Tidelift, etc)

Such tools inspect dependency data and, in some cases, offer tool-assisted or fully automated updates. It is our expectation that no such tools would support the new environment markers at first, and broad ecosystem support could take many months or even some number of years to arrive.

As a result, users of the new environment markers would experience a degradation in their workflows and tool support at the time that they start using them. This is true of any new standard for where and how dependency data are encoded.

Extension Mechanism

In an early discussion of the topic (Environment marker for free-threading), the idea of a general extension mechanism for environment markers was brought up. While it is appealing to forego a whole PEP process should the need for new environment markers arise in the future, there are two main challenges. First, a completely dynamic mechanism would present difficulties for tools that rely on static analysis of dependency specifications.

This means that even if a dynamic mechanism were to be adopted, new environment markers would likely still need to be spelled out in a PEP.

Second, the introduction of a dynamic mechanism would require a more complex implementation in the packaging library, which would be a significant departure from the current approach.

Having said that, the new sys.abi_features attribute provides a natural extension point for any new ABI features, even if specific to a subset of interpreters, should the need arise to add such new features with a subsequent PEP.

Other Environment Markers

If other environment markers are needed right now, this PEP could be extended to include them.

Other Tooling

The reference implementation is based on the packaging library and pip. We have confirmed that this allows for building and installing packages with several build backends. It is possible that other tools should be added to the reference implementation.