Background

Python’s Stable ABI (abi3 for short), as defined in PEP 384 and PEP 652, provides a way to compile extension modules that can be loaded on multiple minor versions of the CPython interpreter. Several projects use this to limit the number of wheels (binary artefacts) that need to be built and distributed for each release, and/or to make it easier to test with pre-release versions of Python.

With free-threading builds (PEP 703) being on track to eventually become the default (PEP 779), we need a way to make a Stable ABI available to those builds.

To build against a Stable ABI, the extension must use a Limited API, that is, only a subset of the functions, structures, etc. that CPython exposes. Both the Limited API and the current Stable ABI are versioned, and building against Stable ABI 3.X requires using only Limited API 3.X, and yields an extension that is ABI-compatible with CPython 3.X and any later version (though bugs in CPython sometimes cause incompatibilities in practice).

The Limited API is not “stable”: newer versions may remove API that were a part of older versions.

This PEP proposes additional API limitations, as required to be compatible with both GIL-enabled and free-threaded builds of CPython.

Ecosystem maintainers want decreased maintenance burden

A major advantage of the limited API and stable ABI wheels is that new Python versions are supported on the day of release. Without stable ABI wheels, maintainers are left with a choice between closely following CPython releases and producing wheels during CPython beta periods or dealing with inevitable user requests for support for new CPython versions.

Bindings generators

Both moocore and cryptography use bindings generators to interface with the C API. Cryptography uses PyO3 and CFFI while moocore uses only CFFI. Both CFFI and PyO3 already handle all the details of abstracting over the C API to enable different build configurations and there is no need to laboriously port extension types to use APIs that are only available on one build or another.

Using bindings generators will enable these projects to quickly adopt the new stable ABI. Initial testing using the experimental _Py_OPAQUE_PYOBJECT flag defined in CPython’s main branch, indicates that PyO3, CFFI, and Cython will all work with PEP 803 using packaging tools that have been patched to account for an abi3.abi3t tag.

PyO3 maintainer David Hewitt said in support of this proposal:

PyO3 greatly benefits from having a stable ABI - one of the biggest challenges for the framework is the need to abstract over a wide range of Python / OS / CPU / environment combinations. We also offer the possibility to build with the stable ABI for each of these environments (targeting a given minimum Python version’s stable ABI). The goal is always that all functionality PyO3 offers works the same on all these combinations (sometimes with a Python version floor to access certain features). We currently support Python 3.7+. All functionality added to the stable ABI is a very welcome promise that PyO3 will not need to introduce further conditional code to support a given feature. In the long run this makes it possible for PyO3 to simplify code paths as support for older Python versions is dropped, helping to keep maintenance burden under control.

When asked to comment about this proposal, Cython maintainer David Woods said:

Cython doesn’t have huge problems with the number of wheels we distribute because ultimately it works fine as pure-Python. We do distribute wheels for a few of the smaller platforms as Stable ABI wheels but that’s more “dogfooding” than because we actually need to. So I’m adding this in anticipation that other people will find it useful rather than because I will.

I do remain slightly concerned that the performance trade-offs for this will turn out to be too much for many Cython users (it’s possible that the trade-off may be different for other binding tools). That’s not a huge disaster since we’re not getting rid of the regular compilation mode so people are free to pick their own personal trade-offs.

Tag name

The tag abi3t is chosen to reflect the fact that this ABI is similar to abi3, with minimal changes necessary to support free-threading (which uses the letter t in existing, version-specific ABI tags like cp314t).

Stable ABI for free-threaded builds

Python will introduce a new stable ABI, called stable ABI for free-threading builds, or abi3t for short. As with the current Stable ABI (abi3), abi3t will be versioned using major (3) and minor versions of Python interpreter. Extensions built for abi3t 3.x will be compatible with CPython 3.x and above.

To build a C/C++ extension for abi3t, the extension will need to only use limited API for free-threaded builds. Like the existing Limited API, this will be a subset of the general CPython C API. Initially, it will also be a subset of the Limited API. We will strive to keep it as a subset, but will not guarantee this.

Since abi3t and abi3 will overlap, it will be possible for a single compiled extension to support both at once, and thus be compatible with CPython 3.15+ (both free-threaded and GIL-eanbled builds). Initially, any extension compiled for abi3t will be compatible with abi3 as well.

Choosing the target ABI

Users of the C API – or build tools acting on their behalf, configured by tool-specific UI – will select the target ABI using the following macros:

• Py_LIMITED_API=<version> (existing): Compile for abi3 of the given version.
• Py_TARGET_ABI3T=<version> (proposed here): Compile for abi3t of the given version.

These two macros are functionally very similar. In hindsight, Py_TARGET_ABI3 (without the T) would be a more fitting name for Py_LIMITED_API. We keep the existing name for backwards compatibility.

For ease of use and implementation simplicity, respectively, Python.h will set the configuration macros automatically in the following situations:

• If Py_LIMITED_API=v and Py_GIL_DISABLED is set, then Py_TARGET_ABI3T will be defined as v by default. (This allows choosing abi3t by defining the pre-existing macro and compiling with free-threaded CPython headers. Note that in CPython 3.14, this case results in a compile-time error.)
• If Py_TARGET_ABI3T=v is set, CPython may define or redefine Py_LIMITED_API as v. (This means that CPython can continue to use the Py_LIMITED_API macro internally to select which APIs are available.)

Opaque PyObject

abi3t will initially have a single diffenerce from abi3: the PyObject structure and APIs that depend on it are not part of abi3t.

Specifically, when building for abi3t, the CPython headers will:

• make the following structures opaque (or in C terminology, incomplete types):
  • PyObject
  • PyVarObject
  • PyModuleDef_Base
  • PyModuleDef
• no longer include the following macros:
  • PyObject_HEAD
  • _PyObject_EXTRA_INIT
  • PyObject_HEAD_INIT
  • PyObject_VAR_HEAD
  • Py_SET_TYPE()

In both the regular stable ABI (abit3 3.15+) and the new abi3t, the following will be exported functions (exposed in the ABI) rather than macros:

• Py_SIZE()
• Py_SET_SIZE()
• Py_IS_TYPE()

Runtime ABI checks

Users – or rather build/install tools acting on users’ behalf – will continue to be responsible for not putting incompatible extensions on Python’s import paths. This decision makes sense since tools typically have much richer metadata than what CPython can check. Typically, build tools and installers use PyPA packaging metadata and platform compatibility tags to communicate compatibility details, but other models are possible.

However, CPython will add a line of defense against outdated or misconfigured tools, or human mistakes, in the form of a new module slot containing basic ABI information. This information will be checked when a module is loaded, and incompatible extensions will be rejected. The specifics are left to the C API working group (see issue 72).

This slot will become mandatory with the new export hook added in PEP 793. (That PEP currently says “there are no required slots”; it will be updated.)

The abi3t wheel tag

Wheels that use a stable ABI compatible with free-threading CPython builds should use a new ABI tag: abi3t.

New API

Implementing this PEP will make it possible to build extensions that can be successfully loaded on free-threaded Python, but not necessarily ones that are thread-safe without a GIL.

Limited API to allow thread-safety without a GIL – presumably PyMutex, PyCriticalSection, and similar – will be added via the C API working group, or in a follow-up PEP.

Compatibility Overview

The following table summarizes compatibility of wheel tags with CPython interpreters. “GIL” stands for GIL-enabled interpreter; “FT” stands for a free-threaded one.

(*): Wheels with these tags cannot be built; see table below

The following table summarizes which wheel tag should be used for an extension built with a given interpreter and Py_LIMITED_API macro:

In the “Compile on” column, FT means that the Py_GIL_DISABLED macro must be defined – either explicitly or, on non-Windows platforms, by including CPython headers configured with --disable-gil. GIL means that Py_GIL_DISABLED must not be defined.

In the Py_LIMITED_API and Py_TARGET_ABI3T, a dash means the macro must not be defined; a version means the macro must be set to the corresponding integer in Py_PACK_VERSION() format.

Values in the Note column:

• existing: The wheel tag is currently in use
• continued: The wheel tag continues the existing scheme
• new: Proposed in this PEP.
• reserved: A mechanism to build a matching extension is not proposed in this PEP, but may be added in the future. Installers should be prepared to handle the tag.

Make PyObject opaque in Limited API 3.15

It would be possible to make PyObject struct opaque in Limited API 3.15 (that is, the new version of the existing Limited API), rather than introduce a new variant of the Stable ABI and Limited API.

This would mean that extension authors would need to adapt their code to the new limitations, or abandon Limited API altogether, in order to use any C API introduced in Python 3.15.

It would also not remove the need for a new wheel tag (abi3t), which would be required to express that an extension is compatible with both GIL-enabled and free-threaded builds of CPython 3.14 or lower.

Shims for compatibility with CPython 3.14

It’s possible to build a cp314-abi3.abi3t extension – one compatible with 3.14 (both free-threaded build and default). There are several challenges around this:

• making it convenient and safe for general extensions
• testing it (as CPython’s test suite doesn’t involve other CPython versions than the one being tested)

So, providing a mechanism to build such extensions is best suited to an external project (for example, one like pythoncapi-compat). It’s out of scope for CPython’s C API, and this PEP.

To sketch how such a mechanism could work:

The main issue that prevents compatibility with Python 3.14 is that with opaque PyObject and PyModuleDef, it is not feasible to initialize an extension module. The solution, PEP 793, is only being added in Python 3.15.

It is possible to work around this using the fact that the 3.14 ABIs (both free-threading and GIL-enabled) are “frozen”, so it is possible for an extension to query the running interpreter, and for 3.14, use a struct definition corresponding to the detected build’s PyModuleDef.

Naming this abi4

Instead of abi3t, we could “bump the version” and use abi4 instead. The difference is largely cosmetic.

However, one thing this PEP does not propose is changing the filename tag: extensions will be named with the extensions like .abi3.so. Changing this while keeping compatibility with GIL-enabled builds would be an unnecessary technical change.

Using abi3.abi4 in wheel tags but only .abi3 in filenames would look more inconsistent than abi3.abi3t and .abi3.

If we added an abi4 tag, the Py_LIMITED_API value would either need to:

• change to start with 4 to match abi4, but no longer correspond to PY_VERSION_HEX (making it harder to generate and check), or
• not change, making it inconsistent with abi4.

Adding abi3t is a smaller change than adding abi4, making it work better as a transitional state before larger changes like PEP 809’s abi2026.

Reusing Py_GIL_DISABLED to enable the new ABI

It would be possible to select abi3t (rather than abi3) when the Py_GIL_DISABLED macro is defined together with Py_LIMITED_API.

This would require annoying fiddling with build flags, and make it especially cumbersome to target both abi3 and abi3t at the same time.

Making the new version of abi3 compatible with free-threading

It would be possible to make PyObject opaque in Limited API 3.15, rather than add a new stable ABI. This would make all extensions built for the Stable ABI 3.15 and above compatible with both free-threading and GIL-enabled Python.

In the PEP discussion, the ability to build for the GIL-only Stable ABI with no source changes was deemed to be worth an extra configuration macro (Py_TARGET_ABI3T).