Background

Python’s Stable ABI, as defined in PEP 384 and PEP 652, provides a way to compile extension modules that can be loaded on multiple minor versions of the CPython interpreter. Several projects use this to limit the number of wheels (binary artefacts) that need to be built and distributed for each release, and/or to make it easier to test with pre-release versions of Python.

With free-threading builds (PEP 703) being on track to eventually become the default (PEP 779), we need a way to make the Stable ABI available to those builds.

To build against the Stable ABI, the extension must use a Limited API, that is, only a subset of the functions, structures, etc. that CPython exposes. The Limited API is versioned, and building against Limited API 3.X yields an extension that is ABI-compatible with CPython 3.X and any later version (though bugs in CPython sometimes cause incompatibilities in practice). Also, the Limited API is not “stable”: newer versions may remove API that were a part of older versions.

This PEP proposes the most significant such removal to date.

Opaque PyObject

Version 3.15 of the Limited API will:

• make the following structures opaque (or in C terminology, incomplete types):
  • PyObject
  • PyVarObject
  • PyModuleDef_Base
  • PyModuleDef
• no longer include the following macros:
  • PyObject_HEAD
  • _PyObject_EXTRA_INIT
  • PyObject_HEAD_INIT
  • PyObject_VAR_HEAD
  • Py_SET_TYPE()
• export the following as functions in the ABI, rather than macros:
  • Py_SIZE()
  • Py_SET_SIZE()

New Export Hook (PEP 793)

Implementation of this PEP requires PEP 793 (PyModExport): A new entry point for C extension modules) to be accepted, providing a new “export hook” for defining extension modules. Using the new hook will become mandatory in Limited API 3.15.

Runtime ABI checks

Users – or rather the tools they use for building and installing extensions – will continue to be responsible for not putting incompatible extensions on Python’s import paths. This decision makes sense since tools typically have much richer metadata than what CPython can check.

However, CPython will add a line of defense against outdated or misconfigured tools, or human mistakes, in the form of a new module slot containing basic ABI information. This information will be checked when a module is loaded, and incompatible extensions will be rejected. The specifics are left to the C API working group (see issue 72).

This slot will become mandatory with the new export hook added in PEP 793. (That PEP currently says “there are no required slots”; it will be updated.)

The abi3t wheel tag

Wheels that use a stable ABI compatible with free-threading CPython builds should use a new ABI tag: abi3t. The name is chosen to reflect the fact that this ABI is similar to abi3, except changes necessary to support free-threading (which uses the letter t in existing, version-specific ABI tags like cp314t).

Since wheels built using Limited API 3.15 will be compatible with both GIL-enabled builds and free-threaded ones, they should use the compressed ABI tag set abi3.abi3t.

New API

Implementing this PEP will make it possible to build extensions that can be successfully loaded on free-threaded Python, but not necessarily ones that are thread-safe without a GIL.

Limited API to allow thread-safety without a GIL – presumably PyMutex, PyCriticalSection, and similar – will be added via the C API working group, or in a follow-up PEP.

Compatibility Overview

The following table summarizes compatibility of wheel tags with CPython interpreters. “GIL” stands for GIL-enabled interpreter; “FT” stands for a free-threaded one.

The following table summarizes which wheel tag should be used for an extension built with a given interpreter and Py_LIMITED_API macro:

Values in the Note column:

• existing: The wheel tag is currently in use
• continued: The wheel tag continues the existing scheme
• discontinued: The wheel tag continues the existing scheme, but it will be discouraged. Older tools may still generate it.
• new: Proposed in this PEP.

Add an alternative stable ABI for free-threading

It would be possible to:

• Add a new stable ABI (“abi3t”) specifically for free-threading, which would be incompatible with the existing abi3. Extensions would need no code changes to target abi3t and builds would be compatible with free-threaded CPython (3.14 and above).
• Define an additional macro (“Py_OPAQUE_PYOBJECT”), which would make PyObject opaque as in this PEP. Extensions would need code changes as in this PEP, and as in this PEP, compiled extensions (“abi3.abi3t”) would be compatible with all builds of CPython 3.15+.

This scheme was rejected as too complex. It would also make the free-threading memory layout of PyObject part of the stable ABI, preventing future adjustments.

Shims for compatibility with CPython 3.14

The main issue that prevents compatibility with Python 3.14 is that with opaque PyObject and PyModuleDef, it is not feasible to initialize an extension module. The solution, PEP 793, is only being added in Python 3.15.

It is possible to work around this using the fact that the 3.14 ABIs (both free-threading and GIL-enabled) are “frozen”, so it is possible for an extension to query the running interpreter, and for 3.14, use a struct definition corresponding to the detected build’s PyModuleDef.

This is too onerous to support and test in CPython’s Limited API at this point, but it may be allowed in the future.

Using the Python wheel tag to determine compatibility

A previous version of this PEP avoided adding a new wheel tag (abi3t), and specified that wheels tagged abi3 would be compatible with free-threading if the Python tag is cp315 or higher.

Such a scheme would work for this PEP, but it cannot express that an extension is compatible with both GIL-enabled and free-threaded builds of CPython 3.14 or lower. Adding a new explicit tag means that if we allow building such wheels in the future, packaging tools should not need additional changes to support them. They would be tagged cp314-abi3.abi3t.

Adding an abi4 wheel tag

Instead of abi3t, we could “bump the version” and use abi4 instead as the wheel ABI tag. In the wheel tag, the difference is largely cosmetic.

However, one thing this PEP does not propose is changing the filename tag: extensions will be named with the extensions like .abi3.so. Changing this while keeping compatibility with GIL-enabled builds would be an unnecessary technical change.

Using abi3.abi4 in wheel tags but only .abi3 in filenames would look more inconsistent than abi3.abi3t and .abi3.