Background

Python’s Stable ABI, as defined in PEP 384 and PEP 652, provides a way to compile extension modules that can be loaded on multiple minor versions of the CPython interpreter. Several projects use this to limit the number of wheels (binary artefacts) that need to be built and distributed for each release, and/or to make it easier to test with pre-release versions of Python.

With free-threading builds (PEP 703) being on track to eventually become the default (PEP 779), we need a way to make the Stable ABI available to those builds.

To build against the Stable ABI, the extension must use a Limited API, that is, only a subset of the functions, structures, etc. that CPython exposes. The Limited API is versioned, and building against Limited API 3.X yields an extension that is ABI-compatible with CPython 3.X and any later version (though bugs in CPython sometimes cause incompatibilities in practice). Also, the Limited API is not “stable”: newer versions may remove API that were a part of older versions.

This PEP proposes the most significant such removal to date.

Opaque PyObject

Version 3.15 of the Limited API will:

• make the following structures opaque (or in C terminology, incomplete types):
  • PyObject
  • PyVarObject
  • PyModuleDef_Base
  • PyModuleDef
• no longer include the following macros:
  • PyObject_HEAD
  • _PyObject_EXTRA_INIT
  • PyObject_HEAD_INIT
  • PyObject_VAR_HEAD
• no longer include these function-like macros:
  • Py_SIZE()
  • Py_SET_SIZE()
  • Py_SET_TYPE()

New Export Hook (PEP 793)

Implementation of this PEP requires PEP 793 (PyModExport: A new entry point for C extension modules) to be accepted, providing a new “export hook” for defining extension modules. Using the new hook will become mandatory in Limited API 3.15.

Runtime ABI checks

Users – or rather the tools they use for building and installing extensions – will continue to be responsible for not putting incompatible extensions on Python’s import paths. This decision makes sense since tools typically have much richer metadata than what CPython can check.

However, CPython will add a line of defense against outdated or misconfigured tools, or human mistakes, in the form of a new module slot containing basic ABI information. This information will be checked when a module is loaded, and incompatible extensions will be rejected. The specifics are left to the C API working group (see issue 72).

This slot will become mandatory with the new export hook added in PEP 793. (That PEP currently says “there are no required slots”; it will be updated.)

Wheel tags

PyPA build tools should not need changes: if they allow the user to set the limited API version, setting it to 3.15 should define Py_LIMITED_API to 0x030f0000. The resulting wheel should be tagged with the Python-ABI tag cp315-abi3.

Installers and other tools should assume that wheels with a python tag cp315 and above and ABI tag abi3 are compatible with free-threaded builds.

New API

Implementing this PEP will make it possible to build extensions that can be successfully loaded on free-threaded Python, but not necessarily ones that are thread-safe without a GIL.

Limited API to allow thread-safety without a GIL – presumably PyMutex, PyCriticalSection, and similar – will be added via the C API working group, or in a follow-up PEP.

Add an alternative stable ABI for free-threading

It would be possible to:

• Keep the current stable ABI (“abi3”) unchanged (except additions, as done in each release). Extensions would need no code changes and builds would be compatible with old and new GIL-enabled CPython versions.
• Add a new stable ABI (“abi3t”) specifically for free-threading. Extensions would need no code changes and builds would be compatible with free-threaded CPython (3.14 and above).
• Defining an additional macro (“Py_OPAQUE_PYOBJECT”) would make PyObject opaque as in this PEP. Extensions would need code changes as in this PEP, and compiled extensions (“abi3.abi3t”) would be compatible with all builds of CPython 3.15+.

This scheme was rejected as too complex. It would also make the free-threading memory layout of PyObject part of the stable ABI, preventing future adjustments.

Shim for compatibility with CPython 3.14

The issue that prevents compatibility with Python 3.14 is that with opaque PyObject and PyModuleDef, it is not feasible to initialize an extension module. The solution, PEP 793, is only being added in Python 3.15.

It is possible to work around this using the fact that the 3.14 ABIs (both free-threading and GIL-enabled) are “frozen”, so it is possible for an extension to query the running interpreter, and for 3.14, use a struct definition corresponding to the detected build’s PyModuleDef.

This is too onerous to support and test in CPython’s Limited API. It would also require adding a new wheel tag (e.g. abi3t) that all install tools would need to recognize. (This PEP’s cp315-abi3 is incompatible with Python 3.14.)