Python releases enforce C API changes

Every Python 3.x release has a long list of C API changes, including incompatible changes. C extensions have to be updated to work on the newly released Python.

Some incompatible changes are driven by new features: they cannot be avoided, unless we decide to not add these features. Other reasons:

• Remove deprecated API (see PEP 387).
• Ease the implementation of another change.
• Change or remove error-prone API.

Currently, there is no middle ground between “not change the C API” and “incompatible C API changes impact everybody”. Either a C extension is updated or the new Python version cannot be used. Such all-or-nothing deal does not satisfy C extension maintainers nor C extensions users.

Limited C API

The limited C API is versioned: the Py_LIMITED_API macro can be set to a Python version to select which API is available. On the Python side, it allows introducing incompatible changes at a specific Py_LIMITED_API version. For example, if Py_LIMITED_API is set to Python 3.11 or newer, the <stdio.h> is no longer included by Python.h, whereas C extensions targeting Python 3.10 are not affected.

The difference here is that upgrading Python does not change if <stdio.h> is included or not, but updating Py_LIMITED_API does. Updating Py_LIMITED_API is an deliberate action made by the C extension maintainer. It gives more freedom to decide when the maintainer is ready to deal with the latest batch of incompatible changes.

A similar version can be used with the regular (non-limited) C API.

Deprecation and compiler warnings

Deprecated functions are marked with Py_DEPRECATED(). Using a deprecated function emits a compiler warning.

The problem is that pip and build tools hide compiler logs by default, unless a build fails. Moreover, it’s easy to miss a single warning in the middle of hundred lines of logs.

Schedule changes

Currently, there is no way to schedule a C API change: announce it but also provide a way to maintainers to test their C extensions with the change. Either a change is not made, or everybody must update their code if they want to update Python.

New macros

Add new Py_COMPAT_API_VERSION and Py_COMPAT_API_VERSION_MAX macros. They can be set to test if a C extension is prepared for future C API changes: compatible with future Python versions.

The Py_COMPAT_API_VERSION macro can be set to a specific Python version. For example, Py_COMPAT_API_VERSION=0x030e0000 tests C API changes scheduled in Python 3.14.

If the Py_COMPAT_API_VERSION macro is set to Py_COMPAT_API_VERSION_MAX, all scheduled C API changes are tested at once.

If the Py_COMPAT_API_VERSION macro is not set, it is to PY_VERSION_HEX by default.

The Py_COMPAT_API_VERSION macro can be set in a single C file or for a whole project in compiler flags. The macro does not affected other projects or Python itself.

Example in Python

For example, the PyImport_ImportModuleNoBlock() function is deprecated in Python 3.13 and scheduled for removal in Python 3.15. The function can be declared in the Python C API with the following declaration:

#if Py_COMPAT_API_VERSION < 0x030f0000
Py_DEPRECATED(3.13) PyAPI_FUNC(PyObject *) PyImport_ImportModuleNoBlock(
    const char *name            /* UTF-8 encoded string */
    );
#endif

If if Py_COMPAT_API_VERSION is equal to or greater than Python 3.15 (0x030f0000), the PyImport_ImportModuleNoBlock() function is not declared, and so using it fails with a build error.

Goals

• Reduce the number of C API changes affecting C extensions when updating Python.
• When testing C extensions (for example, optional CI test), Py_COMPAT_API_VERSION can be set to Py_COMPAT_API_VERSION_MAX to detect future incompatibilities. For mandatory tests, it is recommended to set Py_COMPAT_API_VERSION to a specific Python version.
• For core developers, make sure that the C API can still evolve without being afraid of breaking an unknown number of C extensions.

Non-goals

• Freeze the API forever: this is not the stable ABI. For example, deprecated functions will continue to be removed on a regular basis.
• C extensions maintainers not using Py_COMPAT_API_VERSION will still be affected by C API changes when updating Python.
• Provide a stable ABI: the macro only impacts the regular (non-limited) API.
• Silver bullet solving all C API issues.