Following system colour scheme Selected dark colour scheme Selected light colour scheme

Python Enhancement Proposals

PEP 689 – Unstable C API tier

Author:
Petr Viktorin <encukou at gmail.com>
Discussions-To:
Python-Dev thread
Status:
Deferred
Type:
Standards Track
Requires:
523
Created:
22-Apr-2022
Python-Version:
3.12
Post-History:
27-Apr-2022

Table of Contents

Deferral note

This PEP was accepted provided that function names in the unstable API start with leading underscores. Unfortunately, this uncovered wider disagreement about the meaning of leading underscores. The PEP’s author is not comfortable pushing the PEP forward before we agree on what a leading underscore means.

Abstract

Some functions and types of the C-API are designated unstable, meaning that they will not change in patch (bugfix/security) releases, but may change between minor releases (e.g. between 3.11 and 3.12) without deprecation warnings.

Motivation & Rationale

The Python C-API is currently divided into three tiers:

  • Limited API, with high compatibility expectations
  • Public API, which follows the backwards compatibility policy, and requires deprecation warnings before changes
  • Private (internal) API, which can change at any time.

Tools requring access to CPython internals (e.g. advanced debuggers and JIT compilers) are often built for minor series releases of CPython, and assume that the C-API internals used do not change in patch releases. To support these tools, we need a tier between the Public and Private C-API, with guarantees on stability throughout the minor-series release.

Setting Stability Expectations

Currently, there are no guarantees for the internal API – that is, anything that requires Py_BUILD_CORE or is named with a leading underscore. This API can change without warning at any time, and code that uses it is pinned to a specific build of Python.

However, in practice, even the internal API usually happens to be stable in patch releases:

  • Some CPython core developers take this as an an unwritten rule.
  • Patch releases only contain bugfixes, which are unlikely to change the API.

Unstable API will make the stability expectations more explicit.

It will also hopefully encourage existing users of the private API to reach out to python-dev, so we can expose, standardize and test an API for some of their use cases.

Reserving underscores for Private API

PEP 523 introduced functions for use by debuggers and JIT compilers, which are stable only across minor releases. The functions names have leading underscores to suggest their limited stability.

However, leading underscores usually mark fully private API. CPython developers familiar with the “underscore means internal” convention are unlikely to check if underscored functions they are changing are documented and used outside CPython itself.

This proposal brings us a bit closer to reserving underscores only for truly internal, private, hands-off API.

Warning about API that is changed often

The PyCode_New() family is an example of functions that are documented as unstable (“Calling [it] directly can bind you to a precise Python version”), and also often change in practice.

Moving it to the unstable tier will make its status obvious even to people who don’t read the docs carefully enough, and will make it hard to use accidentally.

Changes during the Beta period

Since the API itself can change continuously up until Beta 1 (feature freeze) of a minor version, major users of this API are unlikely to test Alpha releases and provide feedback. It is very difficult to determine what needs to be exposed as unstable.

Additions to the unstable tier will count as stabilization, and will be allowed up to Release Candidate 1.

Specification

Several functions and types (“APIs”) will be moved to a new unstable tier.

They will be expected to stay stable across patch releases, but may change or be removed without warning in minor releases (3.x.0), including Alpha and Beta releases of 3.x.0.

When they change significantly, code that uses them should no longer compile (e.g. arguments should be added/removed, or a function should be renamed, but the semantic meaning of an argument should not change).

Their definitions will be moved to a new directory, Include/unstable/, and will be included from Python.h.

From Python 3.12 on, these APIs will only be usable when the Py_USING_UNSTABLE_API macro is defined. CPython will only define the macro for building CPython itself (Py_BUILD_CORE).

To make transition to unstable API easier, in Python 3.11 the APIs will be available without Py_USING_UNSTABLE_API defined. In this case, using them will generate a deprecation warning on compilers that support Py_DEPRECATED.

A similar deprecation period will be used when making more APIs unstable in the future:

  • When moving from public API, the deprecation period should follow Python’s backwards compatibility policy (currently, it should last at least two releases).
  • When moving from public API that is documented as unstable, the deprecation period can only last one release.
  • When moving from private API or adding new API, no deprecation period is necessary.

Leading underscores will be removed from the names of the moved APIs. The old underscored name of a renamed API will be available (as an alias using #define) at least until that API changes.

The unstable C-API tier and Py_USING_UNSTABLE_API will be documented, and documentation of each unstable API will be updated.

Adjustments during Beta periods

New APIs can be added to the unstable tier, and private APIs can be moved to it, up to the first release candidate of a new minor version. Consensus on the capi-sig or python-dev is needed in the Beta period.

In the Beta period, no API may be moved to more private tier, e.g. what is public in Beta 1 must stay public until the final release.

Initial unstable API

The following API will initially be unstable. The set may be adjusted for 3.11.

Code object constructors:

  • PyCode_New()
  • PyCode_NewWithPosOnlyArgs()

Frame evaluation API (PEP 523):

  • _PyFrameEvalFunction
  • _PyInterpreterState_GetEvalFrameFunc()
  • _PyInterpreterState_SetEvalFrameFunc()
  • _PyEval_RequestCodeExtraIndex()
  • _PyCode_GetExtra()
  • _PyCode_SetExtra()
  • struct _PyInterpreterFrame (as an incomplete, opaque struct)
  • _PyFrame_GetFrameObject
  • PyEval_EvalFrameDefault (new function that calls _PyEval_EvalFrameDefault, but takes PyFrameObject rather than _PyInterpreterFrame)

(Leading underscores will be removed as mentioned above.)

Backwards Compatibility

The C API backwards compatibility story will be made clearer.

How to Teach This

The changes affect advanced C programmers, who should consult the updated reference documentation, devguide and/or What’s New document·.

Reference Implementation

https://github.com/python/cpython/issues/91744

Rejected Ideas

It might be good to add a similar tier in the Python (not C) API, e.g. for types.CodeType. However, the opt-in mechanism would need to be different (if any). This is outside the scope of the PEP.

Open Issues

The exact set of exposed API may change.


Source: https://github.com/python/peps/blob/main/pep-0689.rst

Last modified: 2022-08-24 22:39:36 GMT