Foreign threads hang during interpreter finalization

Many large libraries might need to call Python code in highly asynchronous situations where the desired interpreter may be finalizing or has already been finalized, but want to continue running code after invoking the interpreter. This desire has been brought up by users. For example, a callback that wants to call Python code might be invoked when:

1. A kernel has finished running on a GPU.
2. A network packet was received.
3. A thread has quit, and a native library is executing static finalizers for thread-local storage.

Generally, this pattern would look something like this:

static void
some_callback(void *arg)
{
    /* Do some work */
    /* ... */

    PyGILState_STATE gstate = PyGILState_Ensure();
    /* Invoke the C API to do some computation */
    PyGILState_Release(gstate);

    /* ... */
}

This comes with a hidden problem. If the target interpreter is finalizing, the current thread will hang! Or, if the target interpreter has been completely deleted (such as with short-lived subinterpreters), then attaching will likely result in a crash.

There are currently a few workarounds for this:

1. Leak resources to prevent the need to invoke Python.
2. Protect against finalization using an atexit callback.

These options generally work, but can be verbose, complex, or result in other issues. Ideally, interpreter finalization should not be such a footgun when working with Python’s C API.

Interpreter guards

type PyInterpreterGuard

An opaque interpreter guard structure.

By holding an interpreter guard, the caller can ensure that the interpreter will not finalize until the guard is closed (through PyInterpreterGuard_Close()).

This is similar to a “readers-writers” lock; threads may concurrently guard an interpreter, and the interpreter will have to wait until all threads have closed their guards before it can enter finalization. After finalization has started, threads are forever unable to acquire guards for that interpreter.

PyInterpreterGuard *PyInterpreterGuard_FromCurrent(void)

Create a finalization guard for the current interpreter.

On success, this function returns a guard for the current interpreter (as determined by the attached thread state); on failure, it returns NULL with an exception set. This function will fail only if the current interpreter has already started finalizing, or if the process is out of memory.

The guard pointer returned by this function must be eventually closed with PyInterpreterGuard_Close(); failing to do so will result in the Python process infinitely hanging.

The caller must hold an attached thread state.

PyInterpreterGuard *PyInterpreterGuard_FromView(PyInterpreterView *view)

Create a finalization guard for an interpreter through a view. view must not be NULL.

On success, this function returns a guard to the interpreter represented by view. The view is still valid after calling this function. The guard must eventually be closed with PyInterpreterGuard_Close().

If the interpreter no longer exists, is already finalizing, or out of memory, then this function returns NULL without setting an exception.

The caller does not need to hold an attached thread state.

void PyInterpreterGuard_Close(PyInterpreterGuard *guard)

Close an interpreter guard, allowing the interpreter to enter finalization if no other guards remain. If an interpreter guard is never closed, the interpreter will infinitely wait when trying to enter finalization.

After an interpreter guard is closed, it may not be used in PyThreadState_Ensure(). Doing so will result in undefined behavior.

This function cannot fail, and the caller doesn’t need to hold an attached thread state.

Interpreter views

type PyInterpreterView

An opaque view of an interpreter.

This is a thread-safe way to access an interpreter that may have be finalizing or already destroyed.

PyInterpreterView *PyInterpreterView_FromCurrent(void)

Create a view to the current interpreter.

This function is generally meant to be used alongside PyInterpreterGuard_FromView() or PyThreadState_EnsureFromView().

On success, this function returns a view to the current interpreter; on failure, it returns NULL with an exception set.

The caller must hold an attached thread state.

void PyInterpreterView_Close(PyInterpreterView *view)

Delete an interpreter view. If an interpreter view is never closed, the view’s memory will never be freed, but there are no other consequences. (In contrast, forgetting to close a guard will infinitely hang the main thread during finalization.)

This function cannot fail, and the caller doesn’t need to hold an attached thread state.

PyInterpreterView *PyInterpreterView_FromMain()

Create a view for the main interpreter (the first and default interpreter in a Python process).

On success, this function returns a view to the main interpreter; on failure, it returns NULL without an exception set. Failure indicates that the process is out of memory.

The caller does not need to hold an attached thread state.

Attaching and detaching thread states

This proposal includes three new high-level threading APIs that intend to replace PyGILState_Ensure() and PyGILState_Release().

PyThreadState *PyThreadState_Ensure(PyInterpreterGuard *guard)

Ensure that the thread has an attached thread state for the interpreter protected by guard, and thus can safely invoke that interpreter.

It is OK to call this function if the thread already has an attached thread state, as long as there is a subsequent call to PyThreadState_Release() that matches this one.

Nested calls to this function will only sometimes create a new thread state.

First, this function checks if an attached thread state is present. If there is, this function then checks if the interpreter of that thread state matches the interpreter guarded by guard. If that is the case, this function simply marks the thread state as being used by a PyThreadState_Ensure call and returns.

If there is no attached thread state, then this function checks if any thread state has been used by the current OS thread. (This is returned by PyGILState_GetThisThreadState().) If there was, then this function checks if that thread state’s interpreter matches guard. If it does, it is re-attached and marked as used.

Otherwise, if both of the above cases fail, a new thread state is created for guard. It is then attached and marked as owned by PyThreadState_Ensure.

This function will return NULL to indicate a memory allocation failure, and otherwise return a pointer to the thread state that was previously attached (which might have been NULL, in which case an non-NULL sentinel value is returned instead to differentiate between failure – this means that this function will sometimes return an invalid PyThreadState pointer).

To visualize, this function is roughly equivalent to the following:

PyThreadState *
PyThreadState_Ensure(PyInterpreterGuard *guard)
{
    assert(guard != NULL);
    PyInterpreterState *interp = PyInterpreterGuard_GetInterpreter(guard);
    assert(interp != NULL);

    PyThreadState *current_tstate = PyThreadState_GetUnchecked();
    if (current_tstate == NULL) {
        PyThreadState *last_used = PyGILState_GetThisThreadState();
        if (last_used != NULL) {
            ++last_used->ensure_counter;
            PyThreadState_Swap(last_used);
            return NO_TSTATE_SENTINEL;
        }
    } else if (current_tstate->interp == interp) {
        ++current_tstate->ensure_counter;
        return current_tstate;
    }

    PyThreadState *new_tstate = PyThreadState_New(interp);
    if (new_tstate == NULL) {
        return NULL;
    }

    ++new_tstate->ensure_counter;
    mark_tstate_owned_by_ensure(new_tstate);
    PyThreadState_Swap(new_tstate);
    return current_tstate == NULL ? NO_TSTATE_SENTINEL : current_tstate;
}

PyThreadState *PyThreadState_EnsureFromView(PyInterpreterView *view)

Get an attached thread state for the interpreter referenced by view.

view must not be NULL. If the interpreter referenced by view has been finalized or is currently finalizing, then this function returns NULL without setting an exception. This function may also return NULL to indicate that the process is out of memory.

The interpreter referenced by view will be implicitly guarded. The guard will be released upon the corresponding PyThreadState_Release() call.

On success, this function will return the thread state that was previously attached. If no thread state was previously attached, this returns a non-NULL sentinel value. The behavior of whether this function creates a thread state is equivalent to that of PyThreadState_Ensure().

To visualize, function is roughly equivalent to the following:

PyThreadState *
PyThreadState_EnsureFromView(PyInterpreterView *view)
{
    assert(view != NULL);
    PyInterpreterGuard *guard = PyInterpreterGuard_FromView(view);
    if (guard == NULL) {
        return NULL;
    }

    PyThreadState *tstate = PyThreadState_Ensure(guard);
    if (tstate == NULL) {
        PyInterpreterGuard_Close(guard);
        return NULL;
    }
    close_guard_upon_tstate_release(tstate, guard);
    return tstate;
}

void PyThreadState_Release(PyThreadState *tstate)

Release a PyThreadState_Ensure() call. This must be called exactly once for each call to PyThreadState_Ensure.

This function will decrement an internal counter on the attached thread state. If this counter ever reaches below zero, this function emits a fatal error (via Py_FatalError()).

If the attached thread state is owned by PyThreadState_Ensure, then the attached thread state will be deallocated and deleted upon the internal counter reaching zero. Otherwise, nothing happens when the counter reaches zero.

If tstate is non-NULL, it will be attached upon returning. If tstate indicates that no prior thread state was attached, there will be no attached thread state upon returning.

To visualize, this function is roughly equivalent to the following:

void
PyThreadState_Release(PyThreadState *old_tstate)
{
    PyThreadState *current_tstate = PyThreadState_Get();
    assert(old_tstate != NULL);
    assert(current_tstate != NULL);
    assert(current_tstate->ensure_counter > 0);
    if (--current_tstate->ensure_counter > 0) {
        // There are remaining PyThreadState_Ensure() calls
        // for this thread state.
        return;
    }

    assert(current_tstate->ensure_counter == 0);
    if (old_tstate == NO_TSTATE_SENTINEL) {
        // No thread state was attached prior the PyThreadState_Ensure()
        // call. So, we can just destroy the current thread state and return.
        assert(should_dealloc_tstate(current_tstate));
        PyThreadState_Clear(current_tstate);
        PyThreadState_DeleteCurrent();
        return;
    }

    if (should_dealloc_tstate(current_tstate)) {
        // The attached thread state was created by the initial PyThreadState_Ensure()
        // call. It's our job to destroy it.
        PyThreadState_Clear(current_tstate);
        PyThreadState_DeleteCurrent();
    }

    PyThreadState_Swap(old_tstate);
}

Soft deprecation of PyGILState APIs

This proposal issues a soft deprecation on all of the existing PyGILState APIs in favor of the existing and new PyThreadState APIs. Soft deprecations only mean that these APIs will not be developed further; there is no plan to remove PyGILState from Python’s C API.

Below is the full list of soft deprecated functions and their replacements:

1. PyGILState_Ensure(): use PyThreadState_Ensure() instead.
2. PyGILState_Release(): use PyThreadState_Release() instead.
3. PyGILState_GetThisThreadState(): use PyThreadState_Get() or PyThreadState_GetUnchecked() instead.
4. PyGILState_Check(): use PyThreadState_GetUnchecked() != NULL instead.

Additions to the Limited API

The following APIs from this PEP are to be added to the limited C API:

1. PyThreadState_Ensure()
2. PyThreadState_EnsureFromView()
3. PyThreadState_Release()
4. PyInterpreterView (as an opaque structure)
5. PyInterpreterView_FromCurrent()
6. PyInterpreterView_Close()
7. PyInterpreterView_FromMain()
8. PyInterpreterGuard (as an opaque structure)
9. PyInterpreterGuard_FromCurrent()
10. PyInterpreterGuard_Close()

Why a new API instead of fixing PyGILState?

Examples

/* Log to a Python file. No attached thread state is required by the caller. */
int
log_to_py_file_object(PyInterpreterView *view, PyObject *file,
                      PyObject *text)
{
    assert(view != NULL);
    PyThreadState *tstate = PyThreadState_EnsureFromView(view);
    if (tstate == NULL) {
        fputs("Cannot call Python.\n", stderr);
        return -1;
    }

    const char *to_write = PyUnicode_AsUTF8(text);
    if (to_write == NULL) {
        // Since the exception may be destroyed upon calling PyThreadState_Release(),
        // print out the exception ourselves.
        PyErr_Print();
        PyThreadState_Release(tstate);
        PyInterpreterGuard_Close(guard);
        return -1;
    }
    int res = PyFile_WriteString(to_write, file);
    if (res < 0) {
        PyErr_Print();
    }

    PyThreadState_Release(tstate);
    return res < 0;
}

static PyObject *
critical_operation(PyObject *self, PyObject *Py_UNUSED(args))
{
    assert(PyThreadState_GetUnchecked() != NULL);
    PyInterpreterGuard *guard = PyInterpreterGuard_FromCurrent();
    if (guard == NULL) {
        /* Python is already finalizing or out of memory. */
        return NULL;
    }

    Py_BEGIN_ALLOW_THREADS;
    PyMutex_Lock(&global_lock);

    /* Do something while holding the lock.
       The interpreter won't finalize during this period. */
    // ...

    PyMutex_Unlock(&global_lock);
    Py_END_ALLOW_THREADS;
    PyInterpreterGuard_Close(guard);

    Py_RETURN_NONE;
}

static int
thread_func(void *arg)
{
    PyGILState_STATE gstate = PyGILState_Ensure();
    /* It's not an issue in this example, but we just attached
       a thread state for the main interpreter. If my_method() was
       originally called in a subinterpreter, then we would be unable
       to safely interact with any objects from it. */

    // This can hang the thread during finalization, because print() will
    // detach the thread state while writing to stdout.
    if (PyRun_SimpleString("print(42)") < 0) {
        PyErr_Print();
    }
    PyGILState_Release(gstate);
    return 0;
}

static PyObject *
my_method(PyObject *self, PyObject *unused)
{
    PyThread_handle_t handle;
    PyThead_indent_t indent;

    if (PyThread_start_joinable_thread(thread_func, NULL, &ident, &handle) < 0) {
        return NULL;
    }

    // Join the thread, for example's sake.
    Py_BEGIN_ALLOW_THREADS;
    PyThread_join_thread(handle);
    Py_END_ALLOW_THREADS;

    Py_RETURN_NONE;
}

static int
thread_func(void *arg)
{
    PyInterpreterGuard *guard = (PyInterpreterGuard *)arg;
    PyThreadState *tstate = PyThreadState_Ensure(guard);
    if (tstate == NULL) {
        PyInterpreterGuard_Close(guard);
        return -1;
    }

    if (PyRun_SimpleString("print(42)") < 0) {
        PyErr_Print();
    }

    PyThreadState_Release(tstate);
    PyInterpreterGuard_Close(guard);
    return 0;
}

static PyObject *
my_method(PyObject *self, PyObject *unused)
{
    PyThread_handle_t handle;
    PyThead_indent_t indent;

    PyInterpreterGuard *guard = PyInterpreterGuard_FromCurrent();
    if (guard == NULL) {
        return NULL;
    }

    if (PyThread_start_joinable_thread(thread_func, guard, &ident, &handle) < 0) {
        PyInterpreterGuard_Close(guard);
        return NULL;
    }

    Py_BEGIN_ALLOW_THREADS
    PyThread_join_thread(handle);
    Py_END_ALLOW_THREADS

    Py_RETURN_NONE;
}

static int
thread_func(void *arg)
{
    PyInterpreterGuard *guard = (PyInterpreterGuard *)arg;
    PyThreadState *tstate = PyThreadState_Ensure(guard);
    if (tstate == NULL) {
        // Out of memory.
        PyInterpreterGuard_Close(guard);
        return -1;
    }

    // If no other guards are left, the interpreter may now finalize.
    PyInterpreterGuard_Close(guard);

    // This will detach the thread state while writing to stdout, which
    // will in turn allow for the thread to hang when attempting to reattach.
    if (PyRun_SimpleString("print(42)") < 0) {
        PyErr_Print();
    }

    PyThreadState_Release(tstate);
    return 0;
}

static PyObject *
my_method(PyObject *self, PyObject *unused)
{
    PyThread_handle_t handle;
    PyThead_indent_t indent;

    PyInterpreterGuard *guard = PyInterpreterGuard_FromCurrent();
    if (guard == NULL) {
        return NULL;
    }

    if (PyThread_start_joinable_thread(thread_func, guard, &ident, &handle) < 0) {
        PyInterpreterGuard_Close(guard);
        return NULL;
    }
    Py_RETURN_NONE;
}

static int
async_callback(void *arg)
{
    PyInterpreterView *view = (PyInterpreterView *)arg;
    // Try to create and attach a thread state based on our view.
    PyThreadState *tstate = PyThreadState_EnsureFromView(view);
    if (tstate == NULL) {
        PyInterpreterView_Close(view);
        return -1;
    }

    // Execute our Python code, now that we have an attached thread state.
    if (PyRun_SimpleString("print(42)") < 0) {
        PyErr_Print();
    }

    PyThreadState_Release(tstate);

    // In this example, we'll close the view for completeness.
    // If we wanted to use this callback again, we'd have to keep it alive.
    PyInterpreterView_Close(view);

    return 0;
}

static PyObject *
setup_callback(PyObject *self, PyObject *unused)
{
    PyInterpreterView *view = PyInterpreterView_FromCurrent();
    if (view == NULL) {
        return NULL;
    }

    MyNativeLibrary_RegisterAsyncCallback(async_callback, view);
    Py_RETURN_NONE;
}

PyThreadState *
MyGILState_Ensure(void)
{
    PyInterpreterView *view = PyInterpreterView_FromMain();
    if (view == NULL) {
        // Out of memory.
        PyThread_hang_thread();
    }

    PyThreadState *tstate = PyThreadState_EnsureFromView(view);
    PyInterpreterView_Close(view);
    return tstate;
}

#define MyGILState_Release PyThreadState_Release

Hard deprecating PyGILState

This PEP used to specify a “hard” deprecation of all APIs in the PyGILState family, with a planned removal in Python 3.20 (five years) or Python 3.25 (ten years).

This was eventually decided against, because while it is acknowledged that PyGILState_Ensure does have some fundamental flaws, it has worked for over twenty years, and migrating everything would simply be too large of a change for Python’s ecosystem.

Even with the finalization issues addressed by this PEP, a large majority of existing code that uses PyGILState_Ensure currently works, and will continue to work regardless of whether new APIs exist.

Interpreter reference counting

There were two iterations of this proposal that both specified that an interpreter maintain a reference count and would wait for that count to reach zero before shutting down.

The first iteration of this idea did this by adding implicit reference counting to PyInterpreterState * pointers. A function known as PyInterpreterState_Hold would increment the reference count (making it a “strong reference”), and PyInterpreterState_Release would decrement it. An interpreter’s ID (a standalone int64_t) was used as a form of weak reference, which could be used to look up an interpreter state and atomically increment its reference count.

These ideas were ultimately rejected because they seemed to make things very confusing. All uses of PyInterpreterState * would be implicitly borrowed or strong, making it difficult for developers to understand which parts of their code require or use a strong reference. This issue has already been acknowledged with PyObject * in Python’s C API.

In response to that pushback, this PEP specified PyInterpreterRef APIs that would also mimic reference counting, but in a more explicit manner that made it easier for developers. PyInterpreterRef was analogous to PyInterpreterGuard in this PEP. Similarly, the older revision included PyInterpreterWeakRef, which was analogous to PyInterpreterView.

Eventually, the notion of reference counting was completely abandoned from this proposal for a few reasons:

1. There was concern over overcomplication in the API design; the reference counting design looked very similar to HPy’s, which had no precedent in CPython. There was fear that this proposal was going to be used as precedent to introduce HPy into CPython.
2. Unlike traditional reference counting APIs, acquiring a strong reference to an interpreter could fail at any time, and an interpreter would not be deallocated immediately when its reference count reached zero.
3. There was prior discussion about adding “true” reference counting to interpreters (which would deallocate upon reaching zero), which would have been very confusing if there was an existing API in CPython titled PyInterpreterRef that did something different.

Non-daemon thread states

In earlier revisions of this PEP, interpreter guards were only ever a property of a thread state rather than a property of an interpreter. This meant that PyThreadState_Ensure() kept an interpreter guard held, and it was closed upon calling PyThreadState_Release(). A thread state that had a guard to an interpreter was known as a “non-daemon thread state”.

Functionally, this proposal still has this behavior under PyThreadState_EnsureFromView(), but this is not the default behavior. Additionally, the term “non-daemon” was confusing in contrast to threading threads, because non-daemon Thread objects are explicitly joined, whereas a non-daemon foreign thread would be only be waited on to release its guard.

Using PyStatus for the return value of PyThreadState_Ensure

In prior iterations of this API, PyThreadState_Ensure() returned a PyStatus to denote failure, which had the benefit of providing an error message.

This was rejected because it’s not clear that an error message would be all that useful, and it would make the new API more cumbersome to use.