Non-Python threads hang during interpreter finalization

Many large libraries might need to call Python code in highly asynchronous situations where the desired interpreter may be finalizing or deleted, but want to continue running code after invoking the interpreter. This desire has been brought up by users. For example, a callback that wants to call Python code might be invoked when:

• A kernel has finished running on a GPU.
• A network packet was received.
• A thread has quit, and a native library is executing static finalizers for thread-local storage.

Generally, this pattern would look something like this:

static void
some_callback(void *arg)
{
    /* Do some work */
    /* ... */

    PyGILState_STATE gstate = PyGILState_Ensure();
    /* Invoke the C API to do some computation */
    PyGILState_Release(gstate);

    /* ... */
}

This comes with a hidden problem. If the target interpreter is finalizing, the current thread will hang! Or, if the target interpreter has been completely deleted, then attaching will likely result in a crash.

There are currently a few workarounds for this:

1. Leak resources to prevent the need to invoke Python.
2. Protect against finalization using an atexit callback.

Ideally, finalization should not be a footgun when working with Python’s C API.

Locks in native extensions can be unusable during finalization

When acquiring locks in a native API, it’s common (and often necessary) to release the GIL (or critical sections on the free-threaded build) to allow other code to execute during lock-aquisition. This can be problematic during finalization, because threads holding locks might be hung. For example:

1. A thread goes to acquire a lock, first detaching its thread state to avoid deadlocks. This is generally done through Py_BEGIN_ALLOW_THREADS.
2. The main thread begins finalization and tells all thread states to hang upon attachment.
3. The thread acquires the lock it was waiting on, but then hangs while attempting to reattach its thread state via Py_END_ALLOW_THREADS.
4. The main thread can no longer acquire the lock, because the thread holding it has hung.

python/cpython#129536 is an example of this problem. In that issue, Python issues a fatal error during finalization, because a daemon thread was hung while holding the lock for sys.stderr, so the main thread could no longer acquire it.

Finalization behavior for PyGILState_Ensure cannot change

There will always have to be a point in a Python program where PyGILState_Ensure() can no longer attach a thread state. If the interpreter is long dead, then Python obviously can’t give a thread a way to invoke it. Unfortunately, PyGILState_Ensure() doesn’t have any meaningful way to return a failure, so it has no choice but to terminate the thread or emit a fatal error. For example, this was discussed in python/cpython#124622:

I think a new GIL acquisition and release C API would be needed. The way the existing ones get used in existing C code is not amenible to suddenly bolting an error state onto; none of the existing C code is written that way. After the call they always just assume they have the GIL and can proceed. The API was designed as “it’ll block and only return once it has the GIL” without any other option.

PyGILState_Ensure can use the wrong (sub)interpreter

As of writing, the PyGILState functions are documented as being unsupported in subinterpreters.

This is because PyGILState_Ensure() doesn’t have any way to know which interpreter created the thread, and as such, it has to assume that it was the main interpreter. This can lead to some spurious issues.

For example:

1. The main thread enters a subinterpreter that creates a subthread.
2. The subthread calls PyGILState_Ensure() with no knowledge of which interpreter created it. Thus, the subthread takes the GIL for the main interpreter.
3. Now, the subthread might attempt to execute some resource for the subinterpreter. For example, the thread could have been passed a PyObject * reference to a list object.
4. The subthread calls list.append(), which attempts to call PyMem_Realloc() to resize the list’s internal buffer.
5. PyMem_Realloc uses the main interpreter’s allocator rather than the subinterpreter’s allocator, because the attached thread state (from PyGILState_Ensure) points to the main interpreter.
6. PyMem_Realloc doesn’t own the buffer in the list; crash!

The term “GIL” in PyGILState is confusing for free-threading

A significant issue with the term “GIL” in the C API is that it is semantically misleading. Again, in modern Python versions, PyGILState_Ensure() is about attaching a thread state, which only incidentally acquires the GIL. An attached thread state is still required to invoke the C API on the free-threaded build, but with a name that contains “GIL”, it is often confused to why it is still needed.

This is more of an incidental issue that can be fixed in addition to this PEP.

Interpreter guards

type PyInterpreterGuard

An opaque interpreter guard.

By holding an interpreter guard, the caller can ensure that the interpreter will not finalize until the guard is closed.

This is similar to a “readers-writers” lock; threads may concurrently guard an interpreter, and the interpreter will have to wait until all threads have closed their guards before it can enter finalization.

This type is guaranteed to be pointer-sized.

PyInterpreterGuard PyInterpreterGuard_FromCurrent(void)

Create a finalization guard for the current interpreter.

On success, this function returns a guard for the current interpreter; on failure, it returns 0 with an exception set. This function will fail only if the current interpreter has already started finalizing, or if the process is out-of-memory.

The caller must hold an attached thread state.

PyInterpreterGuard PyInterpreterGuard_FromView(PyInterpreterView view)

Create a finalization guard for an interpreter through a view.

On success, this function returns a guard to the interpreter represented by view. The view is still valid after calling this function. The guard must eventually be closed with PyInterpreterGuard_Close().

If the interpreter no longer exists or cannot safely run Python code, or if the process is out-of-memory, this function returns 0 without setting an exception.

The caller does not need to hold an attached thread state.

PyInterpreterState *PyInterpreterGuard_GetInterpreter(PyInterpreterGuard guard)

Return the PyInterpreterState pointer protected by guard.

This function cannot fail, and the caller doesn’t need to hold an attached thread state.

PyInterpreterGuard PyInterpreterGuard_Copy(PyInterpreterGuard guard)

Duplicate an interpreter guard.

On success, this function returns a copy of guard; on failure, it returns 0 without an exception set. This will only fail when the process is out-of-memory. The returned guard must eventually be closed with PyInterpreterGuard_Close().

The caller does not need to hold an attached thread state.

void PyInterpreterGuard_Close(PyInterpreterGuard guard)

Close an interpreter guard, allowing the interpreter to enter finalization if no other guards remain. If an interpreter guard is never closed, the interpreter will infinitely wait when trying to enter finalization.

After an interpreter guard is closed, it may not be used in PyThreadState_Ensure(). Doing so is undefined behavior.

This function cannot fail, and the caller doesn’t need to hold an attached thread state.

Interpreter views

type PyInterpreterView

An opaque view of an interpreter.

This is a thread-safe way to access an interpreter that may be finalized in another thread.

This type is guaranteed to be pointer-sized.

PyInterpreterView PyInterpreterView_FromCurrent(void)

Create a view to the current interpreter.

This function is generally meant to be used in tandem with PyInterpreterGuard_FromView().

On success, this function returns a view to the current interpreter; on failure, it returns 0 with an exception set.

The caller must hold an attached thread state.

PyInterpreterView PyInterpreterView_Copy(PyInterpreterView view)

Duplicate a view to an interpreter.

On success, this function returns a non-zero copy of view; on failure, it returns 0 without an exception set.

The caller does not need to hold an attached thread state.

void PyInterpreterView_Close(PyInterpreterView view)

Delete an interpreter view. If an interpreter view is never closed, the view’s memory will never be freed.

This function cannot fail, and the caller doesn’t need to hold an attached thread state.

PyInterpreterView PyUnstable_InterpreterView_FromDefault()

Create a view for an arbitrary “main” interpreter.

This function only exists for exceptional cases where a specific interpreter can’t be saved.

On success, this function returns a view to the main interpreter; on failure, it returns 0 without an exception set. Failure indicates that the process is out-of-memory.

The caller does not need to hold an attached thread state.

Attaching and detaching thread states

This proposal includes two new high-level threading APIs that intend to replace PyGILState_Ensure() and PyGILState_Release().

type PyThreadView

An opaque view of a thread state.

In this PEP, a thread view provides no additional properties beyond a PyThreadState* pointer. However, APIs for PyThreadView may be added in the future.

This type is guaranteed to be pointer-sized.

PyThreadView PyThreadState_Ensure(PyInterpreterGuard guard)

Ensure that the thread has an attached thread state for the interpreter protected by guard, and thus can safely invoke that interpreter.

It is OK to call this function if the thread already has an attached thread state, as long as there is a subsequent call to PyThreadState_Release() that matches this one.

Nested calls to this function will only sometimes create a new thread state.

First, this function checks if an attached thread state is present. If there is, this function then checks if the interpreter of that thread state matches the interpreter guarded by guard. If that is the case, this function simply marks the thread state as being used by a PyThreadState_Ensure call and returns.

If there is no attached thread state, then this function checks if any thread state has been used by the current OS thread. (This is returned by PyGILState_GetThisThreadState().) If there was, then this function checks if that thread state’s interpreter matches guard. If it does, it is re-attached and marked as used.

Otherwise, if both of the above cases fail, a new thread state is created for guard. It is then attached and marked as owned by PyThreadState_Ensure.

This function will return 0 to indicate a memory allocation failure, and otherwise return an opaque view to the thread state that was previously attached (which might have been NULL, in which case an non-NULL sentinel value is returned instead).

void PyThreadState_Release(PyThreadView view)

Release a PyThreadState_Ensure() call. This must be called exactly once for each call to PyThreadState_Ensure.

This function will decrement an internal counter on the attached thread state. If this counter ever reaches below zero, this function emits a fatal error.

If the attached thread state is owned by PyThreadState_Ensure, then the attached thread state will be deallocated and deleted upon the internal counter reaching zero. Otherwise, nothing happens when the counter reaches zero.

The thread state referenced by view, if any, will be attached upon returning. If view indicates that no prior thread state was attached, there will be no attached thread state upon returning.

Deprecation of PyGILState APIs

This proposal deprecates all of the existing PyGILState APIs in favor of the existing and new PyThreadState APIs.

1. PyGILState_Ensure(): use PyThreadState_Ensure() instead.
2. PyGILState_Release(): use PyThreadState_Release() instead.
3. PyGILState_GetThisThreadState(): use PyThreadState_Get() or PyThreadState_GetUnchecked() instead.
4. PyGILState_Check(): use PyThreadState_GetUnchecked() != NULL instead.

These APIs will be removed from public C API headers in Python 3.20 (five years from now). They will remain available in the stable ABI for compatibility.

Examples

/* Log to a Python file. No attached thread state is required by the caller. */
int
LogToPyFile(PyInterpreterView view,
            PyObject *file,
            PyObject *text)
{
    PyInterpreterGuard guard = PyInterpreterGuard_FromView(view);
    if (guard == 0) {
        /* Python interpreter has shut down */
        return -1;
    }

    PyThreadView thread_view = PyThreadState_Ensure(guard);
    if (thread_view == 0) {
        // Out of memory
        PyInterpreterGuard_Close(guard);
        fputs("Cannot call Python.\n", stderr);
        return -1;
    }

    const char *to_write = PyUnicode_AsUTF8(text);
    if (to_write == NULL) {
        // Since the exception may be destroyed upon calling PyThreadState_Release(),
        // print out the exception ourselves.
        PyErr_Print();
        PyThreadState_Release(thread_view);
        PyInterpreterGuard_Close(guard);
        return -1;
    }
    int res = PyFile_WriteString(to_write, file);
    if (res < 0) {
        PyErr_Print();
    }

    PyThreadState_Release(thread_view);
    PyInterpreterGuard_Close(guard);
    return res < 0;
}

static PyObject *
critical_operation(PyObject *self, PyObject *Py_UNUSED(args))
{
    assert(PyThreadState_GetUnchecked() != NULL);
    PyInterpreterGuard guard = PyInterpreterGuard_FromCurrent();
    if (guard == 0) {
        /* Python is already finalizing or out-of-memory. */
        return NULL;
    }

    Py_BEGIN_ALLOW_THREADS;
    PyMutex_Lock(&global_lock);

    /* Do something while holding the lock.
       The interpreter won't finalize during this period. */
    // ...

    PyMutex_Unlock(&global_lock);
    Py_END_ALLOW_THREADS;
    PyInterpreterGuard_Close(guard);

    Py_RETURN_NONE;
}

static int
thread_func(void *arg)
{
    PyGILState_STATE gstate = PyGILState_Ensure();
    /* It's not an issue in this example, but we just attached
       a thread state for the main interpreter. If my_method() was
       originally called in a subinterpreter, then we would be unable
       to safely interact with any objects from it. */

    // This can hang the thread during finalization, because print() will
    // detach the thread state while writing to stdout.
    if (PyRun_SimpleString("print(42)") < 0) {
        PyErr_Print();
    }
    PyGILState_Release(gstate);
    return 0;
}

static PyObject *
my_method(PyObject *self, PyObject *unused)
{
    PyThread_handle_t handle;
    PyThead_indent_t indent;

    if (PyThread_start_joinable_thread(thread_func, NULL, &ident, &handle) < 0) {
        return NULL;
    }

    // Join the thread, for example's sake.
    Py_BEGIN_ALLOW_THREADS;
    PyThread_join_thread(handle);
    Py_END_ALLOW_THREADS;

    Py_RETURN_NONE;
}

static int
thread_func(void *arg)
{
    PyInterpreterGuard guard = (PyInterpreterGuard)arg;
    PyThreadView thread_view = PyThreadState_Ensure(guard);
    if (thread_view == 0) {
        PyInterpreterGuard_Close(guard);
        return -1;
    }

    if (PyRun_SimpleString("print(42)") < 0) {
        PyErr_Print();
    }

    PyThreadState_Release(thread_view);
    PyInterpreterGuard_Close(guard);
    return 0;
}

static PyObject *
my_method(PyObject *self, PyObject *unused)
{
    PyThread_handle_t handle;
    PyThead_indent_t indent;

    PyInterpreterGuard guard = PyInterpreterGuard_FromCurrent();
    if (guard == 0) {
        return NULL;
    }

    // Since PyInterpreterGuard is the size of a pointer, we can just pass it as the void *
    // argument.
    if (PyThread_start_joinable_thread(thread_func, (void *)guard, &ident, &handle) < 0) {
        PyInterpreterGuard_Close(guard);
        return NULL;
    }

    Py_BEGIN_ALLOW_THREADS
    PyThread_join_thread(handle);
    Py_END_ALLOW_THREADS

    Py_RETURN_NONE;
}

static int
thread_func(void *arg)
{
    PyInterpreterGuard guard = (PyInterpreterGuard)arg;
    PyThreadView thread_view = PyThreadState_Ensure(guard);
    if (thread_view == 0) {
        // Out-of-memory.
        PyInterpreterGuard_Close(guard);
        return -1;
    }

    // If no other guards are left, the interpreter may now finalize.
    PyInterpreterGuard_Close(guard);

    // This will detach the thread state while writing to stdout.
    if (PyRun_SimpleString("print(42)") < 0) {
        PyErr_Print();
    }

    PyThreadState_Release(thread_view);
    return 0;
}

static PyObject *
my_method(PyObject *self, PyObject *unused)
{
    PyThread_handle_t handle;
    PyThead_indent_t indent;

    PyInterpreterGuard guard = PyInterpreterGuard_FromCurrent();
    if (guard == 0) {
        return NULL;
    }

    if (PyThread_start_joinable_thread(thread_func, (void *)guard, &ident, &handle) < 0) {
        PyInterpreterGuard_Close(guard);
        return NULL;
    }
    Py_RETURN_NONE;
}

static int
async_callback(void *arg)
{
    PyInterpreterView view = (PyInterpreterView)arg;

    // Try to guard the interpreter. If the interpreter is finalizing or has been finalized, this
    // will safely fail.
    PyInterpreterGuard guard = PyInterpreterGuard_FromView(view);
    if (guard == 0) {
        PyInterpreterView_Close(view);
        return -1;
    }

    // Try to create and attach a thread state based on our now-guarded interpreter.
    PyThreadView thread_view = PyThreadState_Ensure(guard);
    if (thread_view == 0) {
        PyInterpreterView_Close(view);
        PyInterpreterGuard_Close(guard);
        return -1;
    }

    // Execute our Python code, now that we have an attached thread state.
    if (PyRun_SimpleString("print(42)") < 0) {
        PyErr_Print();
    }

    PyThreadState_Release(thread_view);
    PyInterpreterGuard_Close(guard);

    // In this example, we'll close the view for completeness.
    // If we wanted to use this callback again, we'd have to keep it alive.
    PyInterpreterView_Close(view);

    return 0;
}

static PyObject *
setup_callback(PyObject *self, PyObject *unused)
{
    PyInterpreterView view = PyInterpreterView_FromCurrent();
    if (view == 0) {
        return NULL;
    }

    MyNativeLibrary_RegisterAsyncCallback(async_callback, (void *)view);
    Py_RETURN_NONE;
}

PyThreadView
MyGILState_Ensure(void)
{
    PyInterpreterView view = PyUnstable_InterpreterView_FromDefault();
    if (view == 0) {
        // Out-of-memory.
        PyThread_hang_thread();
    }

    PyInterpreterGuard guard = PyInterpreterGuard_FromView(view);
    if (guard == 0) {
        // Main interpreter is not available; hang the thread.
        // We won't bother with cleaning up resources.
        PyThread_hang_thread();
    }

    PyThreadView view = PyThreadState_Ensure(guard);
    PyInterpreterGuard_Close(guard);
    PyInterpreterView_Close(view);
    return view
}

#define MyGILState_Release PyThreadState_Release

How should the APIs fail?

There is some disagreement over how the PyInterpreter[Guard|View] APIs should indicate a failure to the caller. There are two competing ideas:

1. Return -1 to indicate failure, and 0 to indicate success. On success, functions will assign to a PyInterpreter[Guard|View] pointer passed as an argument.
2. Directly return a PyInterpreter[Guard|View], with a value of 0 being equivalent to NULL, indicating failure.

Currently, the PEP spells the latter.

Should the new functions be part of the stable ABI?

This PEP does not currently specify whether the new C API functions should be added to the limited C API, primarily due to a lack of discussion.

Interpreter reference counting

There were two iterations of this proposal that both specified that an interpreter maintain a reference count and would wait for that count to reach zero before shutting down.

The first iteration of this idea did this by adding implicit reference counting to PyInterpreterState * pointers. A function known as PyInterpreterState_Hold would increment the reference count (making it a “strong reference”), and PyInterpreterState_Release would decrement it. An interpreter’s ID (a standalone int64_t) was used as a form of weak reference, which could be used to look up an interpreter state and atomically increment its reference count. These ideas were ultimately rejected because they seemed to make things very confusing. All existing uses of PyInterpreterState * would be borrowed, making it difficult for developers to understand which parts of their code require or use a strong reference.

In response to that pushback, this PEP specified PyInterpreterRef APIs that would also mimic reference counting, but in a more explicit manner that made it easier for developers. PyInterpreterRef was analogous to PyInterpreterGuard in this PEP. Similarly, the older revision included PyInterpreterWeakRef, which was analogous to PyInterpreterView.

Eventually, the notion of reference counting was completely abandoned from this proposal for a few reasons:

1. There was contention over overcomplication in the API design; the reference-counting design looked very similar to HPy’s, which had no precedent in CPython. There was fear that this proposal was being overcomplicated to look more like HPy.
2. Unlike traditional reference-counting APIs, acquiring a strong reference to an interpreter could fail at any time, and an interpreter would not be deallocated immediately when its reference count reached zero.
3. There was prior discussion about adding “true” reference counting to interpreters (which would deallocate upon reaching zero), which would have been very confusing if there was an existing API in CPython titled PyInterpreterRef that did something different.

Non-daemon thread states

In earlier revisions of this PEP, interpreter guards were a property of a thread state rather than a property of an interpreter. This meant that PyThreadState_Ensure() kept an interpreter guard held, and it was closed upon calling PyThreadState_Release(). A thread state that had a guard to an interpreter was known as a “non-daemon thread state”.

At first, this seemed like an improvement because it shifted the management of a guard’s lifetime to the thread rather than the user, which eliminated some boilerplate. However, this ended up making the proposal significantly more complex and hurt the proposal’s goals:

1. Most importantly, non-daemon thread states place too much emphasis on daemon threads as the problem, which made the PEP confusing. Additionally, the phrase “non-daemon” added extra confusion, because non-daemon Python threads are explicitly joined, whereas a non-daemon C thread would only be waited on until it closes its guard(s).
2. In many cases, an interpreter guard should outlive a singular thread state. Stealing the interpreter guard in PyThreadState_Ensure() was particularly troublesome for these cases. If PyThreadState_Ensure() didn’t steal a guard with non-daemon thread states, it would make it less clear as to who owned to interpreter guard, leading to a more confusing API.

Exposing an Activate/Deactivate API instead of Ensure/Release

In prior discussions of this API, it was suggested to provide actual PyThreadState pointers in the API in an attempt to make the ownership and lifetime of the thread state more straightforward:

More importantly though, I think this makes it clearer who owns the thread state - a manually created one is controlled by the code that created it, and once it’s deleted it can’t be activated again.

This was ultimately rejected for two reasons:

1. The proposed API has closer usage to PyGILState_Ensure() & PyGILState_Release(), which helps ease the transition for old codebases.
2. It’s significantly easier for code-generators like Cython to use, as there isn’t any additional complexity with tracking PyThreadState pointers around.

Using PyStatus for the return value of PyThreadState_Ensure

In prior iterations of this API, PyThreadState_Ensure() returned a PyStatus instead of an integer to denote failures, which had the benefit of providing an error message.

This was rejected because it’s not clear that an error message would be all that useful; all the conceived use-cases for this API wouldn’t really care about a message indicating why Python can’t be invoked. As such, the API would only be needlessly more complex to use, which in turn would hurt the transition from PyGILState_Ensure().

In addition, PyStatus isn’t commonly used in the C API. A few functions related to interpreter initialization use it (simply because they can’t raise exceptions), and PyThreadState_Ensure() does not fall under that category.