The GIL Makes Many Types of Parallelism Difficult to Express

Neural network-based AI models expose multiple opportunities for parallelism. For example, individual operations may be parallelized internally (“intra-operator”), multiple operations may be executed simultaneously (“inter-operator”), and requests (spanning multiple operations) may also be parallelized. Efficient execution requires exploiting multiple types of parallelism [1].

The GIL makes it difficult to express inter-operator parallelism, as well as some forms of request parallelism, efficiently in Python. In other programming languages, a system might use threads to run different parts of a neural network on separate CPU cores, but this is inefficient in Python due to the GIL. Similarly, latency-sensitive inference workloads frequently use threads to parallelize across requests, but face the same scaling bottlenecks in Python.

The challenges the GIL poses to exploiting parallelism in Python frequently come up in reinforcement learning. Heinrich Kuttler, author of the NetHack Learning Environment and Member of Technical Staff at Inflection AI, writes:

Recent breakthroughs in reinforcement learning, such as on Dota 2, StarCraft, and NetHack rely on running multiple environments (simulated games) in parallel using asynchronous actor-critic methods. Straightforward multithreaded implementations in Python don’t scale beyond more than a few parallel environments due to GIL contention. Multiprocessing, with communication via shared memory or UNIX sockets, adds much complexity and in effect rules out interacting with CUDA from different workers, severely restricting the design space.

Manuel Kroiss, software engineer at DeepMind on the reinforcement learning team, describes how the bottlenecks posed by the GIL lead to rewriting Python codebases in C++, making the code less accessible:

We frequently battle issues with the Python GIL at DeepMind. In many of our applications, we would like to run on the order of 50-100 threads per process. However, we often see that even with fewer than 10 threads the GIL becomes the bottleneck. To work around this problem, we sometimes use subprocesses, but in many cases the inter-process communication becomes too big of an overhead. To deal with the GIL, we usually end up translating large parts of our Python codebase into C++. This is undesirable because it makes the code less accessible to researchers.

Projects that involve interfacing with multiple hardware devices face similar challenges: efficient communication requires use of multiple CPU cores. The Dose-3D project aims to improve cancer radiotherapy with precise dose planning. It uses medical phantoms (stand-ins for human tissue) together with custom hardware and a server application written in Python. Paweł Jurgielewicz, lead software architect for the data acquisition system on the Dose-3D project, describes the scaling challenges posed by the GIL and how using a fork of Python without the GIL simplified the project:

In the Dose-3D project, the key challenge was to maintain a stable, non-trivial concurrent communication link with hardware units while utilizing a 1 Gbit/s UDP/IP connection to the maximum. Naturally, we started with the multiprocessing package, but at some point, it became clear that most CPU time was consumed by the data transfers between the data processing stages, not by data processing itself. The CPython multithreading implementation based on GIL was a dead end too. When we found out about the “nogil” fork of Python it took a single person less than half a working day to adjust the codebase to use this fork and the results were astonishing. Now we can focus on data acquisition system development rather than fine-tuning data exchange algorithms.

Allen Goodman, author of CellProfiler and staff engineer at Prescient Design and Genentech, describes how the GIL makes biological methods research more difficult in Python:

Issues with Python’s global interpreter lock are a frequent source of frustration throughout biological methods research.

I wanted to better understand the current multithreading situation so I reimplemented parts of HMMER, a standard method for multiple-sequence alignment. I chose this method because it stresses both single-thread performance (scoring) and multi-threaded performance (searching a database of sequences). The GIL became the bottleneck when using only eight threads. This is a method where the current popular implementations rely on 64 or even 128 threads per process. I tried moving to subprocesses but was blocked by the prohibitive IPC costs. HMMER is a relatively elementary bioinformatics method and newer methods have far bigger multi-threading demands.

Method researchers are begging to use Python (myself included), because of its ease of use, the Python ecosystem, and because “it’s what people know.” Many biologists only know a little bit of programming (and that’s almost always Python). Until Python’s multithreading situation is addressed, C and C++ will remain the lingua franca of the biological methods research community.

The GIL Affects Python Library Usability

The GIL is a CPython implementation detail that limits multithreaded parallelism, so it might seem unintuitive to think of it as a usability issue. However, library authors frequently care a great deal about performance and will design APIs that support working around the GIL. These workaround frequently lead to APIs that are more difficult to use. Consequently, users of these APIs may experience the GIL as a usability issue and not just a performance issue.

For example, PyTorch exposes a multiprocessing-based API called DataLoader for building data input pipelines. It uses fork() on Linux because it is generally faster and uses less memory than spawn(), but this leads to additional challenges for users: creating a DataLoader after accessing a GPU can lead to confusing CUDA errors. Accessing GPUs within a DataLoader worker quickly leads to out-of-memory errors because processes do not share CUDA contexts (unlike threads within a process).

Olivier Grisel, scikit-learn developer and software engineer at Inria, describes how having to work around the GIL in scikit-learn related libraries leads to a more complex and confusing user experience:

Over the years, scikit-learn developers have maintained ancillary libraries such as joblib and loky to try to work around some of the limitations of multiprocessing: extra memory usage partially mitigated via semi-automated memory mapping of large data buffers, slow worker startup by transparently reusing a pool of long running workers, fork-safety problems of third-party native runtime libraries such as GNU OpenMP by never using the fork-only start-method, ability to perform parallel calls of interactively defined functions in notebooks and REPLs in cross-platform manner via cloudpickle. Despite our efforts, this multiprocessing-based solution is still brittle, complex to maintain and confusing to datascientists with limited understanding of system-level constraints. Furthermore, there are still irreducible limitations such as the overhead caused by the pickle-based serialization/deserialization steps required for inter-process communication. A lot of this extra work and complexity would not be needed anymore if we could use threads without contention on multicore hosts (sometimes with 64 physical cores or more) to run data science pipelines that alternate between Python-level operations and calls to native libraries.

Ralf Gommers, co-director of Quansight Labs and NumPy and SciPy maintainer, describes how the GIL affects the user experience of NumPy and numeric Python libraries:

A key problem in NumPy and the stack of packages built around it is that NumPy is still (mostly) single-threaded — and that has shaped significant parts of the user experience and projects built around it. NumPy does release the GIL in its inner loops (which do the heavy lifting), but that is not nearly enough. NumPy doesn’t offer a solution to utilize all CPU cores of a single machine well, and instead leaves that to Dask and other multiprocessing solutions. Those aren’t very efficient and are also more clumsy to use. That clumsiness comes mainly in the extra abstractions and layers the users need to concern themselves with when using, e.g., dask.array which wraps numpy.ndarray. It also shows up in oversubscription issues that the user must explicitly be aware of and manage via either environment variables or a third package, threadpoolctl. The main reason is that NumPy calls into BLAS for linear algebra - and those calls it has no control over, they do use all cores by default via either pthreads or OpenMP.

Coordinating on APIs and design decisions to control parallelism is still a major amount of work, and one of the harder challenges across the PyData ecosystem. It would have looked a lot different (better, easier) without a GIL.

GPU-Heavy Workloads Require Multi-Core Processing

Many high-performance computing (HPC) and AI workloads make heavy use of GPUs. These applications frequently require efficient multi-core CPU execution even though the bulk of the computation runs on a GPU.

Zachary DeVito, PyTorch core developer and researcher at FAIR (Meta AI), describes how the GIL makes multithreaded scaling inefficient even when the bulk of computation is performed outside of Python:

In PyTorch, Python is commonly used to orchestrate ~8 GPUs and ~64 CPU threads, growing to 4k GPUs and 32k CPU threads for big models. While the heavy lifting is done outside of Python, the speed of GPUs makes even just the orchestration in Python not scalable. We often end up with 72 processes in place of one because of the GIL. Logging, debugging, and performance tuning are orders-of-magnitude more difficult in this regime, continuously causing lower developer productivity.

The use of many processes (instead of threads) makes common tasks more difficult. Zachary DeVito continues:

On three separate occasions in the past couple of months (reducing redundant compute in data loaders, writing model checkpoints asynchronously, and parallelizing compiler optimizations), I spent an order-of-magnitude more time figuring out how to work around GIL limitations than actually solving the particular problem.

Even GPU-heavy workloads frequently have a CPU-intensive component. For example, computer vision tasks typically require multiple “pre-processing” steps in the data input pipeline, like image decoding, cropping, and resizing. These tasks are commonly performed on the CPU and may use Python libraries like Pillow or Pillow-SIMD. It is necessary to run the data input pipeline on multiple CPU cores in order to keep the GPU “fed” with data.

The increase in GPU performance compared to individual CPU cores makes multi-core performance more important. It is progressively more difficult to keep the GPUs fully occupied. To do so requires efficient use of multiple CPU cores, especially on multi-GPU systems. For example, NVIDIA’s DGX-A100 has 8 GPUs and two 64-core CPUs in order to keep the GPUs “fed” with data.

The GIL Makes Deploying Python AI Models Difficult

Python is widely used to develop neural network-based AI models. In PyTorch, models are frequently deployed as part of multi-threaded, mostly C++, environments. Python is often viewed skeptically because the GIL can be a global bottleneck, preventing efficient scaling even though the vast majority of the computations occur “outside” of Python with the GIL released. The torchdeploy paper [2] shows experimental evidence for these scaling bottlenecks in multiple model architectures.

PyTorch provides a number of mechanisms for deploying Python AI models that avoid or work around the GIL, but they all come with substantial limitations. For example, TorchScript captures a representation of the model that can be executed from C++ without any Python dependencies, but it only supports a limited subset of Python and often requires rewriting some of the model’s code. The torch::deploy API allows multiple Python interpreters, each with its own GIL, in the same process(similar to PEP 684). However, torch::deploy has limited support for Python modules that use C-API extensions.

Motivation Summary

Python’s global interpreter lock makes it difficult to use modern multi-core CPUs efficiently for many scientific and numeric computing applications. Heinrich Kuttler, Manuel Kroiss, and Paweł Jurgielewicz found that multi-threaded implementations in Python did not scale well for their tasks and that using multiple processes was not a suitable alternative.

The scaling bottlenecks are not solely in core numeric tasks. Both Zachary DeVito and Paweł Jurgielewicz described challenges with coordination and communication in Python.

Olivier Grisel, Ralf Gommers, and Zachary DeVito described how current workarounds for the GIL are “complex to maintain” and cause “lower developer productivity.” The GIL makes it more difficult to develop and maintain scientific and numeric computing libraries as well leading to library designs that are more difficult to use.

Build Configuration Changes

The global interpreter lock will remain the default for CPython builds and python.org downloads. A new build configuration flag, --disable-gil will be added to the configure script that will build CPython with support for running without the global interpreter lock.

When built with --disable-gil, CPython will define the Py_GIL_DISABLED macro in Python/patchlevel.h. The ABI tag will include the letter “t” (for “threading”).

The --disable-gil builds of CPython will still support optionally running with the GIL enabled at runtime (see PYTHONGIL Environment Variable and Py_mod_gil Slot).

Overview of CPython Changes

Removing the global interpreter lock requires substantial changes to CPython internals, but relatively few changes to the public Python and C APIs. This section describes the required changes to the CPython implementation followed by the proposed API changes.

The implementation changes can be grouped into the following four categories:

• Reference counting
• Memory management
• Container thread-safety
• Locking and atomic APIs

Reference Counting

Removing the GIL requires changes to CPython’s reference counting implementation to make it thread-safe. Furthermore, it needs to have low execution overhead and allow for efficient scaling with multiple threads. This PEP proposes a combination of three techniques to address these constraints. The first is a switch from plain non-atomic reference counting to biased reference counting, which is a thread-safe reference counting technique with lower execution overhead than plain atomic reference counting. The other two techniques are immortalization and a limited form of deferred reference counting; they address some of the multi-threaded scalability issues with reference counting by avoiding some reference count modifications.

Biased reference counting (BRC) is a technique first described in 2018 by Jiho Choi, Thomas Shull, and Josep Torrellas [3]. It is based on the observation that most objects are only accessed by a single thread, even in multi-threaded programs. Each object is associated with an owning thread (the thread that created it). Reference counting operations from the owning thread use non-atomic instructions to modify a “local” reference count. Other threads use atomic instructions to modify a “shared” reference count. This design avoids many atomic read-modify-write operations that are expensive on contemporary processors.

The implementation of BRC proposed in this PEP largely matches the original description of biased reference counting, but differs in details like the size of reference counting fields and special bits in those fields. BRC requires storing three pieces of information in each object’s header: the “local” reference count, the “shared” reference count, and the identifier of the owning thread. The BRC paper packs these three things into a single 64-bit field. This PEP proposes using three separate fields in each object’s header to avoid potential issues due to reference count overflow. Additionally, the PEP supports a faster deallocation path that avoids an atomic operation in the common case.

The proposed PyObject struct (also called struct _object) is below:

struct _object {
  _PyObject_HEAD_EXTRA
  uintptr_t ob_tid;         // owning thread id (4-8 bytes)
  uint16_t __padding;       // reserved for future use (2 bytes)
  PyMutex ob_mutex;         // per-object mutex (1 byte)
  uint8_t ob_gc_bits;       // GC fields (1 byte)
  uint32_t ob_ref_local;    // local reference count (4 bytes)
  Py_ssize_t ob_ref_shared; // shared reference count and state bits (4-8 bytes)
  PyTypeObject *ob_type;
};

The ob_tid, ob_ref_local, and ob_ref_shared are used by the biased reference counting implementation. The ob_gc_bits field is used store garbage collection flags that were previously stored in PyGC_Head (see Garbage Collection (Cycle Collection)). The ob_mutex field provides a per-object lock in a single byte.

// low two bits of "ob_ref_shared" are used for flags
#define _Py_SHARED_SHIFT 2

void Py_INCREF(PyObject *op)
{
  uint32_t new_local = op->ob_ref_local + 1;
  if (new_local == 0)
    return;  // object is immortal
  if (op->ob_tid == _Py_ThreadId())
    op->ob_ref_local = new_local;
  else
    atomic_add(&op->ob_ref_shared, 1 << _Py_SHARED_SHIFT);
}

void Py_DECREF(PyObject *op)
{
  if (op->ob_ref_local == _Py_IMMORTAL_REFCNT) {
    return;  // object is immortal
  }
  if (op->ob_tid == _Py_ThreadId()) {
    op->ob_ref_local -= 1;
    if (op->ob_ref_local == 0) {
      _Py_MergeZeroRefcount(); // merge refcount
    }
  }
  else {
    _Py_DecRefShared(); // slow path
  }
}

void _Py_MergeZeroRefcount(PyObject *op)
{
  if (op->ob_ref_shared == 0) {
    // quick deallocation code path (common case)
    op->ob_tid = 0;
    _Py_Dealloc(op);
  }
  else {
    // slower merging path not shown
  }
}

Memory Management

CPython currently uses an internal allocator, pymalloc, which is optimized for small object allocation. The pymalloc implementation is not thread-safe without the GIL. This PEP proposes replacing pymalloc with mimalloc, a general-purpose thread-safe allocator with good performance, including for small allocations.

Using mimalloc, with some modifications, also addresses two other issues related to removing the GIL. First, traversing the internal mimalloc structures allows the garbage collector to find all Python objects without maintaining a linked list. This is described in more detail in the garbage collection section. Second, mimalloc heaps and allocations based on size class enable collections like dict to generally avoid acquiring locks during read-only operations. This is described in more detail in the collection thread-safety section.

CPython already requires that objects that support garbage collection use the GC allocator APIs (typically indirectly by calling PyType_GenericAlloc). This PEP would add additional requirements to the use of the Python allocator APIs. First, Python objects must be allocated through object allocation APIs, such as PyType_GenericAlloc, PyObject_Malloc, or other Python APIs that wrap those calls. Python objects should not be allocated through other APIs, such as raw calls to C’s malloc or the C++ new operator. Additionally, PyObject_Malloc should be used only for allocating Python objects; it should not be used for allocating buffers, storages, or other data structures that are not PyObjects.

This PEP also imposes restrictions on the pluggable allocator API (PyMem_SetAllocator). When compiling without the GIL, allocators set using this API must eventually delegate the allocation to the corresponding underlying allocator, such as PyObject_Malloc, for Python object allocations. This allows for allocators that “wrap” underlying allocators, such as Python’s tracemalloc and debug allocator, but not for wholly replacing the allocator.

Garbage Collection (Cycle Collection)

The CPython garbage collector requires the following changes to work with this proposal:

• Use of “stop-the-world” to provide thread-safety guarantees that were previously provided by the GIL.
• Elimination of generational garbage collection in favor of non-generational collector.
• Integration with deferred reference counting and biased reference counting.

Additionally, the above changes enable removing the _gc_prev and _gc_next fields from GC objects. The GC bits that stored the tracked, finalized, and unreachable states are moved to the ob_gc_bits field in the PyObject header.

Container Thread-Safety

In CPython, the global interpreter lock protects against corruption of internal interpreter states when multiple threads concurrently access or modify Python objects. For example, if multiple threads concurrently modify the same list, the GIL ensures that the length of the list (ob_size) accurately matches the number of elements, and that the reference counts of each element accurately reflect the number of references to those elements. Without the GIL — and absent other changes — concurrent modifications would corrupt those fields and likely lead to program crashes.

The GIL does not necessarily ensure that operations are atomic or remain correct when multiple operations occur concurrently. For example, list.extend(iterable) may not appear atomic if the iterable has an iterator implemented in Python (or releases the GIL internally). Similarly, list.remove(x) can remove the wrong object if it overlaps with another operation that modifies the list, depending on the implementation of the equality operator. Still, the GIL ensures that some operations are effectively atomic. For example, the constructor list(set) atomically copies the items of the set to a new list, and some code relies on that copy being atomic (i.e., having a snapshot of the items in the set). This PEP preserves that property.

This PEP proposes using per-object locks to provide many of the same protections that the GIL provides. For example, every list, dictionary, and set will have an associated lightweight lock. All operations that modify the object must hold the object’s lock. Most operations that read from the object should acquire the object’s lock as well; the few read operations that can proceed without holding a lock are described below.

Per-object locks with critical sections provide weaker protections than the GIL. Because the GIL doesn’t necessarily ensure that concurrent operations are atomic or correct, the per-object locking scheme also cannot ensure that concurrent operations are atomic or correct. Instead, per-object locking aims for similar protections as the GIL, but with mutual exclusion limited to individual objects.

Most operations on an instance of a container type require locking that object. For example:

• list.append, list.insert, list.repeat, PyList_SetItem
• dict.__setitem__, PyDict_SetItem
• list.clear, dict.clear
• list.__repr__, dict.__repr__, etc.
• list.extend(iterable)
• setiter_iternext

Some operations operate directly on two container objects, with knowledge about both containers’ internal structure. For example, there are internal specializations of list.extend(iterable) for specific iterable types, like set. These operations need to lock both container objects because they access the internals of both objects simultaneously. Note that the generic implementation of list.extend only needs to lock one object (the list) because the other object is accessed indirectly through the thread-safe iterator API. Operations that lock two containers are:

• list.extend(list), list.extend(set), list.extend (dictitems), and other specializations where the implementation is specialized for argument type.
• list.concat(list)
• list.__eq__(list), dict.__eq__(dict)

Some simple operations can be implemented directly with atomic accesses and do not need locks because they only access a single field. These operations include:

• len(list) i.e., list_length(PyListObject *a)
• len(dict)
• len(set)

A select few operations optimistically avoid locking to improve performance. These require special implementations and cooperation from the memory allocator:

• list[idx] (list_subscript)
• dict[key] (dict_subscript)
• listiter_next, dictiter_iternextkey/value/item
• list.contains

PyObject *item = PyList_GetItem(list, idx);
Py_INCREF(item);

Py_INCREF(a->ob_item[i]);
return a->ob_item[i];

 PyObject **ob_item = atomic_load(&a->ob_item);
 PyObject *item = atomic_load(&ob_item[i]);
 if (!item || !_Py_TRY_INCREF(item)) goto retry;
 if (item != atomic_load(&ob_item[i])) {
   Py_DECREF(item);
   goto retry;
 }
 if (ob_item != atomic_load(&a->ob_item)) {
   Py_DECREF(item);
   goto retry;
}
return item;

retry:
  PyObject *item;
  Py_BEGIN_CRITICAL_SECTION(a->ob_mutex);
  item = a->ob_item[i];
  Py_INCREF(item);
  Py_END_CRITICAL_SECTION(a->ob_mutex);
  return item;

Specializing Interpreter

The specializing interpreter requires some changes to be thread-safe when running without the GIL:

• Concurrent specializations are prevented by using a mutex. This prevents multiple threads writing to the same inline cache.
• In multi-threaded programs running without the GIL, each bytecode is only specialized once. This prevents a thread from reading a partially written inline cache.
• Locking also ensures that cached values of tp_version_tag and keys_version are consistent with the cached descriptors and other values.
• Modifications to inline counters use “relaxed atomics”. In other words, some counter decrements may be missed or overwritten, but that does not affect correctness.

Py_mod_gil Slot

In --disable-gil builds, when loading an extension, CPython will check for a new PEP 489-style Py_mod_gil slot. If the slot is set to Py_mod_gil_not_used, then extension loading proceeds as normal. If the slot is not set, the interpreter pauses all threads and enables the GIL before continuing. Additionally, the interpreter will issue a visible warning naming the extension, that the GIL was enabled (and why) and the steps the user can take to override it.

PYTHONGIL Environment Variable

In --disable-gil builds, the user can also override the behavior at runtime by setting the PYTHONGIL environment variable. Setting PYTHONGIL=0, forces the GIL to be disabled, overriding the module slot logic. Setting PYTHONGIL=1, forces the GIL to be enabled.

The PYTHONGIL=0 override is important because extensions that are not thread-safe can still be useful in multi-threaded applications. For example, one may want to use the extension from only a single thread or guard access by locks. For context, there are already some extensions that are not thread-safe even with the GIL, and users already have to take these sorts of steps.

The PYTHONGIL=1 override is sometimes useful for debugging.

Non-Generational Garbage Collection

This PEP proposes switching from a generational cyclic garbage collector to a non-generational collector (when CPython is built without the GIL). That is equivalent to only having one generation (the “old” generation). There are two reasons for this proposed change.

Cyclic garbage collection, even for just the young generation, requires pausing other threads in the program. The author is concerned that frequent collections of the young generation would inhibit efficient scaling in multi-threaded programs. This is a concern for young generations (but not the old generation) because the young generations are collected after a fixed number of allocations, while the collections for the older generation are scheduled in proportion to the number of live objects in the heap. Additionally, it is difficult to efficiently keep track of objects in each generation without the GIL. For example, CPython currently uses a linked list of objects in each generation. If CPython were to keep that design, those lists would need to be made thread-safe, and it’s not clear how to do that efficiently.

Generational garbage collection is used to good effect in many other language runtimes. For example, many of the Java HotSpot garbage collector implementations use multiple generations [11]. In these runtimes, a young generation is frequently a throughput win: since a large percentage of the young generation is typically “dead,” the GC is able to reclaim a large amount memory relative to the amount of work performed. For example, several Java benchmarks show over 90% of “young” objects are typically collected [12] [13]. This is commonly referred to as the “weak generational hypothesis;” the observation is that most objects die young. This pattern is reversed in CPython due to the use of reference counting. Although most objects still die young, they are collected when their reference counts reach zero. Objects that survive to a garbage collection cycle are most likely to remain alive [14]. This difference means that generational collection is much less effective in CPython than in many other language runtimes [15].

Optimistic Avoiding Locking in dict and list Accesses

This proposal relies on a scheme that mostly avoids acquiring locks when accessing individual elements in lists and dictionaries. Note that this is not “lock free” in the sense of “lock-free” and “wait-free” algorithms that guarantee forward progress. It simply avoids acquiring locks (mutexes) in the common case to improve parallelism and reduce overhead.

A much simpler alternative would be to use reader-writer locks to protect dictionary and list accesses. Reader-writer locks allow concurrent reads, but not updates, which might seem ideal for list and dictionaries. The problem is that reader-writer locks have substantial overhead and poor scalability, particularly when the critical sections are small, as they are for single-element dictionary and list accesses [9]. The poor reader scalability stems from the fact that readers must all update the same data structure, such as the number of readers in pthread_rwlocks.

The technique described in this PEP is related to RCU (“read-copy-update”) [6] and, to a lesser extent, hazard pointers, two well-known schemes for optimizing concurrent, read-mostly data structures. RCU is widely used in the Linux kernel to protect shared data structures in a scalable manner. Both the technique in this PEP and RCU work by deferring reclamation while readers may be accessing the concurrent data structure. RCU is most commonly used to protect individual objects (like hash tables or linked lists), while this PEP proposes a scheme to protect larger blocks of memory (mimalloc “pages”) [10].

The need for this scheme is largely due to the use of reference counting in CPython. If CPython only relied on a tracing garbage collector, then this scheme would probably not be necessary because tracing garbage collectors already defer reclamation in the required manner. This would not “solve” scaling issues, but would shift many of the challenges to the garbage collector implementation.

Multiprocessing

The multiprocessing library allows Python programs to start and communicate with Python subprocesses. This allows for parallelism because each subprocess has its own Python interpreter (i.e., there’s one GIL per process). Multiprocessing has a few substantial limitations. Communication between processes is limited: objects generally need to be serialized or copied to shared memory. This introduces overhead (due to serialization) and complicates building APIs on top of multiprocessing. Starting a subprocess is also more expensive than starting a thread, especially with the “spawn” implementation. Starting a thread takes ~100 µs, while spawning a subprocess takes ~50 ms (50,000 µs) due to Python re-initialization.

Finally, many C and C++ libraries support access from multiple threads but do not support access or use across multiple processes.

Releasing the GIL in C-API Extensions

C-API extensions can release the GIL around long running functions. This allows for some degree of parallelism, since multiple threads can run concurrently when the GIL is released, but the overhead of acquiring and releasing the GIL typically prevents this from scaling efficiently beyond a few threads. Many scientific computing libraries release the GIL in computational heavy functions, and the CPython standard library releases the GIL around blocking I/O.

Internal Parallelization

Functions implemented in C may use multiple threads internally. For example, Intel’s NumPy distribution, PyTorch, and TensorFlow all use this technique to internally parallelize individual operations. This works well when the basic operations are large enough to be parallelized efficiently, but not when there are many small operations or when the operations depend on some Python code. Calling into Python from C requires acquiring the GIL – even short snippets of Python code can inhibit scaling.

Per-Interpreter GIL

The recently accepted PEP 684 proposes a per-interpreter GIL to address multi-core parallelism. This would allow parallelism between interpreters in the same process, but places substantial restrictions on sharing Python data between interpreters. Both this PEP and PEP 684 address the multi-core parallelism, but with different tradeoffs and techniques. It is feasible to implement both PEPs in CPython at the same time.

Gilectomy

Gilectomy [20] was a project by Larry Hastings to remove the GIL in CPython. Like the design proposed by this PEP, the Gilectomy supported multiple threads running in parallel within the same interpreter (i.e., “free-threading”) and made use of fine-grained locking. The reference implementation in this PEP improves on single-threaded performance and scalability compared to the Gilectomy.

PyParallel

PyParallel [21] was a proof-of-concept fork of Python 3.3 by Trent Nelson that supported multiple threads running simultaneously in a single Python process. The fork introduced the concept of “parallel threads” – threads that can run simultaneously while the main Python thread is suspended. Parallel threads had read-only access to objects created by the main thread. Objects created within parallel threads lived for the lifetime of the creating thread. For HTTP servers, this might correspond to the lifetime of a request.

python-safethread

The python-safethread [22] project was a patch to Python 3.0 by Adam Olsen to remove the GIL. Some aspects of the project are similar to the design proposed by this PEP. Both use fine-grained locking and optimize reference counting for cases where the object is created and accessed by the same thread.

Greg Stein’s Free-Threading Patch

In 1996, Greg Stein published a patch against Python 1.4 that removed the GIL [23]. The patch used atomic reference counting on Windows and a global reference count lock on Linux. List and dictionary accesses were protected by mutexes. Parts of the patch were adopted in CPython. In particular, the patch introduced a PyThreadState structure and correct per-thread exception handling.

Dave Beazley revisited the patch in a 2011 blog post [24].

Jython and IronPython

Some alternative Python implementations like Jython [25] and IronPython [26] do not have a global interpreter lock. However, they do not support CPython extensions. (The implementations can interface with code written in Java or C#).

PyPy-STM

The pypy-stm [27] interpreter is a variant of PyPy that uses software transactional memory. The authors report single-threaded performance overhead in the 20%-50% range compared to PyPy. It is not compatible with CPython extensions.

Why Not Use a Concurrent Garbage Collector?

Many recent garbage collectors are mostly concurrent – they avoid long stop-the-world pauses by allowing the garbage collector to run concurrently with the application. So why not use a concurrent collector?

Concurrent collection requires write barriers (or read barriers). The author is not aware of a way to add write barriers to CPython without substantially breaking the C-API.

Why Not Deprecate PyDict_GetItem in Favor of PyDict_FetchItem?

This PEP proposes a new API PyDict_FetchItem which behaves like PyDict_GetItem, but returns a new reference instead of a borrowed reference. As described in Borrowed References, some uses of borrowed references that were safe when running with the GIL are unsafe when running without the GIL and need to be replaced by functions like PyDict_FetchItem that return new references.

This PEP does not propose deprecating PyDict_GetItem and similar functions that return borrowed references for a few reasons:

• Many of the uses of borrowed references are safe, even when running without the GIL. For example, C API functions often use PyDict_GetItem to retrieve items from the keyword argument dictionary. These calls are safe because the keyword argument dictionary is only visible to a single thread.
• I tried this approach early on and found that wholesale replacing of PyDict_GetItem with PyDict_FetchItem frequently introduced new reference counting bugs. In my opinion, the risk of introducing new reference counting bugs generally outweighs the risks of missing a PyDict_GetItem call that is unsafe without the GIL.

Why Not Use PEP 683 Immortalization?

Like PEP 683, this PEP proposes an immortalization scheme for Python objects, but the PEPs use different bit representations to mark immortal objects. The schemes cannot be identical because this PEP depends on biased reference counting, which has two reference count fields instead of one.

Improved Specialization

The Python 3.11 release introduced quickening and specialization as part of the faster CPython project, substantially improving performance. Specialization replaces slow bytecode instructions with faster variants [19]. To maintain thread-safety, applications that use multiple threads (and run without the GIL) will only specialize each bytecode once, which can lower performance on some programs. It is possible to support specializing multiple times, but that requires more investigation and is not part of this PEP.

Python Build Modes

This PEP introduces a new build mode (--disable-gil) that is not ABI compatible with the standard build mode. The additional build mode adds complexity for both Python core developers and extension developers. The author believes a worthwhile goal is to combine these build modes and have the global interpreter lock controlled at runtime, possibly disabled by default. The path to this goal remains an open issue, but a possible path might look like the following:

1. In 2024, CPython 3.13 is released with support for a --disable-gil build time flag. There are two ABIs for CPython, one with the GIL and one without. Extension authors target both ABIs.
2. After 2–3 releases, (i.e., in 2026–2027), CPython is released with the GIL controlled by a runtime environment variable or flag. The GIL is enabled by default. There is only a single ABI.
3. After another 2–3 release (i.e., 2028–2030), CPython switches to the GIL being disabled by default. The GIL can still be enabled at runtime via an environment variable or command line flag.

This PEP covers the first step, with the remaining steps left as open issues. In this scenario, there would be a two to three year period where extension authors would target an extra CPython build per supported CPU architecture and OS.

Integration

The reference implementation changes approximately 15,000 lines of code in CPython and includes mimalloc, which is also approximately 15,000 lines of code. Most changes are not performance sensitive and can be included in both --disable-gil and the default builds. Some macros, like Py_BEGIN_CRITICAL_SECTION will be no-ops in the default build. Thee author does not expect a huge number of #ifdef statements to support the --disable-gil builds.

Mitigations for Single-Threaded Performance

The changes proposed in the PEP will increase execution overhead for --disable-gil builds compared to Python builds with the GIL. In other words, it will have slower single-threaded performance. There are some possible optimizations to reduce execution overhead, especially for --disable-gil builds that only use a single thread. These may be worthwhile if a longer term goal is to have a single build mode, but the choice of optimizations and their trade-offs remain an open issue.