Memory Allocators

This is one of the most sensitive parts of the work to isolate interpreters. The simplest solution is to move the global state of the internal “small block” allocator to PyInterpreterState, as we are doing with nearly all other runtime state. The following elaborates on the details and rationale.

CPython provides a memory management C-API, with three allocator domains: “raw”, “mem”, and “object”. Each provides the equivalent of malloc(), calloc(), realloc(), and free(). A custom allocator for each domain can be set during runtime initialization and the current allocator can be wrapped with a hook using the same API (for example, the stdlib tracemalloc module). The allocators are currently runtime-global, shared by all interpreters.

The “raw” allocator is expected to be thread-safe and defaults to glibc’s allocator (malloc(), etc.). However, the “mem” and “object” allocators are not expected to be thread-safe and currently may rely on the GIL for thread-safety. This is partly because the default allocator for both, AKA “pyobject”, is not thread-safe. This is due to how all state for that allocator is stored in C global variables. (See Objects/obmalloc.c.)

Thus we come back to the question of isolating runtime state. In order for interpreters to stop sharing the GIL, allocator thread-safety must be addressed. If interpreters continue sharing the allocators then we need some other way to get thread-safety. Otherwise interpreters must stop sharing the allocators. In both cases there are a number of possible solutions, each with potential downsides.

To keep sharing the allocators, the simplest solution is to use a granular runtime-global lock around the calls to the “mem” and “object” allocators in PyMem_Malloc(), PyObject_Malloc(), etc. This would impact performance, but there are some ways to mitigate that (e.g. only start locking once the first subinterpreter is created).

Another way to keep sharing the allocators is to require that the “mem” and “object” allocators be thread-safe. This would mean we’d have to make the pyobject allocator implementation thread-safe. That could even involve re-implementing it using an extensible allocator like mimalloc. The potential downside is in the cost to re-implement the allocator and the risk of defects inherent to such an endeavor.

Regardless, a switch to requiring thread-safe allocators would impact anyone that embeds CPython and currently sets a thread-unsafe allocator. We’d need to consider who might be affected and how we reduce any negative impact (e.g. add a basic C-API to help make an allocator thread-safe).

If we did stop sharing the allocators between interpreters, we’d have to do so only for the “mem” and “object” allocators. We might also need to keep a full set of global allocators for certain runtime-level usage. There would be some performance penalty due to looking up the current interpreter and then pointer indirection to get the allocators. Embedders would also likely have to provide a new allocator context for each interpreter. On the plus side, allocator hooks (e.g. tracemalloc) would not be affected.

Ultimately, we will go with the simplest option:

• keep the allocators in the global runtime state
• require that they be thread-safe
• move the state of the default object allocator (AKA “small block” allocator) to PyInterpreterState

We experimented with a rough implementation and found it was fairly straightforward, and the performance penalty was essentially zero.

PyInterpreterConfig.own_gil

If true (1) then the new interpreter will have its own “global” interpreter lock. This means the new interpreter can run without getting interrupted by other interpreters. This effectively unblocks full use of multiple cores. That is the fundamental goal of this PEP.

If false (0) then the new interpreter will use the main interpreter’s lock. This is the legacy (pre-3.12) behavior in CPython, where all interpreters share a single GIL. Sharing the GIL like this may be desirable when using extension modules that still depend on the GIL for thread safety.

In PyInterpreterConfig_INIT, this will be true. In PyInterpreterConfig_LEGACY_INIT, this will be false.

Also, to play it safe, for now we will not allow own_gil to be true if a custom allocator was set during runtime init. Wrapping the allocator, a la tracemalloc, will still be fine.

PyInterpreterConfig.strict_extensions_compat

PyInterpreterConfig.strict_extension_compat is basically the initial value used for “PyInterpreterState.strict_extension_compat”.

Extension Module Compatibility

As noted in Extension Modules, many extensions work fine in multiple interpreters (and under a per-interpreter GIL) without needing any changes. The import system will still fail if such a module doesn’t explicitly indicate support. At first, not many extension modules will, so this is a potential source of frustration.

We will address this by adding a context manager to temporarily disable the check on multiple interpreter support: importlib.util.allow_all_extensions(). More or less, it will modify the current “PyInterpreterState.strict_extension_compat” value (e.g. through a private sys function).

Extension Module Thread Safety

If a module supports use with multiple interpreters, that mostly implies it will work even if those interpreters do not share the GIL. The one caveat is where a module links against a library with internal global state that isn’t thread-safe. (Even something as innocuous as a static local variable as a temporary buffer can be a problem.) With a shared GIL, that state is protected. Without one, such modules must wrap any use of that state (e.g. through calls) with a lock.

Currently, it isn’t clear whether or not supports-multiple-interpreters is sufficiently equivalent to supports-per-interpreter-gil, such that we can avoid any special accommodations. This is still a point of meaningful discussion and investigation. The practical distinction between the two (in the Python community, e.g. PyPI) is not yet understood well enough to settle the matter. Likewise, it isn’t clear what we might be able to do to help extension maintainers mitigate the problem (assuming it is one).

In the meantime, we must proceed as though the difference would be large enough to cause problems for enough extension modules out there. The solution we would apply is:

• add a PyModuleDef slot that indicates an extension can be imported under a per-interpreter GIL (i.e. opt in)
• add that slot as part of the definition of a “compatible” extension, as discussed earlier

The downside is that not a single extension module will be able to take advantage of the per-interpreter GIL without extra effort by the module maintainer, regardless of how minor that effort. This compounds the problem described in Extension Module Compatibility and the same workaround applies. Ideally, we would determine that there isn’t enough difference to matter.

If we do end up requiring an opt-in for imports under a per-interpreter GIL, and later determine it isn’t necessary, then we can switch the default at that point, make the old opt-in slot a noop, and add a new PyModuleDef slot for explicitly opting out. In fact, it makes sense to add that opt-out slot from the beginning.

Extension Modules

Currently the most common usage of Python, by far, is with the main interpreter running by itself. This proposal has zero impact on extension modules in that scenario. Likewise, for better or worse, there is no change in behavior under multiple interpreters created using the existing Py_NewInterpreter().

Keep in mind that some extensions already break when used in multiple interpreters, due to keeping module state in global variables (or due to the internal state of linked libraries). They may crash or, worse, experience inconsistent behavior. That was part of the motivation for PEP 630 and friends, so this is not a new situation nor a consequence of this proposal.

In contrast, when the proposed API is used to create multiple interpreters, with the appropriate settings, the behavior will change for incompatible extensions. In that case, importing such an extension will fail (outside the main interpreter), as explained in Restricting Extension Modules. For extensions that already break in multiple interpreters, this will be an improvement.

Additionally, some extension modules link against libraries with thread-unsafe internal global state. (See Extension Module Thread Safety.) Such modules will have to start wrapping any direct or indirect use of that state in a lock. This is the key difference from other modules that also implement multi-phase init and thus indicate support for multiple interpreters (i.e. isolation).

Now we get to the break in compatibility mentioned above. Some extensions are safe under multiple interpreters (and a per-interpreter GIL), even though they haven’t indicated that. Unfortunately, there is no reliable way for the import system to infer that such an extension is safe, so importing them will still fail. This case is addressed in Extension Module Compatibility above.

Objects Exposed in the C-API

The C-API (including the limited API) exposes all the builtin types, including the builtin exceptions, as well as the builtin singletons. The exceptions are exposed as PyObject * but the rest are exposed as the static values rather than pointers. This was one of the few non-trivial problems we had to solve for per-interpreter GIL.

With immortal objects this is a non-issue.

Benefits to Consolidation

Consolidating the globals has a variety of benefits:

• greatly reduces the number of C globals (best practice for C code)
• the move draws attention to runtime state that is unstable or broken
• encourages more consistency in how runtime state is used
• makes it easier to discover/identify CPython’s runtime state
• makes it easier to statically allocate runtime state in a consistent way
• better memory locality for runtime state

Furthermore all the benefits listed in Indirect Benefits above also apply here, and the same projects listed there benefit.

Scale of Work

The number of global variables to be moved is large enough to matter, but most are Python objects that can be dealt with in large groups (like Py_IDENTIFIER). In nearly all cases, moving these globals to the interpreter is highly mechanical. That doesn’t require cleverness but instead requires someone to put in the time.

State To Be Moved

The remaining global variables can be categorized as follows:

• global objects
  • static types (incl. exception types)
  • non-static types (incl. heap types, structseq types)
  • singletons (static)
  • singletons (initialized once)
  • cached objects
• non-objects
  • will not (or unlikely to) change after init
  • only used in the main thread
  • initialized lazily
  • pre-allocated buffers
  • state

Those globals are spread between the core runtime, the builtin modules, and the stdlib extension modules.

For a breakdown of the remaining globals, run:

./python Tools/c-analyzer/table-file.py Tools/c-analyzer/cpython/globals-to-fix.tsv

Already Completed Work

As mentioned, this work has been going on for many years. Here are some of the things that have already been done:

• cleanup of runtime initialization (see PEP 432 / PEP 587)
• extension module isolation machinery (see PEP 384 / PEP 3121 / PEP 489)
• isolation for many builtin modules
• isolation for many stdlib extension modules
• addition of _PyRuntimeState
• no more _Py_IDENTIFIER()
• statically allocated:
  • empty string
  • string literals
  • identifiers
  • latin-1 strings
  • length-1 bytes
  • empty tuple

Tooling

As already indicated, there are several tools to help identify the globals and reason about them.

• Tools/c-analyzer/cpython/globals-to-fix.tsv - the list of remaining globals
• Tools/c-analyzer/c-analyzer.py
  • analyze - identify all the globals
  • check - fail if there are any unsupported globals that aren’t ignored
• Tools/c-analyzer/table-file.py - summarize the known globals

Also, the check for unsupported globals is incorporated into CI so that no new globals are accidentally added.

Global Objects

Global objects that are safe to be shared (without a GIL) between interpreters can stay on _PyRuntimeState. Not only must the object be effectively immutable (e.g. singletons, strings), but not even the refcount can change for it to be safe. Immortality (PEP 683) provides that. (The alternative is that no objects are shared, which adds significant complexity to the solution, particularly for the objects exposed in the public C-API.)

Builtin static types are a special case of global objects that will be shared. They are effectively immutable except for one part: __subclasses__ (AKA tp_subclasses). We expect that nothing else on a builtin type will change, even the content of __dict__ (AKA tp_dict).

__subclasses__ for the builtin types will be dealt with by making it a getter that does a lookup on the current PyInterpreterState for that type.