Abstract

This PEP proposes adding a zero-overhead debugging interface to CPython that allows debuggers and profilers to safely attach to running Python processes. The interface provides safe execution points for attaching debugger code without modifying the interpreter’s normal execution path or adding runtime overhead.

A key application of this interface will be enabling pdb to attach to live processes by process ID, similar to gdb -p, allowing developers to inspect and debug Python applications interactively in real-time without stopping or restarting them.

Motivation

Debugging Python processes in production and live environments presents unique challenges. Developers often need to analyze application behavior without stopping or restarting services, which is especially crucial for high-availability systems. Common scenarios include diagnosing deadlocks, inspecting memory usage, or investigating unexpected behavior in real-time.

Very few Python tools can attach to running processes, primarily because doing so requires deep expertise in both operating system debugging interfaces and CPython internals. While C/C++ debuggers like GDB and LLDB can attach to processes using well-understood techniques, Python tools must implement all of these low-level mechanisms plus handle additional complexity. For example, when GDB needs to execute code in a target process, it:

1. Uses ptrace to allocate a small chunk of executable memory (easier said than done)
2. Writes a small sequence of machine code - typically a function prologue, the desired instructions, and code to restore registers
3. Saves all the target thread’s registers
4. Changes the instruction pointer to the injected code
5. Lets the process run until it hits a breakpoint at the end of the injected code
6. Restores the original registers and continues execution

Python tools face this same challenge of code injection, but with an additional layer of complexity. Not only do they need to implement the above mechanism, they must also understand and safely interact with CPython’s runtime state, including the interpreter loop, garbage collector, thread state, and reference counting system. This combination of low-level system manipulation and deep domain specific interpreter knowledge makes implementing Python debugging tools exceptionally difficult.

The few tools (see for example DebugPy and Memray) that do attempt this resort to suboptimal and unsafe methods, using system debuggers like GDB and LLDB to forcefully inject code. This approach is fundamentally unsafe because the injected code can execute at any point during the interpreter’s execution cycle - even during critical operations like memory allocation, garbage collection, or thread state management. When this happens, the results are catastrophic: attempting to allocate memory while already inside malloc() causes crashes, modifying objects during garbage collection corrupts the interpreter’s state, and touching thread state at the wrong time leads to deadlocks.

Various tools attempt to minimize these risks through complex workarounds, such as spawning separate threads for injected code or carefully timing their operations or trying to select some good points to stop the process. However, these mitigations cannot fully solve the underlying problem: without cooperation from the interpreter, there’s no way to know if it’s safe to execute code at any given moment. Even carefully implemented tools can crash the interpreter because they’re fundamentally working against it rather than with it.

Rationale

Rather than forcing tools to work around interpreter limitations with unsafe code injection, we can extend CPython with a proper debugging interface that guarantees safe execution. By adding a few thread state fields and integrating with the interpreter’s existing evaluation loop, we can ensure debugging operations only occur at well-defined safe points. This eliminates the possibility of crashes and corruption while maintaining zero overhead during normal execution.

The key insight is that we don’t need to inject code at arbitrary points - we just need to signal to the interpreter that we want code executed at the next safe opportunity. This approach works with the interpreter’s natural execution flow rather than fighting against it.

After describing this idea to the PyPy development team, this proposal has already been implemented in PyPy, proving both its feasibility and effectiveness. Their implementation demonstrates that we can provide safe debugging capabilities with zero runtime overhead during normal execution. The proposed mechanism not only reduces risks associated with current debugging approaches but also lays the foundation for future enhancements. For instance, this framework could enable integration with popular observability tools, providing real-time insights into interpreter performance or memory usage. One compelling use case for this interface is enabling pdb to attach to running Python processes, similar to how gdb allows users to attach to a program by process ID (gdb -p <pid>). With this feature, developers could inspect the state of a running application, evaluate expressions, and step through code dynamically. This approach would align Python’s debugging capabilities with those of other major programming languages and debugging tools that support this mode.

Specification

This proposal introduces a safe debugging mechanism that allows external processes to trigger code execution in a Python interpreter at well-defined safe points. The key insight is that rather than injecting code directly via system debuggers, we can leverage the interpreter’s existing evaluation loop and thread state to coordinate debugging operations.

The mechanism works by having debuggers write to specific memory locations in the target process that the interpreter then checks during its normal execution cycle. When the interpreter detects that a debugger wants to attach, it executes the requested operations only when it’s safe to do so - that is, when no internal locks are held and all data structures are in a consistent state.

typedef struct _remote_debugger_support {
    int debugger_pending_call;
    char debugger_script_path[MAX_SCRIPT_PATH_SIZE];
} _PyRemoteDebuggerSupport;

struct _debugger_support {
    uint64_t eval_breaker;              // Location of the eval breaker flag
    uint64_t remote_debugger_support;   // Offset to our support structure
    uint64_t debugger_pending_call;     // Where to write the pending flag
    uint64_t debugger_script_path;      // Where to write the script path
    uint64_t debugger_script_path_size; // Size of the script path buffer
} debugger_support;

// In ceval.c
if (tstate->eval_breaker) {
    if (tstate->remote_debugger_support.debugger_pending_call) {
        tstate->remote_debugger_support.debugger_pending_call = 0;
        const char *path = tstate->remote_debugger_support.debugger_script;
        if (*path) {
            if (0 != PySys_Audit("debugger_script", "%s", path)) {
                PyErr_Clear();
            } else {
                FILE* f = fopen(path, "r");
                if (!f) {
                    PyErr_SetFromErrno(OSError);
                } else {
                    PyRun_AnyFile(f, path);
                    fclose(f);
                }
                if (PyErr_Occurred()) {
                    PyErr_WriteUnraisable(...);
                }
            }
        }
    }
}

def remote_exec(pid: int, code: str, timeout: int = 0) -> None:
    """
    Executes a block of Python code in a given remote Python process.

    Args:
         pid (int): The process ID of the target Python process.
         code (str): A string containing the Python code to be executed.
         timeout (int): An optional timeout for waiting for the remote
            process to execute the code. If the timeout is exceeded a
            ``TimeoutError`` will be raised.
    """

import sys
# Execute a print statement in a remote Python process with PID 12345
try:
    sys.remote_exec(12345, "print('Hello from remote execution!')", timeout=3)
except TimeoutError:
    print(f"The remote process took too long to execute the code")
except Exception as e:
    print(f"Failed to execute code: {e}")

Backwards Compatibility

This change has no impact on existing Python code or interpreter performance. The added fields are only accessed during debugger attachment, and the checking mechanism piggybacks on existing interpreter safe points.

Security Implications

This interface does not introduce new security concerns as it is only usable by processes that can already write to arbitrary memory within a given process and execute arbitrary code on the machine (in order to create the file containing the Python code to be executed).

Furthermore, the execution of the code is gated by the interpreter’s audit hooks, which can be used to monitor or prevent the execution of the code in sensitive environments.

Existing operating system security mechanisms are effective for guarding against attackers gaining arbitrary memory write access. Although the PEP doesn’t specify how memory should be written to the target process, in practice this will be done using standard system calls that are already being used by other debuggers and tools. Some examples are:

• On Linux, the process_vm_readv() and process_vm_writev() system calls are used to read and write memory from another process. These operations are controlled by ptrace access mode checks - the same ones that govern debugger attachment. A process can only read from or write to another process’s memory if it has the appropriate permissions (typically requiring either root or the CAP_SYS_PTRACE capability, though less security minded distributions may allow any process running as the same uid to attach).
• On macOS, the interface would leverage mach_vm_read_overwrite() and mach_vm_write() through the Mach task system. These operations require task_for_pid() access, which is strictly controlled by the operating system. By default, access is limited to processes running as root or those with specific entitlements granted by Apple’s security framework.
• On Windows, the ReadProcessMemory() and WriteProcessMemory() functions provide similar functionality. Access is controlled through the Windows security model - a process needs PROCESS_VM_READ and PROCESS_VM_WRITE permissions, which typically require the same user context or appropriate privileges. These are the same permissions required by debuggers, ensuring consistent security semantics across platforms.

All mechanisms ensure that:

1. Only authorized processes can read/write memory
2. The same security model that governs traditional debugger attachment applies
3. No additional attack surface is exposed beyond what the OS already provides for debugging
4. Even if an attacker can write arbitrary memory, they cannot escalate this to arbitrary code execution unless they already have filesystem access

The memory operations themselves are well-established and have been used safely for decades in tools like GDB, LLDB, and various system profilers.

It’s important to note that any attempt to attach to a Python process via this mechanism would be detectable by system-level monitoring tools as well as by Python audit hooks. This transparency provides an additional layer of accountability, allowing administrators to audit debugging operations in sensitive environments.

Further, the strict reliance on OS-level security controls ensures that existing system policies remain effective. For enterprise environments, this means administrators can continue to enforce debugging restrictions using standard tools and policies without requiring additional configuration. For instance, leveraging Linux’s ptrace_scope or macOS’s taskgated to restrict debugger access will equally govern the proposed interface.

By maintaining compatibility with existing security frameworks, this design ensures that adopting the new interface requires no changes to established.

How to Teach This

For tool authors, this interface becomes the standard way to implement debugger attachment, replacing unsafe system debugger approaches. A section in the Python Developer Guide could describe the internal workings of the mechanism, including the debugger_support offsets and how to interact with them using system APIs.

End users need not be aware of the interface, benefiting only from improved debugging tool stability and reliability.

Reference Implementation

A reference implementation with a prototype adding remote support for pdb can be found here.

Rejected Ideas

Thanks

We would like to thank CF Bolz-Tereick for their insightful comments and suggestions when discussing this proposal.

Copyright

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.