Audit Hook

In order to observe actions taken by the runtime (on behalf of the caller), an API is required to raise messages from within certain operations. These operations are typically deep within the Python runtime or standard library, such as dynamic code compilation, module imports, DNS resolution, or use of certain modules such as ctypes.

The following new C APIs allow embedders and CPython implementors to send and receive audit hook messages:

# Add an auditing hook
typedef int (*hook_func)(const char *event, PyObject *args,
                         void *userData);
int PySys_AddAuditHook(hook_func hook, void *userData);

# Raise an event with all auditing hooks
int PySys_Audit(const char *event, PyObject *args);

The new Python APIs for receiving and raising audit hooks are:

# Add an auditing hook
sys.addaudithook(hook: Callable[[str, tuple]])

# Raise an event with all auditing hooks
sys.audit(str, *args)

Hooks are added by calling PySys_AddAuditHook() from C at any time, including before Py_Initialize(), or by calling sys.addaudithook() from Python code. Hooks cannot be removed or replaced. For CPython, hooks added from C are global, while hooks added from Python are only for the current interpreter. Global hooks are executed before interpreter hooks.

When events of interest are occurring, code can either call PySys_Audit() from C (while the GIL is held) or sys.audit(). The string argument is the name of the event, and the tuple contains arguments. A given event name should have a fixed schema for arguments, which should be considered a public API (for each x.y version release), and thus should only change between feature releases with updated documentation. To minimize overhead and simplify handling in native code hook implementations, named arguments are not supported.

For maximum compatibility, events using the same name as an event in the reference interpreter CPython should make every attempt to use compatible arguments. Including the name or an abbreviation of the implementation in implementation-specific event names will also help prevent collisions. For example, a pypy.jit_invoked event is clearly distinguished from an ipy.jit_invoked event. Events raised from Python modules should include their module or package name in the event name.

While event names may be arbitrary UTF-8 strings, for consistency across implementations it is recommended to use valid Python dotted names and avoid encoding specific details in the name. For example, an import event with the module name spam as an argument is preferable to a spam module imported event with no arguments. Avoid using embedded null characters or you may upset those who implement hooks using C.

When an event is audited, each hook is called in the order it was added (as much as is possible), passing the event name and arguments. If any hook returns with an exception set, later hooks are ignored and in general the Python runtime should terminate - exceptions from hooks are not intended to be handled or treated as expected occurrences. This allows hook implementations to decide how to respond to any particular event. The typical responses will be to log the event, abort the operation with an exception, or to immediately terminate the process with an operating system exit call.

When an event is audited but no hooks have been set, the audit() function should impose minimal overhead. Ideally, each argument is a reference to existing data rather than a value calculated just for the auditing call.

As hooks may be Python objects, they need to be freed during interpreter or runtime finalization. These should not be triggered at any other time, and should raise an event hook to ensure that any unexpected calls are observed.

Below in Suggested Audit Hook Locations, we recommend some important operations that should raise audit events. In general, events should be raised at the lowest possible level. Given the choice between raising an event from Python code or native code, raising from native code should be preferred.

Python implementations should document which operations will raise audit events, along with the event schema. It is intentional that sys.addaudithook(print) is a trivial way to display all messages.

Verified Open Hook

Most operating systems have a mechanism to distinguish between files that can be executed and those that can not. For example, this may be an execute bit in the permissions field, a verified hash of the file contents to detect potential code tampering, or file system path restrictions. These are an important security mechanism for ensuring that only code that has been approved for a given environment is executed.

Most kernels offer ways to restrict or audit binaries loaded and executed by the kernel. File types owned by Python appear as regular data and these features do not apply. This open hook allows Python embedders to integrate with operating system support when launching scripts or importing Python code.

The new public C API for the verified open hook is:

# Set the handler
typedef PyObject *(*hook_func)(PyObject *path, void *userData)
int PyFile_SetOpenCodeHook(hook_func handler, void *userData)

# Open a file using the handler
PyObject *PyFile_OpenCode(const char *path)

The new public Python API for the verified open hook is:

# Open a file using the handler
io.open_code(path : str) -> io.IOBase

The io.open_code() function is a drop-in replacement for open(abspath(str(pathlike)), 'rb'). Its default behaviour is to open a file for raw, binary access. To change the behaviour a new handler should be set. Handler functions only accept str arguments. The C API PyFile_OpenCode function assumes UTF-8 encoding. Paths must be absolute, and it is the responsibility of the caller to ensure the full path is correctly resolved.

A custom handler may be set by calling PyFile_SetOpenCodeHook() from C at any time, including before Py_Initialize(). However, if a hook has already been set then the call will fail. When open_code() is called with a hook set, the hook will be passed the path and its return value will be returned directly. The returned object should be an open file-like object that supports reading raw bytes. This is explicitly intended to allow a BytesIO instance if the open handler has already read the entire file into memory.

Note that these hooks can import and call the _io.open() function on CPython without triggering themselves. They can also use _io.BytesIO to return a compatible result using an in-memory buffer.

If the hook determines that the file should not be loaded, it should raise an exception of its choice, as well as performing any other logging.

All import and execution functionality involving code from a file will be changed to use open_code() unconditionally. It is important to note that calls to compile(), exec() and eval() do not go through this function - an audit hook that includes the code from these calls is the best opportunity to validate code that is read from the file. Given the current decoupling between import and execution in Python, most imported code will go through both open_code() and the log hook for compile, and so care should be taken to avoid repeating verification steps.

File accesses that are not intentionally planning to execute code are not expected to use this function. This includes loading pickles, XML or YAML files, where code execution is generally considered malicious rather than intentional. These operations should provide their own auditing events, preferably distinguishing between normal functionality (for example, Unpickler.load) and code execution (Unpickler.find_class).

A few examples: if the file type normally requires an execute bit (on POSIX) or would warn when marked as having been downloaded from the internet (on Windows), it should probably use open_code() rather than plain open(). Opening ZIP files using the ZipFile class should use open(), while opening them via zipimport should use open_code() to signal the correct intent. Code that uses the wrong function for a particular context may bypass the hook, which in CPython and the standard library should be considered a bug. Using a combination of open_code hooks and auditing hooks is necessary to trace all executed sources in the presence of arbitrary code.

There is no Python API provided for changing the open hook. To modify import behavior from Python code, use the existing functionality provided by importlib.

API Availability

While all the functions added here are considered public and stable API, the behavior of the functions is implementation specific. Most descriptions here refer to the CPython implementation, and while other implementations should provide the functions, there is no requirement that they behave the same.

For example, sys.addaudithook() and sys.audit() should exist but may do nothing. This allows code to make calls to sys.audit() without having to test for existence, but it should not assume that its call will have any effect. (Including existence tests in security-critical code allows another vector to bypass auditing, so it is preferable that the function always exist.)

io.open_code(path) should at a minimum always return _io.open(path, 'rb'). Code using the function should make no further assumptions about what may occur, and implementations other than CPython are not required to let developers override the behavior of this function with a hook.

Separate module for audit hooks

The proposal is to add a new module for audit hooks, hypothetically audit. This would separate the API and implementation from the sys module, and allow naming the C functions PyAudit_AddHook and PyAudit_Audit rather than the current variations.

Any such module would need to be a built-in module that is guaranteed to always be present. The nature of these hooks is that they must be callable without condition, as any conditional imports or calls provide opportunities to intercept and suppress or modify events.

Given it is one of the most core modules, the sys module is somewhat protected against module shadowing attacks. Replacing sys with a sufficiently functional module that the application can still run is a much more complicated task than replacing a module with only one function of interest. An attacker that has the ability to shadow the sys module is already capable of running arbitrary code from files, whereas an audit module could be replaced with a single line in a .pth file anywhere on the search path:

import sys; sys.modules['audit'] = type('audit', (object,),
    {'audit': lambda *a: None, 'addhook': lambda *a: None})

Multiple layers of protection already exist for monkey patching attacks against either sys or audit, but assignments or insertions to sys.modules are not audited.

This idea is rejected because it makes it trivial to suppress all calls to audit.

Flag in sys.flags to indicate “audited” mode

The proposal is to add a value in sys.flags to indicate when Python is running in a “secure” or “audited” mode. This would allow applications to detect when some features are enabled or when hooks have been added and modify their behaviour appropriately.

Currently, we are not aware of any legitimate reasons for a program to behave differently in the presence of audit hooks.

Both application-level APIs sys.audit and io.open_code are always present and functional, regardless of whether the regular python entry point or some alternative entry point is used. Callers cannot determine whether any hooks have been added (except by performing side-channel analysis), nor do they need to. The calls should be fast enough that callers do not need to avoid them, and the program is responsible for ensuring that any added hooks are fast enough to not affect application performance.

The argument that this is “security by obscurity” is valid, but irrelevant. Security by obscurity is only an issue when there are no other protective mechanisms; obscurity as the first step in avoiding attack is strongly recommended (see this article for discussion).

This idea is rejected because there are no appropriate reasons for an application to change its behaviour based on whether these APIs are in use.