Following system colour scheme Selected dark colour scheme Selected light colour scheme

Python Enhancement Proposals

PEP 667 – Consistent views of namespaces

Author:
Mark Shannon <mark at hotpy.org>, Tian Gao <gaogaotiantian at hotmail.com>
Discussions-To:
Discourse thread
Status:
Accepted
Type:
Standards Track
Created:
30-Jul-2021
Python-Version:
3.13
Post-History:
20-Aug-2021, 22-Feb-2024
Resolution:
Discourse message

Table of Contents

Abstract

In early versions of Python all namespaces, whether in functions, classes or modules, were all implemented the same way: as a dictionary.

For performance reasons, the implementation of function namespaces was changed. Unfortunately this meant that accessing these namespaces through locals() and frame.f_locals ceased to be consistent and some odd bugs crept in over the years as threads, generators and coroutines were added.

This PEP proposes making these namespaces consistent once more. Modifications to frame.f_locals will always be visible in the underlying variables. Modifications to local variables will immediately be visible in frame.f_locals, and they will be consistent regardless of threading or coroutines.

The locals() function will act the same as it does now for class and modules scopes. For function scopes it will return an instantaneous snapshot of the underlying frame.f_locals rather than implicitly refreshing a single shared dictionary cached on the frame object.

Motivation

The implementation of locals() and frame.f_locals in releases up to and including Python 3.12 is slow, inconsistent and buggy. We want to make it faster, consistent, and most importantly fix the bugs.

For example, when attempting to manipulate local variables via frame objects:

class C:
    x = 1
    sys._getframe().f_locals['x'] = 2
    print(x)

prints 2, but:

def f():
    x = 1
    sys._getframe().f_locals['x'] = 2
    print(x)
f()

prints 1.

This is inconsistent, and confusing. Worse than that, the Python 3.12 behavior can result in strange bugs.

With this PEP both examples would print 2 as the function level change would be written directly to the optimized local variables in the frame rather than to a cached dictionary snapshot.

There are no compensating advantages for the Python 3.12 behavior; it is unreliable and slow.

The locals() builtin has its own undesirable behaviours. Refer to PEP 558 for additional details on those concerns.

Rationale

Making the frame.f_locals attribute a write-through proxy

The Python 3.12 implementation of frame.f_locals returns a dictionary that is created on the fly from the array of local variables. The PyFrame_LocalsToFast() C API is then called by debuggers and trace functions that want to write their changes back to the array (until Python 3.11, this API was called implicitly after every trace function invocation rather than being called explicitly by the trace functions).

This can result in the array and dictionary getting out of sync with each other. Writes to the f_locals frame attribute may not show up as modifications to local variables if PyFrame_LocalsToFast() is never called. Writes to local variables can get lost if a dictionary snapshot created before the variables were modified is written back to the frame (since every known variable stored in the snapshot is written back to the frame, even if the value stored on the frame had changed since the snapshot was taken).

By making frame.f_locals return a view on the underlying frame, these problems go away. frame.f_locals is always in sync with the frame because it is a view of it, not a copy of it.

Making the locals() builtin return independent snapshots

PEP 558 considered three potential options for standardising the behavior of the locals() builtin in optimized scopes:

  • retain the historical behaviour of having each call to locals() on a given frame update a single shared snapshot of the local variables
  • make locals() return write-through proxy instances (similar to frame.f_locals)
  • make locals() return genuinely independent snapshots so that attempts to change the values of local variables via exec() would be consistently ignored rather than being accepted in some circumstances

The last option was chosen as the one which could most easily be explained in the language reference, and memorised by users:

  • the locals() builtin gives an instantaneous snapshot of the local variables in optimized scopes, and read/write access in other scopes; and
  • frame.f_locals gives read/write access to the local variables in all scopes, including optimized scopes

This approach allows the intent of a piece of code to be clearer than it would be if both APIs granted full read/write access in optimized scopes, even when write access wasn’t needed or desired. For additional details on this design decision, refer to PEP 558, especially the Motivation section and Additional considerations for eval() and exec() in optimized scopes.

This approach is not without its drawbacks, which are covered in the Backwards Compatibility section below.

Specification

Python API

The frame.f_locals attribute

For module and class scopes (including exec() and eval() invocations), frame.f_locals is a direct reference to the local variable namespace used in code execution.

For function scopes (and other optimized scopes) it will be an instance of a new write-through proxy type that can directly modify the optimized local variable storage array in the underlying frame, as well as the contents of any cell references to non-local variables.

The view objects fully implement the collections.abc.Mapping interface, and also implement the following mutable mapping operations:

  • using assignment to add new key/value pairs
  • using assignment to update the value associated with a key
  • conditional assignment via the setdefault() method
  • bulk updates via the update() method

Views of different frames compare unequal even if they have the same contents.

All writes to the f_locals mapping will be immediately visible in the underlying variables. All changes to the underlying variables will be immediately visible in the mapping.

The f_locals object will be a full mapping, and can have arbitrary key-value pairs added to it. New names added via the proxies will be stored in a dedicated shared dictionary stored on the underlying frame object (so all proxy instances for a given frame will be able to access any names added this way).

Extra keys (which do not correspond to local variables on the underlying frame) may be removed as usual with del statements or the pop() method.

Using del, or the pop() method, to remove keys that correspond to local variables on the underlying frame is NOT supported, and attempting to do so will raise ValueError. Local variables can only be set to None (or some other value) via the proxy, they cannot be unbound completely.

The clear() method is NOT implemented on the write-through proxies, as it is unclear how it should handle the inability to delete entries corresponding to local variables.

To maintain backwards compatibility, proxy APIs that need to produce a new mapping (such as copy()) will produce regular builtin dict instances, rather than write-through proxy instances.

To avoid introducing a circular reference between frame objects and the write-through proxies, each access to frame.f_locals returns a new write-through proxy instance.

The locals() builtin

locals() will be defined as:

def locals():
    frame = sys._getframe(1)
    f_locals = frame.f_locals
    if frame._is_optimized(): # Not an actual frame method
        f_locals = dict(f_locals)
    return f_locals

For module and class scopes (including exec() and eval() invocations), locals() continues to return a direct reference to the local variable namespace used in code execution (which is also the same value reported by frame.f_locals).

In optimized scopes, each call to locals() will produce an independent snapshot of the local variables.

The eval() and exec() builtins

Because this PEP changes the behavior of locals(), the behavior of eval() and exec() also changes.

Assuming a function _eval() which performs the job of eval() with explicit namespace arguments, eval() can be defined as follows:

FrameProxyType = type((lambda: sys._getframe().f_locals)())

def eval(expression, /, globals=None, locals=None):
    if globals is None:
        # No globals -> use calling frame's globals
        _calling_frame = sys._getframe(1)
        globals = _calling_frame.f_globals
        if locals is None:
            # No globals or locals -> use calling frame's locals
            locals = _calling_frame.f_locals
            if isinstance(locals, FrameProxyType):
                # Align with locals() builtin in optimized frame
                locals = dict(locals)
    elif locals is None:
        # Globals but no locals -> use same namespace for both
        locals = globals
    return _eval(expression, globals, locals)

The specified argument handling for exec() is similarly updated.

(In Python 3.12 and earlier, it was not possible to provide locals to eval() or exec() without also providing globals as these were previously positional-only arguments. Independently of this PEP, Python 3.13 updated these builtins to accept keyword arguments)

C API

Additions to the PyEval C API

Three new C-API functions will be added:

PyObject *PyEval_GetFrameLocals(void)
PyObject *PyEval_GetFrameGlobals(void)
PyObject *PyEval_GetFrameBuiltins(void)

PyEval_GetFrameLocals() is equivalent to: locals(). PyEval_GetFrameGlobals() is equivalent to: globals().

All of these functions will return a new reference.

PyFrame_GetLocals C API

The existing PyFrame_GetLocals(f) C API is equivalent to f.f_locals. Its return value will be as described above for accessing f.f_locals.

This function returns a new reference, so it is able to accommodate the creation of a new write-through proxy instance on each call in an optimized scope.

Deprecated C APIs

The following C API functions will be deprecated, as they return borrowed references:

PyEval_GetLocals()
PyEval_GetGlobals()
PyEval_GetBuiltins()

The following functions (which return new references) should be used instead:

PyEval_GetFrameLocals()
PyEval_GetFrameGlobals()
PyEval_GetFrameBuiltins()

The following C API functions will become no-ops, and will be deprecated without replacement:

PyFrame_FastToLocalsWithError()
PyFrame_FastToLocals()
PyFrame_LocalsToFast()

All of the deprecated functions will be marked as deprecated in the Python 3.13 documentation.

Of these functions, only PyEval_GetLocals() poses any significant maintenance burden. Accordingly, calls to PyEval_GetLocals() will emit DeprecationWarning in Python 3.14, with a target removal date of Python 3.16 (two releases after Python 3.14). Alternatives are recommended as described in PyEval_GetLocals compatibility.

Summary of Changes

This section summarises how the specified behaviour in Python 3.13 and later differs from the historical behaviour in Python 3.12 and earlier versions.

Python API changes

frame.f_locals changes

Consider the following example:

def l():
    "Get the locals of caller"
    return sys._getframe(1).f_locals

def test():
    if 0: y = 1 # Make 'y' a local variable
    x = 1
    l()['x'] = 2
    l()['y'] = 4
    l()['z'] = 5
    y
    print(locals(), x)

Given the changes in this PEP, test() will print {'x': 2, 'y': 4, 'z': 5} 2.

In Python 3.12, this example will fail with an UnboundLocalError, as the definition of y by l()['y'] = 4 is lost.

If the second-to-last line were changed from y to z, this will still raise NameError, as it does in Python 3.12. Keys added to frame.f_locals that are not lexically local variables remain visible in frame.f_locals, but do not dynamically become local variables.

locals() changes

Consider the following example:

def f():
    exec("x = 1")
    print(locals().get("x"))
f()

Given the changes in this PEP, this will always print None (regardless of whether x is a defined local variable in the function), as the explicit call to locals() produces a distinct snapshot from the one implicitly used in the exec() call.

In Python 3.12, the exact example shown would print 1, but seemingly unrelated changes to the definition of the function involved could make it print None instead (Additional considerations for eval() and exec() in optimized scopes in PEP 558 goes into more detail on that topic).

eval() and exec() changes

The primary change affecting eval() and exec() is shown in the “locals() changes” example: repeatedly accessing locals() in an optimized scope will no longer implicitly share a common underlying namespace.

C API changes

PyFrame_GetLocals change

PyFrame_GetLocals can already return arbitrary mappings in Python 3.12, as exec() and eval() accept arbitrary mappings as their locals argument, and metaclasses may return arbitrary mappings from their __prepare__ methods.

Returning a frame locals proxy in optimized scopes just adds another case where something other than a builtin dictionary will be returned.

PyEval_GetLocals change

The semantics of PyEval_GetLocals() are technically unchanged, but they do change in practice as the dictionary cached on optimized frames is no longer shared with other mechanisms for accessing the frame locals (locals() builtin, PyFrame_GetLocals function, frame f_locals attributes).

Backwards Compatibility

Python API compatibility

The implementation used in versions up to and including Python 3.12 has many corner cases and oddities. Code that works around those may need to be changed. Code that uses locals() for simple templating, or print debugging, will continue to work correctly. Debuggers and other tools that use f_locals to modify local variables, will now work correctly, even in the presence of threaded code, coroutines and generators.

frame.f_locals compatibility

Although f.f_locals behaves as if it were the namespace of the function, there will be some observable differences. For example, f.f_locals is f.f_locals will be False for optimized frames, as each access to the attribute produces a new write-through proxy instance.

However f.f_locals == f.f_locals will be True, and all changes to the underlying variables, by any means, including the addition of new variable names as mapping keys, will always be visible.

locals() compatibility

locals() is locals() will be False for optimized frames, so code like the following will raise KeyError instead of returning 1:

def f():
    locals()["x"] = 1
    return locals()["x"]

To continue working, such code will need to explicitly store the namespace to be modified in a local variable, rather than relying on the previous implicit caching on the frame object:

def f():
    ns = {}
    ns["x"] = 1
    return ns["x"]

While this technically isn’t a formal backwards compatibility break (since the behaviour of writing back to locals() was explicitly documented as undefined), there is definitely some code that relies on the existing behaviour. Accordingly, the updated behaviour will be explicitly noted in the documentation as a change and it will be covered in the Python 3.13 porting guide.

To work with a copy of locals() in optimized scopes on all versions without making redundant copies on Python 3.13+, users will need to define a version-dependent helper function that only makes an explicit copy on Python versions prior to Python 3.13:

if sys.version_info >= (3, 13):
    def _ensure_func_snapshot(d):
        return d # 3.13+ locals() already returns a snapshot
else:
    def _ensure_func_snapshot(d):
        return dict(d) # Create snapshot on older versions

def f():
    ns = _ensure_func_snapshot(locals())
    ns["x"] = 1
    return ns

In other scopes, locals().copy() can continue to be called unconditionally without introducing any redundant copies.

Impact on exec() and eval()

Even though this PEP does not modify exec() or eval() directly, the semantic change to locals() impacts the behavior of exec() and eval() as they default to running code in the calling namespace.

This poses a potential compatibility issue for some code, as with the previous implementation that returns the same dict when locals() is called multiple times in function scope, the following code usually worked due to the implicitly shared local variable namespace:

def f():
    exec('a = 0')  # equivalent to exec('a = 0', globals(), locals())
    exec('print(a)')  # equivalent to exec('print(a)', globals(), locals())
    print(locals())  # {'a': 0}
    # However, print(a) will not work here
f()

With the semantic changes to locals() in this PEP, the exec('print(a)')' call will fail with NameError, and print(locals()) will report an empty dictionary, as each line will be using its own distinct snapshot of the local variables rather than implicitly sharing a single cached snapshot stored on the frame object.

A shared namespace across exec() calls can still be obtained by using explicit namespaces rather than relying on the previously implicitly shared frame namespace:

def f():
    ns = {}
    exec('a = 0', locals=ns)
    exec('print(a)', locals=ns)  # 0
f()

You can even reliably change the variables in the local scope by explicitly using frame.f_locals, which was not possible before (even using ctypes to invoke PyFrame_LocalsToFast was subject to the state inconsistency problems discussed elsewhere in this PEP):

def f():
    a = None
    exec('a = 0', locals=sys._getframe().f_locals)
    print(a)  # 0
f()

The behavior of exec() and eval() for module and class scopes (including nested invocations) is not changed, as the behaviour of locals() in those scopes is not changing.

Impact on other code execution APIs in the standard library

pdb and bdb use the frame.f_locals API, and hence will be able to reliably update local variables even in optimized frames. Implementing this PEP will resolve several longstanding bugs in these modules relating to threads, generators, coroutines, and other mechanisms that allow concurrent code execution while the debugger is active.

Other code execution APIs in the standard library (such as the code module) do not implicitly access locals() or frame.f_locals, but the behaviour of explicitly passing these namespaces will change as described in the rest of this PEP (passing locals() in optimized scopes will no longer implicitly share the code execution namespace across calls, passing frame.f_locals in optimized scopes will allow reliable modification of local variables and nonlocal cell references).

C API compatibility

PyEval_GetLocals compatibility

PyEval_GetLocals() has never historically distinguished between whether it was emulating locals() or sys._getframe().f_locals at the Python level, as they all returned references to the same shared cache of the local variable bindings.

With this PEP, locals() changes to return independent snapshots on each call for optimized frames, and frame.f_locals (along with PyFrame_GetLocals) changes to return new write-through proxy instances.

Because PyEval_GetLocals() returns a borrowed reference, it isn’t possible to update its semantics to align with either of those alternatives, leaving it as the only remaining API that requires a shared cache dictionary stored on the frame object.

While this technically leaves the semantics of the function unchanged, it no longer allows extra dict entries to be made visible to users of the other APIs, as those APIs are no longer accessing the same underlying cache dictionary.

When PyEval_GetLocals() is being used as an equivalent to the Python locals() builtin, PyEval_GetFrameLocals() should be used instead.

This code:

locals = PyEval_GetLocals();
if (locals == NULL) {
    goto error_handler;
}
Py_INCREF(locals);

should be replaced with:

// Equivalent to "locals()" in Python code
locals = PyEval_GetFrameLocals();
if (locals == NULL) {
    goto error_handler;
}

When PyEval_GetLocals() is being used as an equivalent to calling sys._getframe().f_locals in Python, it should be replaced by calling PyFrame_GetLocals() on the result of PyEval_GetFrame().

In these cases, the original code should be replaced with:

// Equivalent to "sys._getframe()" in Python code
frame = PyEval_GetFrame();
if (frame == NULL) {
    goto error_handler;
}
// Equivalent to "frame.f_locals" in Python code
locals = PyFrame_GetLocals(frame);
frame = NULL; // Minimise visibility of borrowed reference
if (locals == NULL) {
    goto error_handler;
}

Impact on PEP 709 inlined comprehensions

For inlined comprehensions within a function, locals() currently behaves the same inside or outside of the comprehension, and this will not change. The behavior of locals() inside functions will generally change as specified in the rest of this PEP.

For inlined comprehensions at module or class scope, calling locals() within the inlined comprehension returns a new dictionary for each call. This PEP will make locals() within a function also always return a new dictionary for each call, improving consistency; class or module scope inlined comprehensions will appear to behave as if the inlined comprehension is still a distinct function.

Implementation

Each read of frame.f_locals will create a new proxy object that gives the appearance of being the mapping of local (including cell and free) variable names to the values of those local variables.

A possible implementation is sketched out below. All attributes that start with an underscore are invisible and cannot be accessed directly. They serve only to illustrate the proposed design.

NULL: Object # NULL is a singleton representing the absence of a value.

class CodeType:

    _name_to_offset_mapping_impl: dict | NULL
    _cells: frozenset # Set of indexes of cell and free variables
    ...

    def __init__(self, ...):
        self._name_to_offset_mapping_impl = NULL
        self._variable_names = deduplicate(
            self.co_varnames + self.co_cellvars + self.co_freevars
        )
        ...

    @property
    def _name_to_offset_mapping(self):
        "Mapping of names to offsets in local variable array."
        if self._name_to_offset_mapping_impl is NULL:
            self._name_to_offset_mapping_impl = {
                name: index for (index, name) in enumerate(self._variable_names)
            }
        return self._name_to_offset_mapping_impl

class FrameType:

    _locals : array[Object] # The values of the local variables, items may be NULL.
    _extra_locals: dict | NULL # Dictionary for storing extra locals not in _locals.
    _locals_cache: FrameLocalsProxy | NULL # required to support PyEval_GetLocals()

    def __init__(self, ...):
        self._extra_locals = NULL
        self._locals_cache = NULL
        ...

    @property
    def f_locals(self):
        return FrameLocalsProxy(self)

class FrameLocalsProxy:
    "Implements collections.MutableMapping."

    __slots__ = ("_frame", )

    def __init__(self, frame:FrameType):
        self._frame = frame

    def __getitem__(self, name):
        f = self._frame
        co = f.f_code
        if name in co._name_to_offset_mapping:
            index = co._name_to_offset_mapping[name]
            val = f._locals[index]
            if val is NULL:
                raise KeyError(name)
            if index in co._cells
                val = val.cell_contents
                if val is NULL:
                    raise KeyError(name)
            return val
        else:
            if f._extra_locals is NULL:
                raise KeyError(name)
            return f._extra_locals[name]

    def __setitem__(self, name, value):
        f = self._frame
        co = f.f_code
        if name in co._name_to_offset_mapping:
            index = co._name_to_offset_mapping[name]
            kind = co._local_kinds[index]
            if index in co._cells
                cell = f._locals[index]
                cell.cell_contents = val
            else:
                f._locals[index] = val
        else:
            if f._extra_locals is NULL:
                f._extra_locals = {}
            f._extra_locals[name] = val

    def __iter__(self):
        f = self._frame
        co = f.f_code
        yield from iter(f._extra_locals)
        for index, name in enumerate(co._variable_names):
            val = f._locals[index]
            if val is NULL:
                continue
            if index in co._cells:
                val = val.cell_contents
                if val is NULL:
                    continue
            yield name

    def __contains__(self, item):
        f = self._frame
        if item in f._extra_locals:
            return True
        return item in co._variable_names

    def __len__(self):
        f = self._frame
        co = f.f_code
        res = 0
        for index, _ in enumerate(co._variable_names):
            val = f._locals[index]
            if val is NULL:
                continue
            if index in co._cells:
                if val.cell_contents is NULL:
                    continue
            res += 1
        return len(self._extra_locals) + res

C API

PyEval_GetLocals() will be implemented roughly as follows:

PyObject *PyEval_GetLocals(void) {
    PyFrameObject * = ...; // Get the current frame.
    if (frame->_locals_cache == NULL) {
        frame->_locals_cache = PyEval_GetFrameLocals();
    } else {
        PyDict_Update(frame->_locals_cache, PyFrame_GetLocals(frame));
    }
    return frame->_locals_cache;
}

As with all functions that return a borrowed reference, care must be taken to ensure that the reference is not used beyond the lifetime of the object.

Implementation Notes

When accepted, the PEP text suggested that PyEval_GetLocals would start returning a cached instance of the new write-through proxy, while the implementation sketch indicated it would continue to return a dictionary snapshot cached on the frame instance. This discrepancy was identified while implementing the PEP, and resolved by the Steering Council in favour of retaining the Python 3.12 behaviour of returning a dictionary snapshot cached on the frame instance. The PEP text has been updated accordingly.

During the discussions of the C API clarification, it also became apparent that the rationale behind locals() being updated to return independent snapshots in optimized scopes wasn’t clear, as it had been inherited from the original PEP 558 discussions rather than being independently covered in this PEP. The PEP text has been updated to better cover this change, with additional updates to the Specification and Backwards Compatibility sections to cover the impact on code execution APIs that default to executing code in the locals() namespace. Additional motivation and rationale details have also been added to PEP 558.

In 3.13.0, the write-through proxies did not allow deletion of even extra variables with del and pop(). This was subsequently reported as a compatibility regression, and resolved as now described in The frame.f_locals attribute.

Comparison with PEP 558

This PEP and PEP 558 shared a common goal: to make the semantics of locals() and frame.f_locals() intelligible, and their operation reliable.

The key difference between this PEP and PEP 558 is that PEP 558 attempted to store extra variables inside a full internal dictionary copy of the local variables in an effort to improve backwards compatibility with the legacy PyEval_GetLocals() API, whereas this PEP does not (it stores the extra local variables in a dedicated dictionary accessed solely via the new frame proxy objects, and copies them to the PyEval_GetLocals() shared dict only when requested).

PEP 558 did not specify exactly when that internal copy was updated, making the behavior of PEP 558 impossible to reason about in several cases where this PEP remains well specified.

PEP 558 also proposed the introduction of some additional Python scope introspection interfaces to the C API that would allow extension modules to more easily determine whether the currently active Python scope is optimized or not, and hence whether the C API’s locals() equivalent returns a direct reference to the frame’s local execution namespace or a shallow copy of the frame’s local variables and nonlocal cell references. Whether or not to add such introspection APIs is independent of the proposed changes to locals() and frame.f_locals and hence no such proposals have been included in this PEP.

PEP 558 was ultimately withdrawn in favour of this PEP.

Reference Implementation

The implementation is in development as a draft pull request on GitHub.


Source: https://github.com/python/peps/blob/main/peps/pep-0667.rst

Last modified: 2024-10-21 17:45:04 GMT