PEP 667 – Consistent views of namespaces
- Author:
- Mark Shannon <mark at hotpy.org>, Tian Gao <gaogaotiantian at hotmail.com>
- Discussions-To:
- Discourse thread
- Status:
- Final
- Type:
- Standards Track
- Created:
- 30-Jul-2021
- Python-Version:
- 3.13
- Post-History:
- 20-Aug-2021, 22-Feb-2024
- Resolution:
- 25-Apr-2024
Abstract
In early versions of Python all namespaces, whether in functions, classes or modules, were all implemented the same way: as a dictionary.
For performance reasons, the implementation of function namespaces was
changed. Unfortunately this meant that accessing these namespaces through
locals()
and frame.f_locals
ceased to be consistent and some
odd bugs crept in over the years as threads, generators and coroutines
were added.
This PEP proposes making these namespaces consistent once more.
Modifications to frame.f_locals
will always be visible in
the underlying variables. Modifications to local variables will
immediately be visible in frame.f_locals
, and they will be
consistent regardless of threading or coroutines.
The locals()
function will act the same as it does now for class
and modules scopes. For function scopes it will return an instantaneous
snapshot of the underlying frame.f_locals
rather than implicitly
refreshing a single shared dictionary cached on the frame object.
Motivation
The implementation of locals()
and frame.f_locals
in releases up to and
including Python 3.12 is slow, inconsistent and buggy.
We want to make it faster, consistent, and most importantly fix the bugs.
For example, when attempting to manipulate local variables via frame objects:
class C:
x = 1
sys._getframe().f_locals['x'] = 2
print(x)
prints 2
, but:
def f():
x = 1
sys._getframe().f_locals['x'] = 2
print(x)
f()
prints 1
.
This is inconsistent, and confusing. Worse than that, the Python 3.12 behavior can result in strange bugs.
With this PEP both examples would print 2
as the function level
change would be written directly to the optimized local variables in
the frame rather than to a cached dictionary snapshot.
There are no compensating advantages for the Python 3.12 behavior; it is unreliable and slow.
The locals()
builtin has its own undesirable behaviours. Refer to PEP 558
for additional details on those concerns.
Rationale
Making the frame.f_locals
attribute a write-through proxy
The Python 3.12 implementation of frame.f_locals
returns a dictionary
that is created on the fly from the array of local variables. The
PyFrame_LocalsToFast()
C API is then called by debuggers and trace
functions that want to write their changes back to the array (until
Python 3.11, this API was called implicitly after every trace function
invocation rather than being called explicitly by the trace functions).
This can result in the array and dictionary getting out of sync with
each other. Writes to the f_locals
frame attribute may not show up as
modifications to local variables if PyFrame_LocalsToFast()
is never
called. Writes to local variables can get lost if a dictionary snapshot
created before the variables were modified is written back to the frame
(since every known variable stored in the snapshot is written back to
the frame, even if the value stored on the frame had changed since the
snapshot was taken).
By making frame.f_locals
return a view on the
underlying frame, these problems go away. frame.f_locals
is always in
sync with the frame because it is a view of it, not a copy of it.
Making the locals()
builtin return independent snapshots
PEP 558 considered three potential options for standardising the behavior of the
locals()
builtin in optimized scopes:
- retain the historical behaviour of having each call to
locals()
on a given frame update a single shared snapshot of the local variables - make
locals()
return write-through proxy instances (similar toframe.f_locals
) - make
locals()
return genuinely independent snapshots so that attempts to change the values of local variables viaexec()
would be consistently ignored rather than being accepted in some circumstances
The last option was chosen as the one which could most easily be explained in the language reference, and memorised by users:
- the
locals()
builtin gives an instantaneous snapshot of the local variables in optimized scopes, and read/write access in other scopes; and frame.f_locals
gives read/write access to the local variables in all scopes, including optimized scopes
This approach allows the intent of a piece of code to be clearer than it would be if both APIs granted full read/write access in optimized scopes, even when write access wasn’t needed or desired. For additional details on this design decision, refer to PEP 558, especially the Motivation section and Additional considerations for eval() and exec() in optimized scopes.
This approach is not without its drawbacks, which are covered in the Backwards Compatibility section below.
Specification
Python API
The frame.f_locals
attribute
For module and class scopes (including exec()
and eval()
invocations), frame.f_locals
is a direct
reference to the local variable namespace used in code execution.
For function scopes (and other optimized scopes) it will be an instance of a new write-through proxy type that can directly modify the optimized local variable storage array in the underlying frame, as well as the contents of any cell references to non-local variables.
The view objects fully implement the collections.abc.Mapping
interface,
and also implement the following mutable mapping operations:
- using assignment to add new key/value pairs
- using assignment to update the value associated with a key
- conditional assignment via the
setdefault()
method - bulk updates via the
update()
method
Views of different frames compare unequal even if they have the same contents.
All writes to the f_locals
mapping will be immediately visible
in the underlying variables. All changes to the underlying variables
will be immediately visible in the mapping.
The f_locals
object will be a full mapping, and can have arbitrary
key-value pairs added to it. New names added via the proxies
will be stored in a dedicated shared dictionary stored on the
underlying frame object (so all proxy instances for a given frame
will be able to access any names added this way).
Extra keys (which do not correspond to local variables on the underlying
frame) may be removed as usual with del
statements or the pop()
method.
Using del
, or the pop()
method, to remove keys that correspond to local
variables on the underlying frame is NOT supported, and attempting to do so
will raise ValueError
.
Local variables can only be set to None
(or some other value) via the proxy,
they cannot be unbound completely.
The clear()
method is NOT implemented on the write-through proxies, as it
is unclear how it should handle the inability to delete entries corresponding
to local variables.
To maintain backwards compatibility, proxy APIs that need to produce a
new mapping (such as copy()
) will produce regular builtin dict
instances, rather than write-through proxy instances.
To avoid introducing a circular reference between frame objects and the
write-through proxies, each access to frame.f_locals
returns a new
write-through proxy instance.
The locals()
builtin
locals()
will be defined as:
def locals():
frame = sys._getframe(1)
f_locals = frame.f_locals
if frame._is_optimized(): # Not an actual frame method
f_locals = dict(f_locals)
return f_locals
For module and class scopes (including exec()
and eval()
invocations), locals()
continues to return a direct
reference to the local variable namespace used in code execution
(which is also the same value reported by frame.f_locals
).
In optimized scopes,
each call to locals()
will produce an independent
snapshot of the local variables.
The eval()
and exec()
builtins
Because this PEP changes the behavior of locals()
, the
behavior of eval()
and exec()
also changes.
Assuming a function _eval()
which performs the job of
eval()
with explicit namespace arguments, eval()
can be defined as follows:
FrameProxyType = type((lambda: sys._getframe().f_locals)())
def eval(expression, /, globals=None, locals=None):
if globals is None:
# No globals -> use calling frame's globals
_calling_frame = sys._getframe(1)
globals = _calling_frame.f_globals
if locals is None:
# No globals or locals -> use calling frame's locals
locals = _calling_frame.f_locals
if isinstance(locals, FrameProxyType):
# Align with locals() builtin in optimized frame
locals = dict(locals)
elif locals is None:
# Globals but no locals -> use same namespace for both
locals = globals
return _eval(expression, globals, locals)
The specified argument handling for exec()
is similarly updated.
(In Python 3.12 and earlier, it was not possible to provide locals
to eval()
or exec()
without also providing globals
as these
were previously positional-only arguments. Independently of this
PEP, Python 3.13 updated these builtins to accept keyword arguments)
C API
Additions to the PyEval
C API
Three new C-API functions will be added:
PyObject *PyEval_GetFrameLocals(void)
PyObject *PyEval_GetFrameGlobals(void)
PyObject *PyEval_GetFrameBuiltins(void)
PyEval_GetFrameLocals()
is equivalent to: locals()
.
PyEval_GetFrameGlobals()
is equivalent to: globals()
.
All of these functions will return a new reference.
PyFrame_GetLocals
C API
The existing PyFrame_GetLocals(f)
C API is equivalent to f.f_locals
.
Its return value will be as described above for accessing f.f_locals
.
This function returns a new reference, so it is able to accommodate the creation of a new write-through proxy instance on each call in an optimized scope.
Deprecated C APIs
The following C API functions will be deprecated, as they return borrowed references:
PyEval_GetLocals()
PyEval_GetGlobals()
PyEval_GetBuiltins()
The following functions (which return new references) should be used instead:
PyEval_GetFrameLocals()
PyEval_GetFrameGlobals()
PyEval_GetFrameBuiltins()
The following C API functions will become no-ops, and will be deprecated without replacement:
PyFrame_FastToLocalsWithError()
PyFrame_FastToLocals()
PyFrame_LocalsToFast()
All of the deprecated functions will be marked as deprecated in the Python 3.13 documentation.
Of these functions, only PyEval_GetLocals()
poses any significant maintenance burden.
Accordingly, calls to PyEval_GetLocals()
will emit DeprecationWarning
in Python
3.14, with a target removal date of Python 3.16 (two releases after Python 3.14).
Alternatives are recommended as described in PyEval_GetLocals compatibility.
Summary of Changes
This section summarises how the specified behaviour in Python 3.13 and later differs from the historical behaviour in Python 3.12 and earlier versions.
Python API changes
frame.f_locals
changes
Consider the following example:
def l():
"Get the locals of caller"
return sys._getframe(1).f_locals
def test():
if 0: y = 1 # Make 'y' a local variable
x = 1
l()['x'] = 2
l()['y'] = 4
l()['z'] = 5
y
print(locals(), x)
Given the changes in this PEP,
test()
will print {'x': 2, 'y': 4, 'z': 5} 2
.
In Python 3.12, this example will fail with an UnboundLocalError
,
as the definition of y
by l()['y'] = 4
is lost.
If the second-to-last line were changed from y
to z
, this will still
raise NameError
, as it does in Python 3.12.
Keys added to frame.f_locals
that are not lexically local variables
remain visible in frame.f_locals
,
but do not dynamically become local variables.
locals()
changes
Consider the following example:
def f():
exec("x = 1")
print(locals().get("x"))
f()
Given the changes in this PEP, this will always print None
(regardless of whether x
is a defined local variable in the function),
as the explicit call to locals()
produces a distinct snapshot from
the one implicitly used in the exec()
call.
In Python 3.12, the exact example shown would print 1
, but seemingly
unrelated changes to the definition of the function involved could make
it print None
instead (Additional considerations for eval() and exec() in optimized scopes in PEP 558
goes into more detail on that topic).
eval()
and exec()
changes
The primary change affecting eval()
and exec()
is shown
in the “locals() changes” example: repeatedly
accessing locals()
in an optimized scope will no longer
implicitly share a common underlying namespace.
C API changes
PyFrame_GetLocals
change
PyFrame_GetLocals
can already return arbitrary mappings in Python 3.12,
as exec()
and eval()
accept arbitrary mappings as their locals
argument,
and metaclasses may return arbitrary mappings from their __prepare__
methods.
Returning a frame locals proxy in optimized scopes just adds another case where something other than a builtin dictionary will be returned.
PyEval_GetLocals
change
The semantics of PyEval_GetLocals()
are technically unchanged, but they do change in
practice as the dictionary cached on optimized frames is no longer shared with other
mechanisms for accessing the frame locals (locals()
builtin, PyFrame_GetLocals
function, frame f_locals
attributes).
Backwards Compatibility
Python API compatibility
The implementation used in versions up to and including Python 3.12 has many
corner cases and oddities. Code that works around those may need to be changed.
Code that uses locals()
for simple templating, or print debugging,
will continue to work correctly. Debuggers and other tools that use
f_locals
to modify local variables, will now work correctly,
even in the presence of threaded code, coroutines and generators.
frame.f_locals
compatibility
Although f.f_locals
behaves as if it were the namespace of the function,
there will be some observable differences.
For example, f.f_locals is f.f_locals
will be False
for optimized
frames, as each access to the attribute produces a new write-through proxy
instance.
However f.f_locals == f.f_locals
will be True
, and
all changes to the underlying variables, by any means, including the
addition of new variable names as mapping keys, will always be visible.
locals()
compatibility
locals() is locals()
will be False
for optimized frames, so
code like the following will raise KeyError
instead of returning
1
:
def f():
locals()["x"] = 1
return locals()["x"]
To continue working, such code will need to explicitly store the namespace to be modified in a local variable, rather than relying on the previous implicit caching on the frame object:
def f():
ns = {}
ns["x"] = 1
return ns["x"]
While this technically isn’t a formal backwards compatibility break
(since the behaviour of writing back to locals()
was explicitly
documented as undefined), there is definitely some code that relies
on the existing behaviour. Accordingly, the updated behaviour will
be explicitly noted in the documentation as a change and it will be
covered in the Python 3.13 porting guide.
To work with a copy of locals()
in optimized scopes on all
versions without making redundant copies on Python 3.13+, users
will need to define a version-dependent helper function that only
makes an explicit copy on Python versions prior to Python 3.13:
if sys.version_info >= (3, 13):
def _ensure_func_snapshot(d):
return d # 3.13+ locals() already returns a snapshot
else:
def _ensure_func_snapshot(d):
return dict(d) # Create snapshot on older versions
def f():
ns = _ensure_func_snapshot(locals())
ns["x"] = 1
return ns
In other scopes, locals().copy()
can continue to be called
unconditionally without introducing any redundant copies.
Impact on exec()
and eval()
Even though this PEP does not modify exec()
or eval()
directly,
the semantic change to locals()
impacts the behavior of exec()
and eval()
as they default to running code in the calling namespace.
This poses a potential compatibility issue for some code, as with the
previous implementation that returns the same dict when locals()
is called
multiple times in function scope, the following code usually worked due to
the implicitly shared local variable namespace:
def f():
exec('a = 0') # equivalent to exec('a = 0', globals(), locals())
exec('print(a)') # equivalent to exec('print(a)', globals(), locals())
print(locals()) # {'a': 0}
# However, print(a) will not work here
f()
With the semantic changes to locals()
in this PEP, the exec('print(a)')'
call
will fail with NameError
, and print(locals())
will report an empty dictionary, as
each line will be using its own distinct snapshot of the local variables rather than
implicitly sharing a single cached snapshot stored on the frame object.
A shared namespace across exec()
calls can still be obtained by using explicit
namespaces rather than relying on the previously implicitly shared frame namespace:
def f():
ns = {}
exec('a = 0', locals=ns)
exec('print(a)', locals=ns) # 0
f()
You can even reliably change the variables in the local scope by explicitly using
frame.f_locals
, which was not possible before (even using ctypes
to
invoke PyFrame_LocalsToFast
was subject to the state inconsistency problems
discussed elsewhere in this PEP):
def f():
a = None
exec('a = 0', locals=sys._getframe().f_locals)
print(a) # 0
f()
The behavior of exec()
and eval()
for module and class scopes (including
nested invocations) is not changed, as the behaviour of locals()
in those
scopes is not changing.
Impact on other code execution APIs in the standard library
pdb
and bdb
use the frame.f_locals
API, and hence will be able to
reliably update local variables even in optimized frames. Implementing this
PEP will resolve several longstanding bugs in these modules relating to threads,
generators, coroutines, and other mechanisms that allow concurrent code execution
while the debugger is active.
Other code execution APIs in the standard library (such as the code
module)
do not implicitly access locals()
or frame.f_locals
, but the behaviour
of explicitly passing these namespaces will change as described in the rest of
this PEP (passing locals()
in optimized scopes will no longer implicitly
share the code execution namespace across calls, passing frame.f_locals
in optimized scopes will allow reliable modification of local variables and
nonlocal cell references).
C API compatibility
PyEval_GetLocals
compatibility
PyEval_GetLocals()
has never historically distinguished between whether it was
emulating locals()
or sys._getframe().f_locals
at the Python level, as they all
returned references to the same shared cache of the local variable bindings.
With this PEP, locals()
changes to return independent snapshots on each call for
optimized frames, and frame.f_locals
(along with PyFrame_GetLocals
) changes to
return new write-through proxy instances.
Because PyEval_GetLocals()
returns a borrowed reference, it isn’t possible to update
its semantics to align with either of those alternatives, leaving it as the only remaining
API that requires a shared cache dictionary stored on the frame object.
While this technically leaves the semantics of the function unchanged, it no longer allows extra dict entries to be made visible to users of the other APIs, as those APIs are no longer accessing the same underlying cache dictionary.
When PyEval_GetLocals()
is being used as an equivalent to the Python locals()
builtin, PyEval_GetFrameLocals()
should be used instead.
This code:
locals = PyEval_GetLocals();
if (locals == NULL) {
goto error_handler;
}
Py_INCREF(locals);
should be replaced with:
// Equivalent to "locals()" in Python code
locals = PyEval_GetFrameLocals();
if (locals == NULL) {
goto error_handler;
}
When PyEval_GetLocals()
is being used as an equivalent to calling
sys._getframe().f_locals
in Python, it should be replaced by calling
PyFrame_GetLocals()
on the result of PyEval_GetFrame()
.
In these cases, the original code should be replaced with:
// Equivalent to "sys._getframe()" in Python code
frame = PyEval_GetFrame();
if (frame == NULL) {
goto error_handler;
}
// Equivalent to "frame.f_locals" in Python code
locals = PyFrame_GetLocals(frame);
frame = NULL; // Minimise visibility of borrowed reference
if (locals == NULL) {
goto error_handler;
}
Impact on PEP 709 inlined comprehensions
For inlined comprehensions within a function, locals()
currently behaves the
same inside or outside of the comprehension, and this will not change. The
behavior of locals()
inside functions will generally change as specified in
the rest of this PEP.
For inlined comprehensions at module or class scope, calling locals()
within
the inlined comprehension returns a new dictionary for each call. This PEP will
make locals()
within a function also always return a new dictionary for each
call, improving consistency; class or module scope inlined comprehensions will
appear to behave as if the inlined comprehension is still a distinct function.
Implementation
Each read of frame.f_locals
will create a new proxy object that gives
the appearance of being the mapping of local (including cell and free)
variable names to the values of those local variables.
A possible implementation is sketched out below. All attributes that start with an underscore are invisible and cannot be accessed directly. They serve only to illustrate the proposed design.
NULL: Object # NULL is a singleton representing the absence of a value.
class CodeType:
_name_to_offset_mapping_impl: dict | NULL
_cells: frozenset # Set of indexes of cell and free variables
...
def __init__(self, ...):
self._name_to_offset_mapping_impl = NULL
self._variable_names = deduplicate(
self.co_varnames + self.co_cellvars + self.co_freevars
)
...
@property
def _name_to_offset_mapping(self):
"Mapping of names to offsets in local variable array."
if self._name_to_offset_mapping_impl is NULL:
self._name_to_offset_mapping_impl = {
name: index for (index, name) in enumerate(self._variable_names)
}
return self._name_to_offset_mapping_impl
class FrameType:
_locals : array[Object] # The values of the local variables, items may be NULL.
_extra_locals: dict | NULL # Dictionary for storing extra locals not in _locals.
_locals_cache: FrameLocalsProxy | NULL # required to support PyEval_GetLocals()
def __init__(self, ...):
self._extra_locals = NULL
self._locals_cache = NULL
...
@property
def f_locals(self):
return FrameLocalsProxy(self)
class FrameLocalsProxy:
"Implements collections.MutableMapping."
__slots__ = ("_frame", )
def __init__(self, frame:FrameType):
self._frame = frame
def __getitem__(self, name):
f = self._frame
co = f.f_code
if name in co._name_to_offset_mapping:
index = co._name_to_offset_mapping[name]
val = f._locals[index]
if val is NULL:
raise KeyError(name)
if index in co._cells
val = val.cell_contents
if val is NULL:
raise KeyError(name)
return val
else:
if f._extra_locals is NULL:
raise KeyError(name)
return f._extra_locals[name]
def __setitem__(self, name, value):
f = self._frame
co = f.f_code
if name in co._name_to_offset_mapping:
index = co._name_to_offset_mapping[name]
kind = co._local_kinds[index]
if index in co._cells
cell = f._locals[index]
cell.cell_contents = val
else:
f._locals[index] = val
else:
if f._extra_locals is NULL:
f._extra_locals = {}
f._extra_locals[name] = val
def __iter__(self):
f = self._frame
co = f.f_code
yield from iter(f._extra_locals)
for index, name in enumerate(co._variable_names):
val = f._locals[index]
if val is NULL:
continue
if index in co._cells:
val = val.cell_contents
if val is NULL:
continue
yield name
def __contains__(self, item):
f = self._frame
if item in f._extra_locals:
return True
return item in co._variable_names
def __len__(self):
f = self._frame
co = f.f_code
res = 0
for index, _ in enumerate(co._variable_names):
val = f._locals[index]
if val is NULL:
continue
if index in co._cells:
if val.cell_contents is NULL:
continue
res += 1
return len(self._extra_locals) + res
C API
PyEval_GetLocals()
will be implemented roughly as follows:
PyObject *PyEval_GetLocals(void) {
PyFrameObject * = ...; // Get the current frame.
if (frame->_locals_cache == NULL) {
frame->_locals_cache = PyEval_GetFrameLocals();
} else {
PyDict_Update(frame->_locals_cache, PyFrame_GetLocals(frame));
}
return frame->_locals_cache;
}
As with all functions that return a borrowed reference, care must be taken to ensure that the reference is not used beyond the lifetime of the object.
Implementation Notes
When accepted, the PEP text suggested that PyEval_GetLocals
would start returning a
cached instance of the new write-through proxy, while the implementation sketch indicated
it would continue to return a dictionary snapshot cached on the frame instance. This
discrepancy was identified while implementing the PEP, and
resolved by the Steering Council
in favour of retaining the Python 3.12 behaviour of returning a dictionary snapshot
cached on the frame instance.
The PEP text has been updated accordingly.
During the discussions of the C API clarification, it also became apparent that the
rationale behind locals()
being updated to return independent snapshots in
optimized scopes wasn’t clear, as it had been inherited
from the original PEP 558 discussions rather than being independently covered in this
PEP. The PEP text has been updated to better cover this change, with additional updates
to the Specification and Backwards Compatibility sections to cover the impact on code
execution APIs that default to executing code in the locals()
namespace. Additional
motivation and rationale details have also been added to PEP 558.
In 3.13.0, the write-through proxies did not allow deletion of even extra variables
with del
and pop()
. This was subsequently reported as a
compatibility regression,
and resolved as now described
in The frame.f_locals attribute.
Comparison with PEP 558
This PEP and PEP 558 shared a common goal:
to make the semantics of locals()
and frame.f_locals()
intelligible, and their operation reliable.
The key difference between this PEP and PEP 558 is that
PEP 558 attempted to store extra variables inside a full
internal dictionary copy of the local variables in an effort
to improve backwards compatibility with the legacy
PyEval_GetLocals()
API, whereas this PEP does not (it stores
the extra local variables in a dedicated dictionary accessed
solely via the new frame proxy objects, and copies them to the
PyEval_GetLocals()
shared dict only when requested).
PEP 558 did not specify exactly when that internal copy was updated, making the behavior of PEP 558 impossible to reason about in several cases where this PEP remains well specified.
PEP 558 also proposed the introduction of some additional Python
scope introspection interfaces to the C API that would allow
extension modules to more easily determine whether the currently
active Python scope is optimized or not, and hence whether
the C API’s locals()
equivalent returns a direct reference
to the frame’s local execution namespace or a shallow copy of
the frame’s local variables and nonlocal cell references.
Whether or not to add such introspection APIs is independent
of the proposed changes to locals()
and frame.f_locals
and hence no such proposals have been included in this PEP.
PEP 558 was ultimately withdrawn in favour of this PEP.
Reference Implementation
The implementation is in development as a draft pull request on GitHub.
Copyright
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
Source: https://github.com/python/peps/blob/main/peps/pep-0667.rst
Last modified: 2024-10-27 07:11:46 GMT