PEP 454 – Add a new tracemalloc module to trace Python memory allocations
- Author:
- Victor Stinner <vstinner at python.org>
- BDFL-Delegate:
- Charles-François Natali <cf.natali at gmail.com>
- Status:
- Final
- Type:
- Standards Track
- Created:
- 03-Sep-2013
- Python-Version:
- 3.4
- Resolution:
- Python-Dev message
Abstract
This PEP proposes to add a new tracemalloc
module to trace memory
blocks allocated by Python.
Rationale
Classic generic tools like Valgrind can get the C traceback where a
memory block was allocated. Using such tools to analyze Python memory
allocations does not help because most memory blocks are allocated in
the same C function, in PyMem_Malloc()
for example. Moreover, Python
has an allocator for small objects called “pymalloc” which keeps free
blocks for efficiency. This is not well handled by these tools.
There are debug tools dedicated to the Python language like Heapy
Pympler
and Meliae
which lists all alive objects using the
garbage collector module (functions like gc.get_objects()
,
gc.get_referrers()
and gc.get_referents()
), compute their size
(ex: using sys.getsizeof()
) and group objects by type. These tools
provide a better estimation of the memory usage of an application. They
are useful when most memory leaks are instances of the same type and
this type is only instantiated in a few functions. Problems arise when
the object type is very common like str
or tuple
, and it is hard
to identify where these objects are instantiated.
Finding reference cycles is also a difficult problem. There are different tools to draw a diagram of all references. These tools cannot be used on large applications with thousands of objects because the diagram is too huge to be analyzed manually.
Proposal
Using the customized allocation API from PEP 445, it becomes easy to set up a hook on Python memory allocators. A hook can inspect Python internals to retrieve Python tracebacks. The idea of getting the current traceback comes from the faulthandler module. The faulthandler dumps the traceback of all Python threads on a crash, here is the idea is to get the traceback of the current Python thread when a memory block is allocated by Python.
This PEP proposes to add a new tracemalloc
module, a debug tool
to trace memory blocks allocated by Python. The module provides the
following information:
- Traceback where an object was allocated
- Statistics on allocated memory blocks per filename and per line number: total size, number and average size of allocated memory blocks
- Computed differences between two snapshots to detect memory leaks
The API of the tracemalloc module is similar to the API of the faulthandler
module: enable()
/ start()
, disable()
/ stop()
and
is_enabled()
/ is_tracing()
functions, an environment variable
(PYTHONFAULTHANDLER
and PYTHONTRACEMALLOC
), and a -X
command line
option (-X faulthandler
and -X tracemalloc
). See the documentation of
the faulthandler module.
The idea of tracing memory allocations is not new. It was first implemented in the PySizer project in 2005. PySizer was implemented differently: the traceback was stored in frame objects and some Python types were linked the trace with the name of object type. PySizer patch on CPython adds an overhead on performances and memory footprint, even if the PySizer was not used. tracemalloc attaches a traceback to the underlying layer, to memory blocks, and has no overhead when the module is not tracing memory allocations.
The tracemalloc module has been written for CPython. Other implementations of Python may not be able to provide it.
API
To trace most memory blocks allocated by Python, the module should be
started as early as possible by setting the PYTHONTRACEMALLOC
environment variable to 1
, or by using -X tracemalloc
command
line option. The tracemalloc.start()
function can be called at
runtime to start tracing Python memory allocations.
By default, a trace of an allocated memory block only stores the most
recent frame (1 frame). To store 25 frames at startup: set the
PYTHONTRACEMALLOC
environment variable to 25
, or use the -X
tracemalloc=25
command line option. The set_traceback_limit()
function can be used at runtime to set the limit.
Functions
clear_traces()
function:
Clear traces of memory blocks allocated by Python.See also
stop()
.
get_object_traceback(obj)
function:
Get the traceback where the Python object obj was allocated. Return aTraceback
instance, orNone
if thetracemalloc
module is not tracing memory allocations or did not trace the allocation of the object.See also
gc.get_referrers()
andsys.getsizeof()
functions.
get_traceback_limit()
function:
Get the maximum number of frames stored in the traceback of a trace.The
tracemalloc
module must be tracing memory allocations to get the limit, otherwise an exception is raised.The limit is set by the
start()
function.
get_traced_memory()
function:
Get the current size and maximum size of memory blocks traced by thetracemalloc
module as a tuple:(size: int, max_size: int)
.
get_tracemalloc_memory()
function:
Get the memory usage in bytes of thetracemalloc
module used to store traces of memory blocks. Return anint
.
is_tracing()
function:
True
if thetracemalloc
module is tracing Python memory allocations,False
otherwise.See also
start()
andstop()
functions.
start(nframe: int=1)
function:
Start tracing Python memory allocations: install hooks on Python memory allocators. Collected tracebacks of traces will be limited to nframe frames. By default, a trace of a memory block only stores the most recent frame: the limit is1
. nframe must be greater or equal to1
.Storing more than
1
frame is only useful to compute statistics grouped by'traceback'
or to compute cumulative statistics: see theSnapshot.compare_to()
andSnapshot.statistics()
methods.Storing more frames increases the memory and CPU overhead of the
tracemalloc
module. Use theget_tracemalloc_memory()
function to measure how much memory is used by thetracemalloc
module.The
PYTHONTRACEMALLOC
environment variable (PYTHONTRACEMALLOC=NFRAME
) and the-X
tracemalloc=NFRAME
command line option can be used to start tracing at startup.See also
stop()
,is_tracing()
andget_traceback_limit()
functions.
stop()
function:
Stop tracing Python memory allocations: uninstall hooks on Python memory allocators. Clear also traces of memory blocks allocated by PythonCall
take_snapshot()
function to take a snapshot of traces before clearing them.See also
start()
andis_tracing()
functions.
take_snapshot()
function:
Take a snapshot of traces of memory blocks allocated by Python. Return a newSnapshot
instance.The snapshot does not include memory blocks allocated before the
tracemalloc
module started to trace memory allocations.Tracebacks of traces are limited to
get_traceback_limit()
frames. Use the nframe parameter of thestart()
function to store more frames.The
tracemalloc
module must be tracing memory allocations to take a snapshot, see thestart()
function.See also the
get_object_traceback()
function.
Filter
Filter(inclusive: bool, filename_pattern: str, lineno: int=None, all_frames: bool=False)
class:
Filter on traces of memory blocks.See the
fnmatch.fnmatch()
function for the syntax of filename_pattern. The'.pyc'
and'.pyo'
file extensions are replaced with'.py'
.Examples:
Filter(True, subprocess.__file__)
only includes traces of thesubprocess
moduleFilter(False, tracemalloc.__file__)
excludes traces of thetracemalloc
moduleFilter(False, "<unknown>")
excludes empty tracebacks
inclusive
attribute:
If inclusive isTrue
(include), only trace memory blocks allocated in a file with a name matchingfilename_pattern
at line numberlineno
.If inclusive is
False
(exclude), ignore memory blocks allocated in a file with a name matchingfilename_pattern
at line numberlineno
.
lineno
attribute:
Line number (int
) of the filter. If lineno isNone
, the filter matches any line number.
filename_pattern
attribute:
Filename pattern of the filter (str
).
all_frames
attribute:
If all_frames isTrue
, all frames of the traceback are checked. If all_frames isFalse
, only the most recent frame is checked.This attribute is ignored if the traceback limit is less than
2
. See theget_traceback_limit()
function andSnapshot.traceback_limit
attribute.
Frame
Frame
class:
Frame of a traceback.The
Traceback
class is a sequence ofFrame
instances.
filename
attribute:
Filename (str
).
lineno
attribute:
Line number (int
).
Snapshot
Snapshot
class:
Snapshot of traces of memory blocks allocated by Python.The
take_snapshot()
function creates a snapshot instance.
compare_to(old_snapshot: Snapshot, group_by: str, cumulative: bool=False)
method:
Compute the differences with an old snapshot. Get statistics as a sorted list ofStatisticDiff
instances grouped by group_by.See the
statistics()
method for group_by and cumulative parameters.The result is sorted from the biggest to the smallest by: absolute value of
StatisticDiff.size_diff
,StatisticDiff.size
, absolute value ofStatisticDiff.count_diff
,Statistic.count
and then byStatisticDiff.traceback
.
dump(filename)
method:
Write the snapshot into a file.Use
load()
to reload the snapshot.
filter_traces(filters)
method:
Create a newSnapshot
instance with a filteredtraces
sequence, filters is a list ofFilter
instances. If filters is an empty list, return a newSnapshot
instance with a copy of the traces.All inclusive filters are applied at once, a trace is ignored if no inclusive filters match it. A trace is ignored if at least one exclusive filter matches it.
load(filename)
classmethod:
Load a snapshot from a file.See also
dump()
.
statistics(group_by: str, cumulative: bool=False)
method:
Get statistics as a sorted list ofStatistic
instances grouped by group_by:
group_by description 'filename'
filename 'lineno'
filename and line number 'traceback'
traceback If cumulative is
True
, cumulate size and count of memory blocks of all frames of the traceback of a trace, not only the most recent frame. The cumulative mode can only be used with group_by equals to'filename'
and'lineno'
andtraceback_limit
greater than1
.The result is sorted from the biggest to the smallest by:
Statistic.size
,Statistic.count
and then byStatistic.traceback
.
traceback_limit
attribute:
Maximum number of frames stored in the traceback oftraces
: result of theget_traceback_limit()
when the snapshot was taken.
traces
attribute:
Traces of all memory blocks allocated by Python: sequence ofTrace
instances.The sequence has an undefined order. Use the
Snapshot.statistics()
method to get a sorted list of statistics.
Statistic
Statistic
class:
Statistic on memory allocations.
Snapshot.statistics()
returns a list ofStatistic
instances.See also the
StatisticDiff
class.
count
attribute:
Number of memory blocks (int
).
size
attribute:
Total size of memory blocks in bytes (int
).
traceback
attribute:
Traceback where the memory block was allocated,Traceback
instance.
StatisticDiff
StatisticDiff
class:
Statistic difference on memory allocations between an old and a newSnapshot
instance.
Snapshot.compare_to()
returns a list ofStatisticDiff
instances. See also theStatistic
class.
count
attribute:
Number of memory blocks in the new snapshot (int
):0
if the memory blocks have been released in the new snapshot.
count_diff
attribute:
Difference of number of memory blocks between the old and the new snapshots (int
):0
if the memory blocks have been allocated in the new snapshot.
size
attribute:
Total size of memory blocks in bytes in the new snapshot (int
):0
if the memory blocks have been released in the new snapshot.
size_diff
attribute:
Difference of total size of memory blocks in bytes between the old and the new snapshots (int
):0
if the memory blocks have been allocated in the new snapshot.
traceback
attribute:
Traceback where the memory blocks were allocated,Traceback
instance.
Trace
Trace
class:
Trace of a memory block.The
Snapshot.traces
attribute is a sequence ofTrace
instances.
size
attribute:
Size of the memory block in bytes (int
).
traceback
attribute:
Traceback where the memory block was allocated,Traceback
instance.
Traceback
Traceback
class:
Sequence ofFrame
instances sorted from the most recent frame to the oldest frame.A traceback contains at least
1
frame. If thetracemalloc
module failed to get a frame, the filename"<unknown>"
at line number0
is used.When a snapshot is taken, tracebacks of traces are limited to
get_traceback_limit()
frames. See thetake_snapshot()
function.The
Trace.traceback
attribute is an instance ofTraceback
instance.
Rejected Alternatives
Log calls to the memory allocator
A different approach is to log calls to malloc()
, realloc()
and
free()
functions. Calls can be logged into a file or send to another
computer through the network. Example of a log entry: name of the
function, size of the memory block, address of the memory block, Python
traceback where the allocation occurred, timestamp.
Logs cannot be used directly, getting the current status of the memory
requires to parse previous logs. For example, it is not possible to get
directly the traceback of a Python object, like
get_object_traceback(obj)
does with traces.
Python uses objects with a very short lifetime and so makes an extensive
use of memory allocators. It has an allocator optimized for small
objects (less than 512 bytes) with a short lifetime. For example, the
Python test suites calls malloc()
, realloc()
or free()
270,000 times per second in average. If the size of log entry is 32
bytes, logging produces 8.2 MB per second or 29.0 GB per hour.
The alternative was rejected because it is less efficient and has less features. Parsing logs in a different process or a different computer is slower than maintaining traces on allocated memory blocks in the same process.
Prior Work
- Python Memory Validator (2005-2013): commercial Python memory validator developed by Software Verification. It uses the Python Reflection API.
- PySizer: Google Summer of Code 2005 project by Nick Smallbone.
- Heapy (2006-2013): part of the Guppy-PE project written by Sverker Nilsson.
- Draft PEP: Support Tracking Low-Level Memory Usage in CPython (Brett Canon, 2006)
- Muppy: project developed in 2008 by Robert Schuppenies.
- asizeof: a pure Python module to estimate the size of objects by Jean Brouwers (2008).
- Heapmonitor: It provides facilities to size individual objects and can track all objects of certain classes. It was developed in 2008 by Ludwig Haehne.
- Pympler (2008-2011): project based on asizeof, muppy and HeapMonitor
- objgraph (2008-2012)
- Dozer: WSGI Middleware version of the CherryPy memory leak debugger, written by Marius Gedminas (2008-2013)
- Meliae: Python Memory Usage Analyzer developed by John A Meinel since 2009
- gdb-heap: gdb script written in Python by Dave Malcolm (2010-2011) to analyze the usage of the heap memory
- memory_profiler: written by Fabian Pedregosa (2011-2013)
- caulk: written by Ben Timby in 2012
See also Pympler Related Work.
Links
tracemalloc:
Copyright
This document has been placed in the public domain.
Source: https://github.com/python/peps/blob/main/peps/pep-0454.rst
Last modified: 2023-09-09 17:39:29 GMT