PEP: 3134 Title: Exception Chaining and Embedded Tracebacks Version:
$Revision$ Last-Modified: $Date$ Author: Ka-Ping Yee Status: Final Type:
Standards Track Content-Type: text/x-rst Created: 12-May-2005
Python-Version: 3.0 Post-History:

Numbering Note

This PEP started its life as PEP 344. Since it is now targeted for
Python 3000, it has been moved into the 3xxx space.

Abstract

This PEP proposes three standard attributes on exception instances: the
__context__ attribute for implicitly chained exceptions, the __cause__
attribute for explicitly chained exceptions, and the __traceback__
attribute for the traceback. A new raise ... from statement sets the
__cause__ attribute.

Motivation

During the handling of one exception (exception A), it is possible that
another exception (exception B) may occur. In today's Python (version
2.4), if this happens, exception B is propagated outward and exception A
is lost. In order to debug the problem, it is useful to know about both
exceptions. The __context__ attribute retains this information
automatically.

Sometimes it can be useful for an exception handler to intentionally
re-raise an exception, either to provide extra information or to
translate an exception to another type. The __cause__ attribute provides
an explicit way to record the direct cause of an exception.

In today's Python implementation, exceptions are composed of three
parts: the type, the value, and the traceback. The sys module, exposes
the current exception in three parallel variables, exc_type, exc_value,
and exc_traceback, the sys.exc_info() function returns a tuple of these
three parts, and the raise statement has a three-argument form accepting
these three parts. Manipulating exceptions often requires passing these
three things in parallel, which can be tedious and error-prone.
Additionally, the except statement can only provide access to the value,
not the traceback. Adding the __traceback__ attribute to exception
values makes all the exception information accessible from a single
place.

History

Raymond Hettinger[1] raised the issue of masked exceptions on Python-Dev
in January 2003 and proposed a PyErr_FormatAppend() function that C
modules could use to augment the currently active exception with more
information. Brett Cannon[2] brought up chained exceptions again in June
2003, prompting a long discussion.

Greg Ewing[3] identified the case of an exception occurring in a finally
block during unwinding triggered by an original exception, as distinct
from the case of an exception occurring in an except block that is
handling the original exception.

Greg Ewing[4] and Guido van Rossum[5], and probably others, have
previously mentioned adding a traceback attribute to Exception
instances. This is noted in PEP 3000.

This PEP was motivated by yet another recent Python-Dev reposting of the
same ideas[6][7].

Rationale

The Python-Dev discussions revealed interest in exception chaining for
two quite different purposes. To handle the unexpected raising of a
secondary exception, the exception must be retained implicitly. To
support intentional translation of an exception, there must be a way to
chain exceptions explicitly. This PEP addresses both.

Several attribute names for chained exceptions have been suggested on
Python-Dev[8], including cause, antecedent, reason, original, chain,
chainedexc, exc_chain, excprev, previous, and precursor. For an
explicitly chained exception, this PEP suggests __cause__ because of its
specific meaning. For an implicitly chained exception, this PEP proposes
the name __context__ because the intended meaning is more specific than
temporal precedence but less specific than causation: an exception
occurs in the context of handling another exception.

This PEP suggests names with leading and trailing double-underscores for
these three attributes because they are set by the Python VM. Only in
very special cases should they be set by normal assignment.

This PEP handles exceptions that occur during except blocks and finally
blocks in the same way. Reading the traceback makes it clear where the
exceptions occurred, so additional mechanisms for distinguishing the two
cases would only add unnecessary complexity.

This PEP proposes that the outermost exception object (the one exposed
for matching by except clauses) be the most recently raised exception
for compatibility with current behaviour.

This PEP proposes that tracebacks display the outermost exception last,
because this would be consistent with the chronological order of
tracebacks (from oldest to most recent frame) and because the actual
thrown exception is easier to find on the last line.

To keep things simpler, the C API calls for setting an exception will
not automatically set the exception's __context__. Guido van Rossum has
expressed concerns with making such changes[9].

As for other languages, Java and Ruby both discard the original
exception when another exception occurs in a catch/rescue or
finally/ensure clause. Perl 5 lacks built-in structured exception
handling. For Perl 6, RFC number 88[10] proposes an exception mechanism
that implicitly retains chained exceptions in an array named @@. In that
RFC, the most recently raised exception is exposed for matching, as in
this PEP; also, arbitrary expressions (possibly involving @@) can be
evaluated for exception matching.

Exceptions in C# contain a read-only InnerException property that may
point to another exception. Its documentation[11] says that "When an
exception X is thrown as a direct result of a previous exception Y, the
InnerException property of X should contain a reference to Y." This
property is not set by the VM automatically; rather, all exception
constructors take an optional innerException argument to set it
explicitly. The __cause__ attribute fulfills the same purpose as
InnerException, but this PEP proposes a new form of raise rather than
extending the constructors of all exceptions. C# also provides a
GetBaseException method that jumps directly to the end of the
InnerException chain; this PEP proposes no analog.

The reason all three of these attributes are presented together in one
proposal is that the __traceback__ attribute provides convenient access
to the traceback on chained exceptions.

Implicit Exception Chaining

Here is an example to illustrate the __context__ attribute:

    def compute(a, b):
        try:
            a/b
        except Exception, exc:
            log(exc)

    def log(exc):
        file = open('logfile.txt')  # oops, forgot the 'w'
        print >>file, exc
        file.close()

Calling compute(0, 0) causes a ZeroDivisionError. The compute() function
catches this exception and calls log(exc), but the log() function also
raises an exception when it tries to write to a file that wasn't opened
for writing.

In today's Python, the caller of compute() gets thrown an IOError. The
ZeroDivisionError is lost. With the proposed change, the instance of
IOError has an additional __context__ attribute that retains the
ZeroDivisionError.

The following more elaborate example demonstrates the handling of a
mixture of finally and except clauses:

    def main(filename):
        file = open(filename)       # oops, forgot the 'w'
        try:
            try:
                compute()
            except Exception, exc:
                log(file, exc)
        finally:
            file.clos()             # oops, misspelled 'close'

    def compute():
        1/0

    def log(file, exc):
        try:
            print >>file, exc       # oops, file is not writable
        except:
            display(exc)

    def display(exc):
        print ex                    # oops, misspelled 'exc'

Calling main() with the name of an existing file will trigger four
exceptions. The ultimate result will be an AttributeError due to the
misspelling of clos, whose __context__ points to a NameError due to the
misspelling of ex, whose __context__ points to an IOError due to the
file being read-only, whose __context__ points to a ZeroDivisionError,
whose __context__ attribute is None.

The proposed semantics are as follows:

1.  Each thread has an exception context initially set to None.
2.  Whenever an exception is raised, if the exception instance does not
    already have a __context__ attribute, the interpreter sets it equal
    to the thread's exception context.
3.  Immediately after an exception is raised, the thread's exception
    context is set to the exception.
4.  Whenever the interpreter exits an except block by reaching the end
    or executing a return, yield, continue, or break statement, the
    thread's exception context is set to None.

Explicit Exception Chaining

The __cause__ attribute on exception objects is always initialized to
None. It is set by a new form of the raise statement:

    raise EXCEPTION from CAUSE

which is equivalent to:

    exc = EXCEPTION
    exc.__cause__ = CAUSE
    raise exc

In the following example, a database provides implementations for a few
different kinds of storage, with file storage as one kind. The database
designer wants errors to propagate as DatabaseError objects so that the
client doesn't have to be aware of the storage-specific details, but
doesn't want to lose the underlying error information.

    class DatabaseError(Exception):
        pass

    class FileDatabase(Database):
        def __init__(self, filename):
            try:
                self.file = open(filename)
            except IOError, exc:
                raise DatabaseError('failed to open') from exc

If the call to open() raises an exception, the problem will be reported
as a DatabaseError, with a __cause__ attribute that reveals the IOError
as the original cause.

Traceback Attribute

The following example illustrates the __traceback__ attribute.

    def do_logged(file, work):
        try:
            work()
        except Exception, exc:
            write_exception(file, exc)
            raise exc

    from traceback import format_tb

    def write_exception(file, exc):
        ...
        type = exc.__class__
        message = str(exc)
        lines = format_tb(exc.__traceback__)
        file.write(... type ... message ... lines ...)
        ...

In today's Python, the do_logged() function would have to extract the
traceback from sys.exc_traceback or sys.exc_info()[12] and pass both the
value and the traceback to write_exception(). With the proposed change,
write_exception() simply gets one argument and obtains the exception
using the __traceback__ attribute.

The proposed semantics are as follows:

1.  Whenever an exception is caught, if the exception instance does not
    already have a __traceback__ attribute, the interpreter sets it to
    the newly caught traceback.

Enhanced Reporting

The default exception handler will be modified to report chained
exceptions. The chain of exceptions is traversed by following the
__cause__ and __context__ attributes, with __cause__ taking priority. In
keeping with the chronological order of tracebacks, the most recently
raised exception is displayed last; that is, the display begins with the
description of the innermost exception and backs up the chain to the
outermost exception. The tracebacks are formatted as usual, with one of
the lines:

    The above exception was the direct cause of the following exception:

or

    During handling of the above exception, another exception occurred:

between tracebacks, depending whether they are linked by __cause__ or
__context__ respectively. Here is a sketch of the procedure:

    def print_chain(exc):
        if exc.__cause__:
            print_chain(exc.__cause__)
            print '\nThe above exception was the direct cause...'
        elif exc.__context__:
            print_chain(exc.__context__)
            print '\nDuring handling of the above exception, ...'
        print_exc(exc)

In the traceback module, the format_exception, print_exception,
print_exc, and print_last functions will be updated to accept an
optional chain argument, True by default. When this argument is True,
these functions will format or display the entire chain of exceptions as
just described. When it is False, these functions will format or display
only the outermost exception.

The cgitb module should also be updated to display the entire chain of
exceptions.

C API

The PyErr_Set* calls for setting exceptions will not set the __context__
attribute on exceptions. PyErr_NormalizeException will always set the
traceback attribute to its tb argument and the __context__ and __cause__
attributes to None.

A new API function, PyErr_SetContext(context), will help C programmers
provide chained exception information. This function will first
normalize the current exception so it is an instance, then set its
__context__ attribute. A similar API function, PyErr_SetCause(cause),
will set the __cause__ attribute.

Compatibility

Chained exceptions expose the type of the most recent exception, so they
will still match the same except clauses as they do now.

The proposed changes should not break any code unless it sets or uses
attributes named __context__, __cause__, or __traceback__ on exception
instances. As of 2005-05-12, the Python standard library contains no
mention of such attributes.

Open Issue: Extra Information

Walter Dörwald[13] expressed a desire to attach extra information to an
exception during its upward propagation without changing its type. This
could be a useful feature, but it is not addressed by this PEP. It could
conceivably be addressed by a separate PEP establishing conventions for
other informational attributes on exceptions.

Open Issue: Suppressing Context

As written, this PEP makes it impossible to suppress __context__, since
setting exc.__context__ to None in an except or finally clause will only
result in it being set again when exc is raised.

Open Issue: Limiting Exception Types

To improve encapsulation, library implementors may want to wrap all
implementation-level exceptions with an application-level exception. One
could try to wrap exceptions by writing this:

    try:
        ... implementation may raise an exception ...
    except:
        import sys
        raise ApplicationError from sys.exc_value

or this:

    try:
        ... implementation may raise an exception ...
    except Exception, exc:
        raise ApplicationError from exc

but both are somewhat flawed. It would be nice to be able to name the
current exception in a catch-all except clause, but that isn't addressed
here. Such a feature would allow something like this:

    try:
        ... implementation may raise an exception ...
    except *, exc:
        raise ApplicationError from exc

Open Issue: yield

The exception context is lost when a yield statement is executed;
resuming the frame after the yield does not restore the context.
Addressing this problem is out of the scope of this PEP; it is not a new
problem, as demonstrated by the following example:

    >>> def gen():
    ...     try:
    ...         1/0
    ...     except:
    ...         yield 3
    ...         raise
    ...
    >>> g = gen()
    >>> g.next()
    3
    >>> g.next()
    TypeError: exceptions must be classes, instances, or strings
    (deprecated), not NoneType

Open Issue: Garbage Collection

The strongest objection to this proposal has been that it creates cycles
between exceptions and stack frames[14]. Collection of cyclic garbage
(and therefore resource release) can be greatly delayed.

    >>> try:
    >>>     1/0
    >>> except Exception, err:
    >>>     pass

will introduce a cycle from err -> traceback -> stack frame -> err,
keeping all locals in the same scope alive until the next GC happens.

Today, these locals would go out of scope. There is lots of code which
assumes that "local" resources -- particularly open files -- will be
closed quickly. If closure has to wait for the next GC, a program (which
runs fine today) may run out of file handles.

Making the __traceback__ attribute a weak reference would avoid the
problems with cyclic garbage. Unfortunately, it would make saving the
Exception for later (as unittest does) more awkward, and it would not
allow as much cleanup of the sys module.

A possible alternate solution, suggested by Adam Olsen, would be to
instead turn the reference from the stack frame to the err variable into
a weak reference when the variable goes out of scope[15].

Possible Future Compatible Changes

These changes are consistent with the appearance of exceptions as a
single object rather than a triple at the interpreter level.

-   If PEP 340 or PEP 343 is accepted, replace the three (type, value,
    traceback) arguments to __exit__ with a single exception argument.
-   Deprecate sys.exc_type, sys.exc_value, sys.exc_traceback, and
    sys.exc_info() in favour of a single member, sys.exception.
-   Deprecate sys.last_type, sys.last_value, and sys.last_traceback in
    favour of a single member, sys.last_exception.
-   Deprecate the three-argument form of the raise statement in favour
    of the one-argument form.
-   Upgrade cgitb.html() to accept a single value as its first argument
    as an alternative to a (type, value, traceback) tuple.

Possible Future Incompatible Changes

These changes might be worth considering for Python 3000.

-   Remove sys.exc_type, sys.exc_value, sys.exc_traceback, and
    sys.exc_info().
-   Remove sys.last_type, sys.last_value, and sys.last_traceback.
-   Replace the three-argument sys.excepthook with a one-argument API,
    and changing the cgitb module to match.
-   Remove the three-argument form of the raise statement.
-   Upgrade traceback.print_exception to accept an exception argument
    instead of the type, value, and traceback arguments.

Implementation

The __traceback__ and __cause__ attributes and the new raise syntax were
implemented in revision 57783[16].

Acknowledgements

Brett Cannon, Greg Ewing, Guido van Rossum, Jeremy Hylton, Phillip J.
Eby, Raymond Hettinger, Walter Dörwald, and others.

References

Copyright

This document has been placed in the public domain.

[1] Raymond Hettinger, "Idea for avoiding exception masking"
https://mail.python.org/pipermail/python-dev/2003-January/032492.html

[2] Brett Cannon explains chained exceptions
https://mail.python.org/pipermail/python-dev/2003-June/036063.html

[3] Greg Ewing points out masking caused by exceptions during finally
https://mail.python.org/pipermail/python-dev/2003-June/036290.html

[4] Greg Ewing suggests storing the traceback in the exception object
https://mail.python.org/pipermail/python-dev/2003-June/036092.html

[5] Guido van Rossum mentions exceptions having a traceback attribute
https://mail.python.org/pipermail/python-dev/2005-April/053060.html

[6] Ka-Ping Yee, "Tidier Exceptions"
https://mail.python.org/pipermail/python-dev/2005-May/053671.html

[7] Ka-Ping Yee, "Chained Exceptions"
https://mail.python.org/pipermail/python-dev/2005-May/053672.html

[8] Brett Cannon explains chained exceptions
https://mail.python.org/pipermail/python-dev/2003-June/036063.html

[9] Guido van Rossum discusses automatic chaining in PyErr_Set*
https://mail.python.org/pipermail/python-dev/2003-June/036180.html

[10] Tony Olensky, "Omnibus Structured Exception/Error Handling
Mechanism" http://dev.perl.org/perl6/rfc/88.html

[11] MSDN .NET Framework Library, "Exception.InnerException Property"
http://msdn.microsoft.com/library/en-us/cpref/html/frlrfsystemexceptionclassinnerexceptiontopic.asp

[12] Brett Cannon explains chained exceptions
https://mail.python.org/pipermail/python-dev/2003-June/036063.html

[13] Walter Dörwald suggests wrapping exceptions to add details
https://mail.python.org/pipermail/python-dev/2003-June/036148.html

[14] Guido van Rossum restates the objection to cyclic trash
https://mail.python.org/pipermail/python-3000/2007-January/005322.html

[15] Adam Olsen suggests using a weakref from stack frame to exception
https://mail.python.org/pipermail/python-3000/2007-January/005363.html

[16] Patch to implement the bulk of the PEP
http://svn.python.org/view/python/branches/py3k/Include/?rev=57783&view=rev