Following system colour scheme Selected dark colour scheme Selected light colour scheme

Python Enhancement Proposals

PEP 698 – Override Decorator for Static Typing

Author:
Steven Troxler <steven.troxler at gmail.com>, Joshua Xu <jxu425 at fb.com>, Shannon Zhu <szhu at fb.com>
Status:
Draft
Sponsor:
Jelle Zijlstra <jelle.zijlstra at gmail.com>
Type:
Standards Track
Created:
05-Sep-2022
Python-Version:
3.12

Table of Contents

Abstract

This PEP proposes adding an @override decorator to the Python type system. This will allow type checkers to prevent a class of bugs that occur when a base class changes methods that are inherited by derived classes.

Motivation

A primary purpose of type checkers is to flag when refactors or changes break pre-existing semantic structures in the code, so users can identify and make fixes across their project without doing a manual audit of their code.

Safe Refactoring

Python’s type system does not provide a way to identify call sites that need to be changed to stay consistent when an overridden function API changes. This makes refactoring and transforming code more dangerous.

Consider this simple inheritance structure:

class Parent:
    def foo(self, x: int) -> int:
        return x

class Child(Parent):
    def foo(self, x: int) -> int:
        return x + 1

def parent_callsite(parent: Parent) -> None:
    parent.foo(1)

def child_callsite(child: Child) -> None:
    child.foo(1)

If the overridden method on the superclass is renamed or deleted, type checkers will only alert us to update call sites that deal with the base type directly. But the type checker can only see the new code, not the change we made, so it has no way of knowing that we probably also needed to rename the same method on child classes.

A type checker will happily accept this code, even though we are likely introducing bugs:

class Parent:
    # Rename this method
    def new_foo(self, x: int) -> int:
        return x

class Child(Parent):
    # This (unchanged) method used to override `foo` but is unrelated to `new_foo`
    def foo(self, x: int) -> int:
        return x + 1

def parent_callsite(parent: Parent) -> None:
    # If we pass a Child instance we’ll now run Parent.new_foo - likely a bug
    parent.new_foo(1)

def child_callsite(child: Child) -> None:
    # We probably wanted to invoke new_foo here. Instead, we forked the method
    child.foo(1)

This code will type check, but there are two potential sources of bugs:

  • If we pass a Child instance to the parent_callsite function, it will invoke the implementation in Parent.new_foo. rather than Child.foo. This is probably a bug - we presumably would not have written Child.foo in the first place if we didn’t need custom behavior.
  • Our system was likely relying on Child.foo behaving in a similar way to Parent.foo. But unless we catch this early, we have now forked the methods, and future refactors it is likely no one will realize that major changes to the behavior of new_foo likely require updating Child.foo as well, which could lead to major bugs later.

The incorrectly-refactored code is type-safe, but is probably not what we intended and could cause our system to behave incorrectly. The bug can be difficult to track down because our new code likely does execute without throwing exceptions. Tests are less likely to catch the problem, and silent errors can take longer to track down in production.

We are aware of several production outages in multiple typed codebases caused by such incorrect refactors. This is our primary motivation for adding an @override decorator to the type system, which lets developers express the relationship between Parent.foo and Child.foo so that type checkers can detect the problem.

Rationale

Subclass Implementations Become More Explicit

We believe that explicit overrides will make unfamiliar code easier to read than implicit overrides. A developer reading the implementation of a subclass that uses @override can immediately see which methods are overriding functionality in some base class; without this decorator, the only way to quickly find out is using a static analysis tool.

Precedent in Other Languages and Runtime Libraries

Static Override Checks in Other Languages

Many popular programming languages support override checks. For example:

Runtime Override Checks in Python

Today, there is an Overrides library that provides decorators @overrides (sic) and @final and will enforce them at runtime.

PEP 591 added a @final decorator with the same semantics as those in the Overrides library. But the override component of the runtime library is not supported statically at all, which has added some confusion around the mix/matched support.

Providing support for @override in static checks would add value because

  • Bugs can be caught earlier, often in-editor.
  • Static checks come with no performance overhead, unlike runtime checks.
  • Bugs will be caught quickly even in rarely-used modules, whereas with runtime checks these might go undetected for a time without automated tests of all imports.

Disadvantages

The two disadvantages we are aware of to using @override are that

  • The code becomes more verbose - overriding methods require one additional line.
  • Adding or removing base class methods that impact overrides will require updating subclass code.

Specification

When type checkers encounter a method decorated with @typing.override they should treat it as a type error unless that method is overriding a compatible method or attribute in some ancestor class.

from typing import override

class Parent:
    def foo(self) -> int:
        return 1

    def bar(self, x: str) -> str:
        return x

class Child(Parent):
    @override
    def foo(self) -> int:
        return 2

    @override
    def baz() -> int:  # Type check error: no matching signature in ancestor
        return 1

The @override decorator should be permitted anywhere a type checker considers a method to be a valid override, which typically includes not only normal methods but also @property, @staticmethod, and @classmethod.

Override Compatibility Rules are Unchanged

Type checkers already enforce compatibility rules for overrides; for example, a subclass method’s type signature should be compatible with that of the superclass method. These compatibility rules do not change due to the presence or absence of @override.

Note that when a @property overrides a regular attribute of the base class, this should not be considered an error due to the use of @override, but the type checker may still consider the override to be incompatible. For example a type checker may consider it illegal to override a non-final attribute with a getter property and no setter, as this does not respect the substitution principle.

Strict Enforcement Per-Project

We plan to make the use of @override required in Pyre’s strict mode. This is a feature we believe most type checkers would benefit from.

Motivation

The primary reason for a strict mode that requires @override is that developers can only trust that refactors are override-safe if they know that the @override decorator is used throughout the project.

There is another class of bug related to overrides that we can only catch using a strict mode.

Consider the following code:

class Parent:
    pass

class Child(Parent):
    def foo() -> int:
        return 2

Imagine we refactor it as follows:

class Parent
    def foo() -> int:   # This method is new
        return 1

class Child(Parent):
    def foo() -> int:  # This is now an override!
        return 2

def call_foo(parent: Parent) -> int:
    return base.foo()  # This could invoke Child.foo, which may be surprising.

The semantics of our code changed here, which could cause two problems:

  • If the author of the code change did not know that Child.foo already existed (which is very possible in a large codebase), they might be surprised to see that call_foo does not always invoke Parent.foo.
  • If the codebase authors tried to manually apply @override everywhere when writing overrides in subclasses, they are likely to miss the fact that Child.foo needs it here.

At first glance this kind of change may seem unlikely, but it can actually happen often if one or more subclasses have functionality that developers later realize belongs in the base class.

With a strict mode, we will always alert developers when this occurs.

Precedent

Most of the typed, object-oriented programming languages we looked at have an easy way to require explicit overrides throughout a project:

  • C#, Kotlin, Scala, and Swift always require explicit overrides
  • Typescript has a –no-implicit-override flag to force explicit overrides
  • In Hack and Java the type checker always treats overrides as opt-in, but widely-used linters can warn if explicit overrides are missing.

Backward Compatibility

By default, the @override decorator will be opt-in. Codebases that do not use it will type-check as before, without the additional type safety.

Runtime Behavior

At runtime, @typing.override will do nothing but return its argument.

We considered other options but rejected them because the downsides seemed to outweigh the benefits, see the Rejected Alternatives section.

Rejected Alternatives

Rely on Integrated Development Environments for safety

Modern Integrated Development Environments (IDEs) often provide the ability to automatically update subclasses when renaming a method. But we view this as insufficient for several reasons:

  • If a codebase is split into multiple projects, an IDE will not help and the bug appears when upgrading dependencies. Type checkers are a fast way to catch breaking changes in dependencies.
  • Not all developers use such IDEs. And library maintainers, even if they do use an IDE, should not need to assume pull request authors use the same IDE. We prefer being able to detect problems in continuous integration without assuming anything about developers’ choice of editor.

Runtime enforcement

We considered having @typing.overrride enforce override safety at runtime, similarly to how @overrides.overrrides does today.

We rejected this for three reasons:

  • For users of static type checking, it is not clear this brings any benefits.
  • There would be at least some performance overhead, leading to projects importing slower with runtime enforcement. We estimate the @overrides.overrrides implementation takes around 100 microseconds, which is fast but could still add up to a second or more of extra initialization time in million-plus line codebases, which is exactly where we think @typing.override will be most useful.
  • An implementation may have edge cases where it doesn’t work well (we heard from a maintainer of one such closed-source library that this has been a problem). We expect static enforcement to be simple and reliable.
  • The implementation approaches we know of are not simple. The decorator executes before the class is finished evaluating, so the options we know of are either to inspect the bytecode of the caller (as @overrides.overrrides does) or to use a metaclass-based approach. Neither approach seems ideal.

Marking overrides at runtime with an __override__ attribute

The @overrides.overrrides decorator marks methods it decorates with an __override__ attribute.

We considered having @typing.override do the same, since many typing features are made available at runtime for runtime libraries to use them. We decided against this because again the downsides seem to outweigh the benefits:

Setting an attribute significantly complicates correct use of the decorator

If we have any runtime behavior at all in our decorator, we have to worry about the order of decorators.

A decorator usually wraps a function in another function, and @override would behave correctly if it were placed above all such decorators.

But some decorators instead define descriptors - for example @classmethod, @staticmethod, and @property all use descriptors. In these cases, placing @override below these decorators would work, but it would be possible for libraries to define decorators in ways where even that would not work.

Moreover, we believe that it would be bad for most users - many of whom may not even understand descriptors - to be faced with a feature where correct use of @override depends on placing it in between decorators that are implemented as wrapped functions and those that are implemented as

We prefer to have no runtime behavior, which allows us to not care about the ordering and recommend, for style reasons, that @override always comes first.

Lack of any clear benefit

We are not aware of any use for explicit marking of overrides other than the extra type safety it provides. This is in contrast to other typing features such as type annotations, which have important runtime uses such as metaprogramming and runtime type checking.

In light of the downsides described above, we decided the benefits are insufficient to justify runtime behavior.

Mark a base class to force explicit overrides on subclasses

We considered including a class decorator @require_explicit_overrides, which would have provided a way for base classes to declare that all subclasses must use the @override decorator on method overrides. The overrides library has a mixin class, EnforceExplicitOverrides, which provides similar behavior in runtime checks.

We decided against this because we expect owners of large codebases will benefit most from @override, and for these use cases having a strict mode where explicit @override is required (see the Backward Compatibility section) provides more benefits than a way to mark base classes.

Moreover we believe that authors of projects who do not consider the extra type safety to be worth the additional boilerplate of using @override should not be forced to do so. Having an optional strict mode puts the decision in the hands of project owners, whereas the use of @require_explicit_overrides in libraries would force project owners to use @override even if they prefer not to.

Reference Implementation

Pyre: A proof of concept is implemented in Pyre:


Source: https://github.com/python/peps/blob/main/pep-0698.rst

Last modified: 2022-09-23 17:42:25 GMT