Invalid class definitions

The following class definition raises an error at runtime, because int and str are distinct solid bases:

class C(int, str): pass

Without knowledge of solid bases, type checkers are not currently able to detect the reason why this class definition is invalid, though they may detect that if this class were to exist, some of its methods would be incompatible. (When it sees this class definition, mypy will point at incompatible definitions of __add__ and several other methods.)

This is not a particularly compelling problem by itself, as the error would usually be caught the first time the code is imported, but it is mentioned here for completeness.

Reachability

We already mentioned the reachability of code using isinstance(). Similar issues arise with other type narrowing constructs such as match statements: correct inference of reachability requires an understanding of solid bases.

class A: pass
class B: pass

def f(x: A):
    match x:
        case B():  # reachable
            print("It's both!")

def g(x: int):
    match x:
        case str():  # unreachable
            print("It's both!")

Overloads

Functions decorated with @overload may be unsafe if the parameter types of some overloads overlap, but the return types do not. For example, the following set of overloads could be exploited to achieve unsound behavior:

from typing import overload

class A: pass
class B: pass

@overload
def f(x: A) -> str: ...
@overload
def f(x: B) -> int: ...

If a class exists that inherits from both A and B, then type checkers could pick the wrong overload on a call to f().

Type checkers could detect this source of unsafety and warn about it, but a correct implementation requires an understanding of solid bases, because it relies on knowing whether values that are instances of both A and B can exist. Although many type checkers already perform a version of this check for overlapping overloads, the typing specification does not currently prescribe how this check should work. This PEP does not propose to change that, but it helps provide a building block for a sound check for overlapping overloads.

Intersection types

Explicit intersection types, denoting a type that contains values that are instances of all of the given types, are not currently part of the type system. They do, however, arise naturally in a set-theoretic type system like Python’s as a result of type narrowing, and future extensions to the type system may add support for explicit intersection types.

With intersection types, it is often important to know whether a particular intersection is inhabited, that is, whether there are values that can be members of that intersection. This allows type checkers to understand reachability and provide more precise type information to users.

As a concrete example, a possible implementation of assignability with intersection types could be that given an intersection type A & B, a type C is assignable to it if C is assignable to at least one of A and B, and overlaps with all of A and B. (“Overlaps” here means that at least one runtime value could exist that would be a member of both types. That is, A and B overlap if A & B is inhabited.) The second part of the rule ensures that str is not assignable to a type like int & Any: while str is assignable to Any, it does not overlap with int. But of course, we can only know that str and int do not overlap if we know that both classes are solid bases.

Overview

Solid bases can be helpful in many corners of the type system. Though some of these corners are underspecified, speculative, or of marginal importance, in each case the concept of solid bases enables type checkers to gain a more precise understanding than the current type system allows. Thus, solid bases provide a firm foundation (a solid base, if you will) for improving the Python type system.

Runtime restrictions on multiple inheritance

While Python generally allows multiple inheritance, the runtime imposes various restrictions, as documented in CPython PR 136844 (hopefully soon to be merged). Two sets of restrictions, around a consistent MRO and a consistent metaclass, can already be implemented by type checkers using information available in the type system. The third restriction, around instance layout, is the one that requires knowledge of solid bases. Classes that contain a non-empty __slots__ definition are automatically solid bases, as are many built-in classes implemented in C.

Alternative implementations of Python, such as PyPy, tend to behave similarly to CPython but may differ in details, such as exactly which standard library classes are solid bases. As the type system does not currently contain any explicit support for alternative Python implementations, this PEP recommends that stub libraries such as typeshed use CPython’s behavior to determine when to use the @solid_base decorator. If future extensions to the type system add support for alternative implementations (for example, branching on the value of sys.implementation.name), stubs could condition the presence of the @solid_base decorator on the implementation where necessary.

@solid_base in implementation files

The most obvious use case for the @solid_base decorator will be in stub files for C libraries, such as the standard library, for marking solid bases implemented in C.

However, there are also use cases for marking solid bases in implementation files, where the effect would be to disallow the existence of child classes that inherit from the decorated class and another solid base, such as a standard library class or another user class decorated with @solid_base. For example, this could allow type checkers to flag code that can only be reachable if a class exists that inherits from both a user class and a standard library class such as int or str, which may be technically possible but not practically plausible.

@solid_base
class BaseModel:
    # ... General logic for model classes
    pass

class Species(BaseModel):
    name: str
    # ... more fields

def process_species(species: Species):
    if isinstance(species, str):  # oops, forgot `.name`
        pass  # type checker should warn about this branch being unreachable
        # BaseModel and str are solid bases, so a class that inherits from both cannot exist

This is similar in principle to the existing @final decorator, which also acts to restrict subclassing: in stubs, it is used to mark classes that programmatically disallow subclassing, but in implementation files, it is often used to indicate that a class is not intended to be subclassed, without runtime enforcement.

@solid_base on special classes

The @solid_base decorator is primarily intended for nominal classes, but the type system contains some other constructs that syntactically use class definitions, so we have to consider whether the decorator should be allowed on them as well, and if so, what it would mean.

For Protocol definitions, the most consistent interpretation would be that the only classes that can implement the protocol would be classes that use nominal inheritance from the protocol, or @final classes that implement the protocol. Other classes either have or could potentially have a solid base that is not the protocol. This is convoluted and not useful, so we disallow @solid_base on Protocol definitions.

Similarly, the concept of a “solid base” is not meaningful on TypedDict definitions, as TypedDicts are purely structural types.

Although they receive some special treatment in the type system, NamedTuple definitions create real nominal classes that can have child classes, so it makes sense to allow @solid_base on them and treat them like regular classes for the purposes of the solid base mechanism. All NamedTuple classes have tuple, a solid base, in their MRO, so they cannot double inherit from other solid bases.

Solid bases in CPython

The concept of “solid bases” has been part of the CPython implementation for a long time; the concept dates back to a 2001 commit. Nevertheless, the concept has received little attention in the documentation. Although details of the mechanism are closely tied to CPython’s internal object representation, it is useful to explain at a high level how and why CPython works this way.

Every object in CPython is essentially a pointer to a C struct, a contiguous piece of memory that contains information about the object. Some information is managed by the interpreter and shared by many or all objects, such as a reference to the type of the object, and the attribute __dict__ for user-defined objects. Some classes contain additional information that is specific to that class. For example, user-defined classes with __slots__ contain a place in memory for each slot, and the built-in float class contains a C double value that stores the value of the float. This memory layout must be preserved for all instances of the class: C code that interacts with a float expects to find the value at a particular offset in the object’s memory.

When a child class is created, CPython must create a memory layout for the new class that is compatible with all of its parent classes. For example, when a child class of float is created, it must be possible to pass instances of the child class to C code that interacts directly with the underlying struct for the float class. Therefore, such a subclass must store the double value at the same offset as the parent float class does. It may, however, add additional fields at the end of the struct. CPython knows how to do this with the __dict__ attribute, which is why it is possible to create a child class of float that adds a __dict__.

However, there is no way to combine a float, which must have a double in its struct, with another C type like int, which stores different data at the same spot. Therefore, a common subclass of float and int cannot exist. We say that float and int are solid bases.

A class implemented in C is a solid base if it has an underlying struct that stores data at a fixed offset, and that struct is different from the struct of its parent class. A C class may also store a variable-size array of data (such as the contents of a string); if this differs from the parent class, the class also becomes a solid base. CPython’s implementation deduces this from the tp_itemsize and tp_basicsize fields of the type object, which are also accessible from Python code as the undocumented attributes __itemsize__ and __basicsize__ on type objects.

Similarly, classes implemented in Python are solid bases if they have __slots__, because slots force a particular memory layout.

Mypy’s incompatibility check

The mypy type checker considers two classes to be incompatible if they have incompatible methods. For example, mypy considers the int and str classes to be incompatible because they have incompatible definitions of various methods. Given a class definition like:

class C(int, str):
    pass

Mypy will output Definition of "__add__" in base class "int" is incompatible with definition in base class "str", and similar errors for a number of other methods. These errors are correct, because the definitions of __add__ in the two classes are indeed incompatible: int.__add__ expects an int argument, while str.__add__ expects a str. If this class were to exist, at runtime __add__ would resolve to int.__add__. Instances of C would also be members of the str type, but they would not support some of the operations that str supports, such as concatenation with another str.

So far, so good. But mypy also uses very similar logic to conclude that no class can inherit from both int and str. Nevertheless, it accepts the following class definition without error:

from typing import Never

class C(int, str):
    def __add__(self, other: object) -> Never:
        raise TypeError
    def __mod__(self, other: object) -> Never:
        raise TypeError
    def __mul__(self, other: object) -> Never:
        raise TypeError
    def __rmul__(self, other: object) -> Never:
        raise TypeError
    def __ge__(self, other: int | str) -> bool:
        return int(self) > other if isinstance(other, int) else str(self) > other
    def __gt__(self, other: int | str) -> bool:
        return int(self) >= other if isinstance(other, int) else str(self) >= other
    def __lt__(self, other: int | str) -> bool:
        return int(self) < other if isinstance(other, int) else str(self) < other
    def __le__(self, other: int | str) -> bool:
        return int(self) <= other if isinstance(other, int) else str(self) <= other
    def __getnewargs__(self) -> Never:
        raise TypeError

There is a similar situation with attributes. Given two classes with incompatible attributes, mypy claims that a common subclass cannot exist, yet it accepts a subclass that overrides these attributes to make them compatible:

from typing import Never

class X:
    a: int

class Y:
    a: str

class Z(X, Y):
    @property
    def a(self) -> Never:
        raise RuntimeError("no luck")
    @a.setter
    def a(self, value: int | str) -> None:
        pass

While the examples given so far rely on overrides that return Never, mypy’s rule can also reject classes that have more practically useful implementations:

from typing import Literal

class Carnivore:
    def eat(self, food: Literal["meat"]) -> None:
        print("devouring meat")

class Herbivore:
    def eat(self, food: Literal["plants"]) -> None:
        print("nibbling on plants")

class Omnivore(Carnivore, Herbivore):
    def eat(self, food: str) -> None:
        print(f"eating {food}")

def is_it_both(obj: Carnivore):
    # mypy --warn-unreachable:
    # Subclass of "Carnivore" and "Herbivore" cannot exist: would have incompatible method signatures
    if isinstance(obj, Herbivore):
        pass

Mypy’s rule works reasonably well in practice for deducing whether an intersection of two classes is inhabited. Most builtin classes that are solid bases happen to implement common dunder methods such as __add__ and __iter__ in incompatible ways, so mypy will consider them incompatible. There are some exceptions: mypy allows class C(BaseException, int): ..., though both of these classes are solid bases and the class definition is rejected at runtime. Conversely, when multiple inheritance is used in practice, usually the parent classes will not have incompatible methods.

Thus, mypy’s approach to deciding that two classes cannot intersect is both too broad (it incorrectly considers some intersections to be uninhabited) and too narrow (it misses some intersections that are uninhabited because of solid bases). This is discussed in an issue on the mypy tracker.