Following system colour scheme Selected dark colour scheme Selected light colour scheme

Python Enhancement Proposals

PEP 728 – TypedDict with Typed Extra Items

Author:
Zixuan James Li <p359101898 at gmail.com>
Sponsor:
Jelle Zijlstra <jelle.zijlstra at gmail.com>
Discussions-To:
Discourse thread
Status:
Draft
Type:
Standards Track
Topic:
Typing
Created:
12-Sep-2023
Python-Version:
3.14
Post-History:
09-Feb-2024

Table of Contents

Abstract

This PEP adds two class parameters, closed and extra_items to type the extra items on a TypedDict. This addresses the need to define closed TypedDict types or to type a subset of keys that might appear in a dict while permitting additional items of a specified type.

Motivation

A typing.TypedDict type can annotate the value type of each known item in a dictionary. However, due to structural assignability, a TypedDict can have extra items that are not visible through its type. There is currently no way to restrict the types of items that might be present in the TypedDict type’s consistent subtypes.

Disallowing Extra Items Explicitly

The current behavior of TypedDict prevents users from defining a TypedDict type when it is expected that the type contains no extra items.

Due to the possible presence of extra items, type checkers cannot infer more precise return types for .items() and .values() on a TypedDict. This can also be resolved by defining a closed TypedDict type.

Another possible use case for this is a sound way to enable type narrowing with the in check:

class Movie(TypedDict):
    name: str
    director: str

class Book(TypedDict):
    name: str
    author: str

def fun(entry: Movie | Book) -> None:
    if "author" in entry:
        reveal_type(entry)  # Revealed type is still 'Movie | Book'

Nothing prevents a dict that is assignable with Movie to have the author key, and under the current specification it would be incorrect for the type checker to narrow its type.

Allowing Extra Items of a Certain Type

For supporting API interfaces or legacy codebase where only a subset of possible keys are known, it would be useful to explicitly specify extra items of certain value types.

However, the typing spec is more restrictive when checking the construction of a TypedDict, preventing users from doing this:

class MovieBase(TypedDict):
    name: str

def foo(movie: MovieBase) -> None:
    # movie can have extra items that are not visible through MovieBase
    ...

movie: MovieBase = {"name": "Blade Runner", "year": 1982}  # Not OK
foo({"name": "Blade Runner", "year": 1982})  # Not OK

While the restriction is enforced when constructing a TypedDict, due to structural assignability, the TypedDict may have extra items that are not visible through its type. For example:

class Movie(MovieBase):
    year: int

movie: Movie = {"name": "Blade Runner", "year": 1982}
foo(movie)  # OK

It is not possible to acknowledge the existence of the extra items through in checks and access them without breaking type safety, even though they might exist from some consistent subtypes of MovieBase:

def bar(movie: MovieBase) -> None:
    if "year" in movie:
        reveal_type(movie["year"])  # Error: TypedDict 'MovieBase' has no key 'year'

Some workarounds have already been implemented to allow extra items, but none of them is ideal. For mypy, --disable-error-code=typeddict-unknown-key suppresses type checking error specifically for unknown keys on TypedDict. This sacrifices type safety over flexibility, and it does not offer a way to specify that the TypedDict type expects additional keys whose value types are assignable with a certain type.

Support Additional Keys for Unpack

PEP 692 adds a way to precisely annotate the types of individual keyword arguments represented by **kwargs using TypedDict with Unpack. However, because TypedDict cannot be defined to accept arbitrary extra items, it is not possible to allow additional keyword arguments that are not known at the time the TypedDict is defined.

Given the usage of pre-PEP 692 type annotation for **kwargs in existing codebases, it will be valuable to accept and type extra items on TypedDict so that the old typing behavior can be supported in combination with Unpack.

Rationale

A type that allows extra items of type str on a TypedDict can be loosely described as the intersection between the TypedDict and Mapping[str, str].

Index Signatures in TypeScript achieve this:

type Foo = {
    a: string
    [key: string]: string
}

This proposal aims to support a similar feature without introducing general intersection of types or syntax changes, offering a natural extension to the existing assignability rules.

We propose to add a class parameter extra_items to TypedDict. It accepts a type expression as the argument; when it is present, extra items are allowed, and their value types must be assignable to the type expression value.

An application of this is to disallow extra items. We propose to add a closed class parameter, which only accepts a literal True or False as the argument. It should be a runtime error when closed and extra_items are used at the same time.

Different from index signatures, the types of the known items do not need to be assignable to the extra_items argument.

There are some advantages to this approach:

  • We can build on top of the assignability rules defined in the typing spec, where extra_items can be treated as a pseudo-item.
  • There is no need to introduce a grammar change to specify the type of the extra items.
  • We can precisely type the extra items without requiring the value types of the known items to be assignable to extra_items.
  • We do not lose backwards compatibility as both extra_items and closed are opt-in only features.

Specification

This specification is structured to parallel PEP 589 to highlight changes to the original TypedDict specification.

If extra_items is specified, extra items are treated as non-required items matching the extra_items argument, whose keys are allowed when determining supported and unsupported operations.

The extra_items Class Parameter

For a TypedDict type that specifies extra_items, during construction, the value type of each unknown item is expected to be non-required and assignable to the extra_items argument. For example:

class Movie(TypedDict, extra_items=bool):
    name: str

a: Movie = {"name": "Blade Runner", "novel_adaptation": True}  # OK
b: Movie = {
    "name": "Blade Runner",
    "year": 1982,  # Not OK. 'int' is not assignable to 'bool'
}

Here, extra_items=bool specifies that items other than 'name' have a value type of bool and are non-required.

The alternative inline syntax is also supported:

Movie = TypedDict("Movie", {"name": str}, extra_items=bool)

Accessing extra items is allowed. Type checkers must infer their value type from the extra_items argument:

def f(movie: Movie) -> None:
    reveal_type(movie["name"])              # Revealed type is 'str'
    reveal_type(movie["novel_adaptation"])  # Revealed type is 'bool'

extra_items is inherited through subclassing:

class MovieBase(TypedDict, extra_items=int | None):
    name: str

class Movie(MovieBase):
    year: int

a: Movie = {"name": "Blade Runner", "year": None}  # Not OK. 'None' is incompatible with 'int'
b: Movie = {
    "name": "Blade Runner",
    "year": 1982,
    "other_extra_key": None,
}  # OK

Here, 'year' in a is an extra key defined on Movie whose value type is int. 'other_extra_key' in b is another extra key whose value type must be assignable to the value of extra_items defined on MovieBase.

The closed Class Parameter

When closed=True is set, no extra items are allowed. This is a shorthand for extra_items=Never, because there can’t be a value type that is assignable to Never.

Similar to total, only a literal True or False is supported as the value of the closed argument; closed is False by default, which preserves the previous TypedDict behavior.

The value of closed is not inherited through subclassing, but the implicitly set extra_items=Never is. It should be an error to use the default closed=False when subclassing a closed TypedDict type:

class BaseMovie(TypedDict, closed=True):
    name: str

class MovieA(BaseMovie):  # Not OK. An explicit 'closed=True' is required
    pass

class MovieB(BaseMovie, closed=True):  # OK
    pass

Setting both closed and extra_items when defining a TypedDict type should always be a runtime error:

class Person(TypedDict, closed=False, extra_items=bool):  # Not OK. 'closed' and 'extra_items' are incompatible
    name: str

As a consequence of closed=True being equivalent to extra_items=Never. The same rules that apply to extra_items=Never should also apply to closed=True. It is possible to use closed=True when subclassing if the extra_items argument is a read-only type:

class Movie(TypedDict, extra_items=ReadOnly[str]):
    pass

class MovieClosed(Movie, closed=True):  # OK
    pass

class MovieNever(Movie, extra_items=Never):  # Not OK. 'closed=True' is preferred
    pass

This will be further discussed in a later section.

When neither extra_items nor closed=True is specified, the TypedDict is assumed to allow non-required extra items of value type ReadOnly[object] during inheritance or assignability checks. This preserves the existing behavior of TypedDict.

Interaction with Totality

It is an error to use Required[] or NotRequired[] with extra_items. total=False and total=True have no effect on extra_items itself.

The extra items are non-required, regardless of the totality of the TypedDict. Operations that are available to NotRequired items should also be available to the extra items:

class Movie(TypedDict, extra_items=int):
    name: str

def f(movie: Movie) -> None:
    del movie["name"]  # Not OK. The value type of 'name' is 'Required[int]'
    del movie["year"]  # OK. The value type of 'year' is 'NotRequired[int]'

Interaction with Unpack

For type checking purposes, Unpack[SomeTypedDict] with extra items should be treated as its equivalent in regular parameters, and the existing rules for function parameters still apply:

class Movie(TypedDict, extra_items=int):
    name: str

def f(**kwargs: Unpack[Movie]) -> None: ...

# Should be equivalent to:
def f(*, name: str, **kwargs: int) -> None: ...

Interaction with Read-only Items

When the extra_items argument is annotated with the ReadOnly[] type qualifier, the extra items on the TypedDict have the properties of read-only items. This interacts with inheritance rules specified in Read-only Items.

Notably, if the TypedDict type specifies extra_items to be read-only, subclasses of the TypedDict type may redeclare extra_items.

Because a non-closed TypedDict type implicitly allows non-required extra items of value type ReadOnly[object], its subclass can override the extra_items argument with more specific types.

More details are discussed in the later sections.

Inheritance

extra_items is inherited in a similar way as a regular key: value_type item. As with the other keys, the inheritance rules and Read-only Items inheritance rules apply.

We need to reinterpret these rules to define how extra_items interacts with them.

  • Changing a field type of a parent TypedDict class in a subclass is not allowed.

First, it is not allowed to change the value of extra_items in a subclass unless it is declared to be ReadOnly in the superclass:

class Parent(TypedDict, extra_items=int | None):
    pass

class Child(Parent, extra_items=int): # Not OK. Like any other TypedDict item, extra_items's type cannot be changed

Second, extra_items=T effectively defines the value type of any unnamed items accepted to the TypedDict and marks them as non-required. Thus, the above restriction applies to any additional items defined in a subclass. For each item added in a subclass, all of the following conditions should apply:

  • If extra_items is read-only
    • The item can be either required or non-required
    • The item’s value type is assignable to T
  • If extra_items is not read-only
    • The item is non-required
    • The item’s value type is consistent with T
  • If extra_items is not overriden, the subclass inherits it as-is.

For example:

class MovieBase(TypedDict, extra_items=int | None):
    name: str

class AdaptedMovie(MovieBase):  # Not OK. 'bool' is not assignable to 'int | None'
    adapted_from_novel: bool

class MovieRequiredYear(MovieBase):  # Not OK. Required key 'year' is not known to 'Parent'
    year: int | None

class MovieNotRequiredYear(MovieBase):  # Not OK. 'int | None' is not assignable to 'int'
    year: NotRequired[int]

class MovieWithYear(MovieBase):  # OK
    year: NotRequired[int | None]

class BookBase(TypedDict, extra_items=ReadOnly[int | str]):
    title: str

class Book(BookBase, extra_items=str):  # OK
    year: int  # OK

An important side effect of the inheritance rules is that we can define a TypedDict type that disallows additional items:

class MovieClosed(TypedDict, extra_items=Never):
    name: str

Here, passing the value Never to extra_items specifies that there can be no other keys in MovieFinal other than the known ones. Because of its potential common use, there is a preferred alternative:

class MovieClosed(TypedDict, closed=True):
    name: str

where we implicitly assume that extra_items=Never.

Assignability

Let S be the set of keys of the explicitly defined items on a TypedDict type. If it specifies extra_items=T, the TypedDict type is considered to have an infinite set of items that all satisfy the following conditions.

  • If extra_items is read-only:
    • The key’s value type is assignable to T.
    • The key is not in S.
  • If extra_items is not read-only:
    • The key is non-required.
    • The key’s value type is consistent with T.
    • The key is not in S.

For type checking purposes, let extra_items be a non-required pseudo-item when checking for assignability according to rules defined in the Read-only Items section, with a new rule added in bold text as follows:

A TypedDict type B is assignable to a TypedDict type A if B is structurally assignable to A. This is true if and only if all of the following are satisfied:
  • [If no key with the same name can be found in ``B``, the ‘extra_items’ argument is considered the value type of the corresponding key.]
  • For each item in A, B has the corresponding key, unless the item in A is read-only, not required, and of top value type (ReadOnly[NotRequired[object]]).
  • For each item in A, if B has the corresponding key, the corresponding value type in B is assignable to the value type in A.
  • For each non-read-only item in A, its value type is assignable to the corresponding value type in B, and the corresponding key is not read-only in B.
  • For each required key in A, the corresponding key is required in B.
  • For each non-required key in A, if the item is not read-only in A, the corresponding key is not required in B.

The following examples illustrate these checks in action.

extra_items puts various restrictions on additional items for assignability checks:

class Movie(TypedDict, extra_items=int | None):
    name: str

class MovieDetails(TypedDict, extra_items=int | None):
    name: str
    year: NotRequired[int]

details: MovieDetails = {"name": "Kill Bill Vol. 1", "year": 2003}
movie: Movie = details  # Not OK. While 'int' is assignable to 'int | None',
                        # 'int | None' is not assignable to 'int'

class MovieWithYear(TypedDict, extra_items=int | None):
    name: str
    year: int | None

details: MovieWithYear = {"name": "Kill Bill Vol. 1", "year": 2003}
movie: Movie = details  # Not OK. 'year' is not required in 'Movie',
                        # so it shouldn't be required in 'MovieWithYear' either

Because 'year' is absent in Movie, extra_items is considered the corresponding key. 'year' being required violates this rule:

  • For each required key in A, the corresponding key is required in B.

When extra_items is specified to be read-only on a TypedDict type, it is possible for an item to have a narrower type than the extra_items argument:

class Movie(TypedDict, extra_items=ReadOnly[str | int]):
    name: str

class MovieDetails(TypedDict, extra_items=int):
    name: str
    year: NotRequired[int]

details: MovieDetails = {"name": "Kill Bill Vol. 2", "year": 2004}
movie: Movie = details  # OK. 'int' is assignable to 'str | int'.

This behaves the same way as if year: ReadOnly[str | int] is an item explicitly defined in Movie.

extra_items as a pseudo-item follows the same rules that other items have, so when both TypedDicts types specify extra_items, this check is naturally enforced:

class MovieExtraInt(TypedDict, extra_items=int):
    name: str

class MovieExtraStr(TypedDict, extra_items=str):
    name: str

extra_int: MovieExtraInt = {"name": "No Country for Old Men", "year": 2007}
extra_str: MovieExtraStr = {"name": "No Country for Old Men", "description": ""}
extra_int = extra_str  # Not OK. 'str' is not assignable to extra items type 'int'
extra_str = extra_int  # Not OK. 'int' is not assignable to extra items type 'str'

A non-closed TypedDict type implicitly allows non-required extra keys of value type ReadOnly[object]. Applying the assignability rules between this type and a closed TypedDict type is allowed:

class MovieNotClosed(TypedDict):
    name: str

extra_int: MovieExtraInt = {"name": "No Country for Old Men", "year": 2007}
not_closed: MovieNotClosed = {"name": "No Country for Old Men"}
extra_int = not_closed  # Not OK.
                        # 'extra_items=ReadOnly[object]' implicitly on 'MovieNotClosed'
                        # is not assignable to with 'extra_items=int'
not_closed = extra_int  # OK

Interaction with Constructors

TypedDicts that allow extra items of type T also allow arbitrary keyword arguments of this type when constructed by calling the class object:

class NonClosedMovie(TypedDict):
    name: str

NonClosedMovie(name="No Country for Old Men")  # OK
NonClosedMovie(name="No Country for Old Men", year=2007)  # Not OK. Unrecognized item

class ExtraMovie(TypedDict, extra_items=int):
    name: str

ExtraMovie(name="No Country for Old Men")  # OK
ExtraMovie(name="No Country for Old Men", year=2007)  # OK
ExtraMovie(
    name="No Country for Old Men",
    language="English",
)  # Not OK. Wrong type for extra item 'language'

# This implies 'extra_items=Never',
# so extra keyword arguments would produce an error
class ClosedMovie(TypedDict, closed=True):
    name: str

ClosedMovie(name="No Country for Old Men")  # OK
ClosedMovie(
    name="No Country for Old Men",
    year=2007,
)  # Not OK. Extra items not allowed

Interaction with Mapping[KT, VT]

A TypedDict type can be assignable to Mapping[KT, VT] types other than Mapping[str, object] as long as all value types of the items on the TypedDict type is assignable to VT. This is an extension of this assignability rule from the typing spec:

  • A TypedDict with all int values is not assignable to Mapping[str, int], since there may be additional non-int values not visible through the type, due to structural assignability. These can be accessed using the values() and items() methods in Mapping,

For example:

class MovieExtraStr(TypedDict, extra_items=str):
    name: str

extra_str: MovieExtraStr = {"name": "Blade Runner", "summary": ""}
str_mapping: Mapping[str, str] = extra_str  # OK

int_mapping: Mapping[str, int] = extra_int  # Not OK. 'int | str' is not assignable with 'int'
int_str_mapping: Mapping[str, int | str] = extra_int  # OK

Type checkers should be able to infer the precise return types of values() and items() on such TypedDict types:

def fun(movie: MovieExtraStr) -> None:
    reveal_type(movie.items())  # Revealed type is 'dict_items[str, str]'
    reveal_type(movie.values())  # Revealed type is 'dict_values[str, str]'

Interaction with dict[KT, VT]

Note that because the presence of extra_items on a closed TypedDict type prohibits additional required keys in its structural typing:subtypes, we can determine if the TypedDict type and its structural subtypes will ever have any required key during static analysis.

The TypedDict type is assignable to dict[str, VT] if all items on the TypedDict type satisfy the following conditions:

  • The value type of the item is consistent with VT.
  • The item is not read-only.
  • The item is not required.

For example:

class IntDict(TypedDict, extra_items=int):
    pass

class IntDictWithNum(IntDict):
    num: NotRequired[int]

def f(x: IntDict) -> None:
    v: dict[str, int] = x  # OK
    v.clear()  # OK

not_required_num_dict: IntDictWithNum = {"num": 1, "bar": 2}
regular_dict: dict[str, int] = not_required_num_dict  # OK
f(not_required_num_dict)  # OK

In this case, methods that are previously unavailable on a TypedDict are allowed:

not_required_num.clear()  # OK

reveal_type(not_required_num.popitem())  # OK. Revealed type is tuple[str, int]

However, dict[str, VT] is not necessarily assignable to a TypedDict type, because such dict can be a subtype of dict:

class CustomDict(dict[str, int]):
    pass

not_a_regular_dict: CustomDict = {"num": 1}
int_dict: IntDict = not_a_regular_dict  # Not OK

How to Teach This

The choice of the spelling "extra_items" is intended to make this feature more understandable to new users compared to shorter alternatives like "extra".

Details of this should be documented in both the typing spec and the typing documentation.

Backwards Compatibility

Because extra_items is an opt-in feature, no existing codebase will break due to this change.

Note that closed and extra_items as keyword arguments do not collide with othere keys when using something like TD = TypedDict("TD", foo=str, bar=int), because this syntax has already been removed in Python 3.13.

Because this is a type-checking feature, it can be made available to older versions as long as the type checker supports it.

Open Issues

Use a Special __extra_items__ Key with the closed Class Parameter

In an earlier revision of this proposal, we discussed an approach that would utilize __extra_items__’s value type to specify the type of extra items accepted, like so:

class IntDict(TypedDict, closed=True):
    __extra_items__: int

where closed=True is required for __extra_items__ to be treated specially, to avoid key collision.

Some members of the community concern about the elegance of the syntax. Practiaclly, the key collision with a regular key can be mitigated with workarounds, but since using a reserved key is central to this proposal, there are limited ways forward to address the concerns.

Support a New Syntax of Specifying Keys

By introducing a new syntax that allows specifying string keys, we could deprecate the functional syntax of defining TypedDict types and address the key conflict issues if we decide to reserve a special key to type extra items.

For example:

class Foo(TypedDict):
    name: str  # Regular item
    _: bool    # Type of extra items
    __items__ = {
        "_": int,   # Literal "_" as a key
        "class": str,  # Keyword as a key
        "tricky.name?": float,  # Arbitrary str key
    }

This was proposed here by Jukka. The '_' key is chosen for not needing to invent a new name, and its similarity with the match statement.

This will allow us to deprecate the functional syntax of defining TypedDict types altogether, but there are some disadvantages. For example:

  • It’s less apparent to a reader that _: bool makes the TypedDict special, relative to adding a class argument like extra_items=bool.
  • It’s backwards incompatible with existing TypedDicts using the _: bool key. While such users have a way to get around the issue, it’s still a problem for them if they upgrade Python (or typing-extensions).
  • The types don’t appear in an annotation context, so their evaluation will not be deferred.

Rejected Ideas

Allowing Extra Items without Specifying the Type

extra=True was originally proposed for defining a TypedDict that accepts extra items regardless of the type, like how total=True works:

class ExtraDict(TypedDict, extra=True):
    pass

Because it did not offer a way to specify the type of the extra items, the type checkers will need to assume that the type of the extra items is Any, which compromises type safety. Furthermore, the current behavior of TypedDict already allows untyped extra items to be present in runtime, due to structural assignability. closed=True plays a similar role in the current proposal.

Support Extra Items with Intersection

Supporting intersections in Python’s type system requires a lot of careful consideration, and it can take a long time for the community to reach a consensus on a reasonable design.

Ideally, extra items in TypedDict should not be blocked by work on intersections, nor does it necessarily need to be supported through intersections.

Moreover, the intersection between Mapping[...] and TypedDict is not equivalent to a TypedDict type with the proposed extra_items special item, as the value type of all known items in TypedDict needs to satisfy the is-subtype-of relation with the value type of Mapping[...].

Requiring Type Compatibility of the Known Items with extra_items

extra_items restricts the value type for keys that are unknown to the TypedDict type. So the value type of any known item is not necessarily assignable to extra_items, and extra_items is not necessarily assignable to the value types of all known items.

This differs from TypeScript’s Index Signatures syntax, which requires all properties’ types to match the string index’s type. For example:

interface MovieWithExtraNumber {
    name: string // Property 'name' of type 'string' is not assignable to 'string' index type 'number'.
    [index: string]: number
}

interface MovieWithExtraNumberOrString {
    name: string // OK
    [index: string]: number | string
}

This is a known limitation discussed in TypeScript’s issue tracker, where it is suggested that there should be a way to exclude the defined keys from the index signature so that it is possible to define a type like MovieWithExtraNumber.

Reference Implementation

An earlier revision of proposal is supported in pyright 1.1.352, and pyanalyze 0.12.0.

Acknowledgments

Thanks to Jelle Zijlstra for sponsoring this PEP and providing review feedback, Eric Traut who proposed the original design this PEP iterates on, and Alice Purcell for offering their perspective as the author of PEP 705.


Source: https://github.com/python/peps/blob/main/peps/pep-0728.rst

Last modified: 2024-10-19 02:14:30 GMT