Abstract

Currently **kwargs can be type hinted as long as all of the keyword arguments specified by them are of the same type. However, that behaviour can be very limiting. Therefore, in this PEP we propose a new way to enable more precise **kwargs typing. The new approach revolves around using TypedDict to type **kwargs that comprise keyword arguments of different types. It also involves introducing a grammar change and a new dunder __typing_unpack__.

Motivation

Currently annotating **kwargs with a type T means that the kwargs type is in fact dict[str, T]. For example:

def foo(**kwargs: str) -> None: ...

means that all keyword arguments in foo are strings (i.e., kwargs is of type dict[str, str]). This behaviour limits the ability to type annotate **kwargs only to the cases where all of them are of the same type. However, it is often the case that keyword arguments conveyed by **kwargs have different types that are dependent on the keyword’s name. In those cases type annotating **kwargs is not possible. This is especially a problem for already existing codebases where the need of refactoring the code in order to introduce proper type annotations may be considered not worth the effort. This in turn prevents the project from getting all of the benefits that type hinting can provide.

Moreover, **kwargs can be used to reduce the amount of code needed in cases when there is a top-level function that is a part of a public API and it calls a bunch of helper functions, all of which expect the same keyword arguments. Unfortunately, if those helper functions were to use **kwargs, there is no way to properly type hint them if the keyword arguments they expect are of different types. In addition, even if the keyword arguments are of the same type, there is no way to check whether the function is being called with keyword names that it actually expects.

As described in the Intended Usage section, using **kwargs is not always the best tool for the job. Despite that, it is still a widely used pattern. As a consequence, there has been a lot of discussion around supporting more precise **kwargs typing and it became a feature that would be valuable for a large part of the Python community. This is best illustrated by the mypy GitHub issue 4441 which contains a lot of real world cases that could benefit from this propsal.

One more use case worth mentioning for which **kwargs are also convenient, is when a function should accommodate optional keyword-only arguments that don’t have default values. A need for a pattern like that can arise when values that are usually used as defaults to indicate no user input, such as None, can be passed in by a user and should result in a valid, non-default behavior. For example, this issue came up in the popular httpx library.

Rationale

PEP 589 introduced the TypedDict type constructor that supports dictionary types consisting of string keys and values of potentially different types. A function’s keyword arguments represented by a formal parameter that begins with double asterisk, such as **kwargs, are received as a dictionary. Additionally, such functions are often called using unpacked dictionaries to provide keyword arguments. This makes TypedDict a perfect candidate to be used for more precise **kwargs typing. In addition, with TypedDict keyword names can be taken into account during static type analysis. However, specifying **kwargs type with a TypedDict means, as mentioned earlier, that each keyword argument specified by **kwargs is a TypedDict itself. For instance:

class Movie(TypedDict):
    name: str
    year: int

def foo(**kwargs: Movie) -> None: ...

means that each keyword argument in foo is itself a Movie dictionary that has a name key with a string type value and a year key with an integer type value. Therefore, in order to support specifying kwargs type as a TypedDict without breaking current behaviour, a new syntax has to be introduced.

Specification

To support the aforementioned use case we propose to use the double asterisk syntax inside of the type annotation. The required grammar change is discussed in more detail in section Grammar Changes. Continuing the previous example:

def foo(**kwargs: **Movie) -> None: ...

would mean that the **kwargs comprise two keyword arguments specified by Movie (i.e. a name keyword of type str and a year keyword of type int). This indicates that the function should be called as follows:

kwargs: Movie = {name: "Life of Brian", year: 1979}

foo(**kwargs)                               # OK!
foo(name="The Meaning of Life", year=1983)  # OK!

Inside the function itself, the type checkers should treat the kwargs parameter as a TypedDict:

def foo(**kwargs: **Movie) -> None:
    assert_type(kwargs, Movie)  # OK!

Using the new annotation will not have any runtime effect - it should only be taken into account by type checkers. Any mention of errors in the following sections relates to type checker errors.

def foo(**kwargs: **Movie) -> None: ...

movie: dict[str, object] = {"name": "Life of Brian", "year": 1979}
foo(**movie)  # WRONG! Movie is of type dict[str, object]

typed_movie: Movie = {"name": "The Meaning of Life", "year": 1983}
foo(**typed_movie)  # OK!

another_movie = {"name": "Life of Brian", "year": 1979}
foo(**another_movie)  # Depends on the type checker.

def foo(name, **kwargs: **Movie) -> None: ...     # WRONG! "name" will
                                                  # always bind to the
                                                  # first parameter.

def foo(name, /, **kwargs: **Movie) -> None: ...  # OK! "name" is a
                                                  # positional parameter,
                                                  # so **kwargs can contain
                                                  # a "name" keyword.

class Movie(TypedDict):
    title: str
    year: NotRequired[int]

class Animal(TypedDict):
    name: str

class Dog(Animal):
    breed: str

def accept_animal(**kwargs: **Animal): ...
def accept_dog(**kwargs: **Dog): ...

accept_dog = accept_animal  # OK! Expression of type Dog can be
                            # assigned to a variable of type Animal.

accept_animal = accept_dog  # WRONG! Expression of type Animal
                            # cannot be assigned to a variable of type Dog.

class Example(TypedDict):
    animal: Animal
    string: str
    number: NotRequired[int]

def src(**kwargs: **Example): ...
def dest(*, animal: Dog, string: str, number: int = ...): ...

dest = src  # OK!

def dest(animal: Dog, string: str, number: int = ...): ...
dest(animal_instance, "some string")  # OK!
dest = src
dest(animal_instance, "some string")  # WRONG! The same call fails at
                                      # runtime now because 'src' expects
                                      # keyword arguments.

def dest(**Animal): ...
def src(name: str): ...

dog: Dog = {"name": "Daisy", "breed": "Labrador"}
animal: Animal = dog

dest = src      # WRONG!
dest(**animal)  # Fails at runtime.

def src(**kwargs): ...
def dest(**kwargs: **Movie): ...

dest = src  # OK!

class Vehicle:
    ...

class Car(Vehicle):
    ...

class Motorcycle(Vehicle):
    ...

class Vehicles(TypedDict):
    car: Car
    moto: Motorcycle

def dest(**kwargs: **Vehicles): ...
def src(**kwargs: Vehicle): ...

dest = src  # OK!

class Animal(TypedDict):
    name: str

class Dog(Animal):
    breed: str

def takes_name(name: str): ...

dog: Dog = {"name": "Daisy", "breed": "Labrador"}
animal: Animal = dog

def foo(**kwargs: **Animal):
    print(kwargs["name"].capitalize())

def bar(**kwargs: **Animal):
    takes_name(**kwargs)

def baz(animal: Animal):
    takes_name(**animal)

def spam(**kwargs: **Animal):
    baz(kwargs)

foo(**animal)   # OK! foo only expects and uses keywords of 'Animal'.

bar(**animal)   # WRONG! This will fail at runtime because 'breed' keyword
                # will be passed to 'takes_name' as well.

spam(**animal)  # WRONG! Again, 'breed' keyword will be eventually passed
                # to 'takes_name'.

def bar(**kwargs: **Animal):
    name = kwargs["name"]
    takes_name(name)

def foo(name: str, year: int): ...  # Preferred way.
def foo(**kwargs: **Movie): ...

def foo(name, year): ...              # Function in a third party library.

def foo(**Movie): ...                 # Function signature in a stub file.

foo("Life of Brian", 1979)            # This would be now failing type
                                      # checking but is fine.

foo(name="Life of Brian", year=1979)  # This would be the only way to call
                                      # the function now that passes type
                                      # checking.

def foo(name: str, year: int): ...

Grammar Changes

This PEP requires a grammar change so that the double asterisk syntax is allowed for **kwargs annotations. The proposed change is to extend the kwds rule in the grammar as follows:

Before:

kwds: '**' param_no_default

After:

kwds:
    | '**' param_no_default_double_star_annotation
    | '**' param_no_default

param_no_default_double_star_annotation:
    | param_double_star_annotation ','? &')'

param_double_star_annotation: NAME double_star_annotation

double_star_annotation: ':' double_star_expression

double_star_expression: '**' expression

A new AST node needs to be created so that type checkers can differentiate the semantics of the new syntax from the existing one, which indicates that all **kwargs should be of the same type. Then, whenever the new syntax is used, type checkers will be able to take into account that **kwargs should be unpacked. The proposition is to add a new DoubleStarred AST node. Then, an AST node for the function defined as:

def foo(**kwargs: **Movie): ...

should look as below:

FunctionDef(
  name='foo',
  args=arguments(
    posonlyargs=[],
    args=[],
    kwonlyargs=[],
    kw_defaults=[],
    kwarg=arg(
      arg='kwargs',
      annotation=DoubleStarred(
        value=Name(id='Movie', ctx=Load()),
        ctx=Load())),
    defaults=[]),
  body=[
    Expr(
      value=Constant(value=Ellipsis))],
  decorator_list=[])

The runtime annotations should be consistent with the AST. Continuing the previous example:

>>> def foo(**kwargs: **Movie): ...
...
>>> foo.__annotations__
{'kwargs': Unpack[Movie]}

To accomplish this, we propose a new dunder called __typing_unpack__. The double asterisk syntax should result in a call to the __typing_unpack__ special method on an object it was used on. This means that at runtime, def foo(**kwargs: **T): ... is equivalent to def foo(**kwargs: type(T).__typing_unpack__(T)): .... TypedDict is the only type in the standard library that is expected to implement __typing_unpack__, which should return Unpack[self]. The motivation for reusing PEP 646’s Unpack is described in the Backwards Compatibility section.

It is worth pointing out that currently using Unpack in the context of typing is interchangeable with using the asterisk syntax:

>>> Unpack[Movie]
*<class '__main__.Movie'>

Therefore, in order to be compatible with the new usecase, Unpack’s repr should be changed to simply Unpack[T].

def foo(**kwargs: Unpack[Movie]) -> None: ...

How to Teach This

This PEP could be linked in the typing module’s documentation. Moreover, a new section on using Unpack as well as the new double asterisk syntax could be added to the aforementioned docs. Similar sections could be also added to the mypy documentation and the typing RTD documentation.

Reference Implementation

The mypy type checker already supports more precise **kwargs typing using Unpack.

Pyright type checker also provides provisional support for this feature.

A proof-of-concept implementation of the CPython grammar changes described in this PEP is available on GitHub.

Rejected Ideas

class Book(TypedDict):
    genre: str
    pages: int

TypedDictUnion = Movie | Book

def foo(**kwargs: **TypedDictUnion) -> None: ...  # WRONG! Unsupported use
                                                  # of a union of
                                                  # TypedDicts to type
                                                  # **kwargs

@overload
def foo(**kwargs: **Movie): ...

@overload
def foo(**kwargs: **Book): ...

Copyright

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.