Following system colour scheme Selected dark colour scheme Selected light colour scheme

Python Enhancement Proposals

PEP 696 – Type defaults for TypeVarLikes

Author:
James Hilton-Balfe <gobot1234yt at gmail.com>
Sponsor:
Jelle Zijlstra <jelle.zijlstra at gmail.com>
Discussions-To:
Typing-SIG list
Status:
Draft
Type:
Standards Track
Topic:
Typing
Created:
14-Jul-2022
Python-Version:
3.12
Table of Contents

Abstract

This PEP introduces the concept of type defaults for TypeVarLikes (TypeVar, ParamSpec and TypeVarTuple), which act as defaults for a type parameter when none is specified.

Default type argument support is available in some popular languages such as C++, TypeScript, and Rust. A survey of type parameter syntax in some common languages has been conducted by the author of PEP 695 and can be found in its Appendix A.

Motivation

T = TypeVar("T", default=int)  # This means that if no type is specified T = int

@dataclass
class Box(Generic[T]):
    value: T | None = None

reveal_type(Box())                      # type is Box[int]
reveal_type(Box(value="Hello World!"))  # type is Box[str]

One place this regularly comes up is Generator. I propose changing the stub definition to something like:

YieldT = TypeVar("YieldT")
SendT = TypeVar("SendT", default=None)
ReturnT = TypeVar("ReturnT", default=None)

class Generator(Generic[YieldT, SendT, ReturnT]): ...

Generator[int] == Generator[int, None] == Generator[int, None, None]

This is also useful for a Generic that is commonly over one type.

class Bot: ...

BotT = TypeVar("BotT", bound=Bot, default=Bot)

class Context(Generic[BotT]):
    bot: BotT

class MyBot(Bot): ...

reveal_type(Context().bot)         # type is Bot  # notice this is not Any which is what it would be currently
reveal_type(Context[MyBot]().bot)  # type is MyBot

Not only does this improve typing for those who explicitly use it, it also helps non-typing users who rely on auto-complete to speed up their development.

This design pattern is common in projects like:
  • discord.py - where the example above was taken from.
  • NumPy - the default for types like ndarray’s dtype would be float64. Currently it’s Unknown or Any.
  • TensorFlow (this could be used for Tensor similarly to numpy.ndarray and would be useful to simplify the definition of Layer).

Specification

Default ordering and subscription rules

The order for defaults should follow the standard function parameter rules, so a TypeVarLike with no default cannot follow one with a default value. Doing so should ideally raise a TypeError in typing._GenericAlias/types.GenericAlias, and a type checker should flag this an error.

DefaultStrT = TypeVar("DefaultStrT", default=str)
DefaultIntT = TypeVar("DefaultIntT", default=int)
DefaultBoolT = TypeVar("DefaultBoolT", default=bool)
T = TypeVar("T")
T2 = TypeVar("T2")

class NonDefaultFollowsDefault(Generic[DefaultStrT, T]): ...  # Invalid: non-default TypeVars cannot follow ones with defaults


class NoNonDefaults(Generic[DefaultStrT, DefaultIntT]): ...

(
    NoNoneDefaults ==
    NoNoneDefaults[str] ==
    NoNoneDefaults[str, int]
)  # All valid


class OneDefault(Generic[T, DefaultBoolT]): ...

OneDefault[float] == OneDefault[float, bool]  # Valid


class AllTheDefaults(Generic[T1, T2, DefaultStrT, DefaultIntT, DefaultBoolT]): ...

AllTheDefaults[int]  # Invalid: expected 2 arguments to AllTheDefaults
(
    AllTheDefaults[int, complex] ==
    AllTheDefaults[int, complex, str] ==
    AllTheDefaults[int, complex, str, int] ==
    AllTheDefaults[int, complex, str, int, bool]
)  # All valid

This cannot be enforced at runtime for functions, for now, but in the future, this might be possible (see Interaction with PEP 695).

ParamSpec Defaults

ParamSpec defaults are defined using the same syntax as TypeVar s but use a list or tuple of types or an ellipsis literal “...”.

DefaultP = ParamSpec("DefaultP", default=(str, int))

class Foo(Generic[DefaultP]): ...

reveal_type(Foo())                # type is Foo[(str, int)]
reveal_type(Foo[(bool, bool)]())  # type is Foo[(bool, bool)]

TypeVarTuple Defaults

TypeVarTuple defaults are defined using the same syntax as TypeVar s but use an unpacked tuple of types instead of a single type.

DefaultTs = TypeVarTuple("DefaultTs", default=Unpack[tuple[str, int]])

class Foo(Generic[DefaultTs]): ...

reveal_type(Foo())             # type is Foo[str, int]
reveal_type(Foo[int, bool]())  # type is Foo[int, bool]

Using another TypeVarLike as the default

To use another TypeVarLikes as the default they have to be of the same type. When using another TypeVarLike (T1) as the default, the default for the TypeVarLike (T2), T1 must be used before in the signature of the class it appears in before T2. T2’s bound must be a subtype of T1’s bound.

This could be used on builtins.slice where the start parameter should default to int, stop default to the type of start and step default to int | None.

StartT = TypeVar("StartT", default=int)
StopT = TypeVar("StopT", default=StartT)
StepT = TypeVar("StepT", default=int | None)

class slice(Generic[StartT, StopT, StepT]): ...  # Valid

reveal_type(slice())                       # type is slice[int, int, int | None]
reveal_type(slice[str]())                  # type is slice[str, str, int | None]
reveal_type(slice[str, str, timedelta]())  # type is slice[str, str, timedelta]

StartT = TypeVar("StartT", default="StopT")
StopT = TypeVar("StopT", default=int)
class slice(Generic[StartT, StopT, StepT]): ...
                    ^^^^^^  # Invalid: ordering does not allow StopT to bound yet

Generic TypeAliases

Generic TypeAliases should be able to be further subscripted following normal subscription rules. If a TypeVarLike has a default that hasn’t been overridden it should be treated like it was substituted into the TypeAlias. However, it can be specialised further down the line.

class SomethingWithNoDefaults(Generic[T, T2]): ...

MyAlias: TypeAlias = SomethingWithNoDefaults[int, DefaultStrT]  # valid
reveal_type(MyAlias())        # type is SomethingWithNoDefaults[int, str]
reveal_type(MyAlias[bool]())  # type is SomethingWithNoDefaults[int, bool]

MyAlias[bool, int]  # Invalid: too many arguments passed to MyAlias

Subclassing

Subclasses of Generics with TypeVarLikes that have defaults behave similarly to Generic TypeAliases.

class SubclassMe(Generic[T, DefaultStrT]): ...

class Bar(SubclassMe[int, DefaultStrT]): ...
reveal_type(Bar())        # type is Bar[str]
reveal_type(Bar[bool]())  # type is Bar[bool]

class Foo(SubclassMe[int]): ...

reveal_type(Foo())  # type is <subclass of SubclassMe[int, int]>

Foo[str]  # Invalid: Foo cannot be further subscripted

class Baz(Generic[DefaultIntT, DefaultStrT]): ...

class Spam(Baz): ...
reveal_type(Spam())  # type is <subclass of Baz[int, str]>

Using bound and default

If both bound and default are passed default must be a subtype of bound. Otherwise the type checker should generate an error.

TypeVar("Ok", bound=float, default=int)     # Valid
TypeVar("Invalid", bound=str, default=int)  # Invalid: the bound and default are incompatible

Constraints

For constrained TypeVars, the default needs to be one of the constraints. A type checker should generate an error even if it is a subtype of one of the constraints.

TypeVar("Ok", float, str, default=float)     # Valid
TypeVar("Invalid", float, str, default=int)  # Invalid: expected one of float or str got int

Function Defaults

TypeVarLikes currently can only be used where a parameter can go unsolved.

def foo(a: DefaultStrT | None = None) -> DefaultStrT: ...

reveal_type(foo(1)) # type is int
reveal_type(foo())  # type is str

If they are used where the parameter type is known, the defaults should just be ignored and a type checker can emit a warning.

Implementation

At runtime, this would involve the following changes to the typing module.

  • a new class _DefaultMixin needs to be added which is a superclass of TypeVar, ParamSpec, and TypeVarTuple.
    • the type passed to default would be available as a __default__ attribute.

The following changes would be required to both GenericAliases:

  • logic to determine the defaults required for a subscription.
  • ideally, logic to determine if subscription (like Generic[T, DefaultT]) would be valid.

A reference implementation of the type checker can be found at https://github.com/Gobot1234/mypy/tree/TypeVar-defaults

Interaction with PEP 695

If this PEP is accepted, the syntax proposed in PEP 695 will be extended to introduce a way to specify defaults for type parameters using the “=” operator inside of the square brackets like so:

# TypeVars
class Foo[T = str]: ...
def bar[U = int](): ...

# ParamSpecs
class Baz[**P = (int, str)]: ...
def spam[**Q = (bool,)](): ...

# TypeVarTuples
class Qux[*Ts = *tuple[int, bool]]: ...
def ham[*Us = *tuple[str]](): ...

This functionality was included in the initial draft of PEP 695 but was removed due to scope creep.

Grammar Changes

type_param:
    | a=NAME b=[type_param_bound] d=[type_param_default]
    | a=NAME c=[type_param_constraint] d=[type_param_default]
    | '*' a=NAME d=[type_param_default]
    | '**' a=NAME d=[type_param_default]

type_param_default: '=' e=expression

This would mean that TypeVarLikes with defaults proceeding those with non-defaults can be checked at compile time.

Rejected Alternatives

Allowing the TypeVarLikes defaults to be passed to type.__new__’s **kwargs

T = TypeVar("T")

@dataclass
class Box(Generic[T], T=int):
    value: T | None = None

While this is much easier to read and follows a similar rationale to the TypeVar unary syntax, it would not be backwards compatible as T might already be passed to a metaclass/superclass or support classes that don’t subclass Generic at runtime.

Ideally, if PEP 637 wasn’t rejected, the following would be acceptable:

T = TypeVar("T")

@dataclass
class Box(Generic[T = int]):
    value: T | None = None

Allowing non-defaults to follow defaults

YieldT = TypeVar("YieldT", default=Any)
SendT = TypeVar("SendT", default=Any)
ReturnT = TypeVar("ReturnT")

class Coroutine(Generic[YieldT, SendT, ReturnT]): ...

Coroutine[int] == Coroutine[Any, Any, int]

Allowing non-defaults to follow defaults would alleviate the issues with returning types like Coroutine from functions where the most used type argument is the last (the return). Allowing non-defaults to follow defaults is too confusing and potentially ambiguous, even if only the above two forms were valid. Changing the argument order now would also break a lot of codebases. This is also solvable in most cases using a TypeAlias.

Coro: TypeAlias = Coroutine[Any, Any, T]
Coro[int] == Coroutine[Any, Any, int]

Having default implicitly be bound

In an earlier version of this PEP, the default was implicitly set to bound if no value was passed for default. This while convenient, could have a TypeVarLike with no default follow a TypeVarLike with a default. Consider:

T = TypeVar("T", bound=int)  # default is implicitly int
U = TypeVar("U")

class Foo(Generic[T, U]):
    ...

# would expand to

T = TypeVar("T", bound=int, default=int)
U = TypeVar("U")

class Foo(Generic[T, U]):
    ...

This would have also been a breaking change for a small number of cases where the code relied on Any being the implicit default.

Acknowledgements

Thanks to the following people for their feedback on the PEP:

Eric Traut, Jelle Zijlstra, Joshua Butt, Danny Yamamoto, Kaylynn Morgan and Jakub Kuczys

Source: https://github.com/python/peps/blob/main/pep-0696.rst

Last modified: 2022-07-24 16:07:39 GMT