PEP 696 – Type defaults for TypeVarLikes

Author:: James Hilton-Balfe <gobot1234yt at gmail.com>
Sponsor:: Jelle Zijlstra <jelle.zijlstra at gmail.com>
Discussions-To:: Typing-SIG thread
Status:: Draft
Type:: Standards Track
Topic:: Typing
Created:: 14-Jul-2022
Python-Version:: 3.12
Post-History:: 22-Mar-2022, 08-Jan-2023

Table of Contents

Abstract

This PEP introduces the concept of type defaults for TypeVarLikes (TypeVar, ParamSpec and TypeVarTuple), which act as defaults for a type parameter when one is not specified or the constraint solver isn’t able to solve a type parameter to anything.

Default type argument support is available in some popular languages such as C++, TypeScript, and Rust. A survey of type parameter syntax in some common languages has been conducted by the author of PEP 695 and can be found in its Appendix A.

Motivation

T = TypeVar("T", default=int)  # This means that if no type is specified T = int

@dataclass
class Box(Generic[T]):
    value: T | None = None

reveal_type(Box())                      # type is Box[int]
reveal_type(Box(value="Hello World!"))  # type is Box[str]

One place this regularly comes up is Generator. I propose changing the stub definition to something like:

YieldT = TypeVar("YieldT")
SendT = TypeVar("SendT", default=None)
ReturnT = TypeVar("ReturnT", default=None)

class Generator(Generic[YieldT, SendT, ReturnT]): ...

Generator[int] == Generator[int, None] == Generator[int, None, None]

This is also useful for a Generic that is commonly over one type.

class Bot: ...

BotT = TypeVar("BotT", bound=Bot, default=Bot)

class Context(Generic[BotT]):
    bot: BotT

class MyBot(Bot): ...

reveal_type(Context().bot)         # type is Bot  # notice this is not Any which is what it would be currently
reveal_type(Context[MyBot]().bot)  # type is MyBot

Not only does this improve typing for those who explicitly use it, it also helps non-typing users who rely on auto-complete to speed up their development.

This design pattern is common in projects like:

discord.py — where the example above was taken from.
NumPy — the default for types like ndarray’s dtype would be float64. Currently it’s Unknown or Any.
TensorFlow — this could be used for Tensor similarly to numpy.ndarray and would be useful to simplify the definition of Layer.

Specification

Default Ordering and Subscription Rules

The order for defaults should follow the standard function parameter rules, so a TypeVarLike with no default cannot follow one with a default value. Doing so should ideally raise a TypeError in typing._GenericAlias/types.GenericAlias, and a type checker should flag this an error.

DefaultStrT = TypeVar("DefaultStrT", default=str)
DefaultIntT = TypeVar("DefaultIntT", default=int)
DefaultBoolT = TypeVar("DefaultBoolT", default=bool)
T = TypeVar("T")
T2 = TypeVar("T2")

class NonDefaultFollowsDefault(Generic[DefaultStrT, T]): ...  # Invalid: non-default TypeVars cannot follow ones with defaults


class NoNonDefaults(Generic[DefaultStrT, DefaultIntT]): ...

(
    NoNoneDefaults ==
    NoNoneDefaults[str] ==
    NoNoneDefaults[str, int]
)  # All valid


class OneDefault(Generic[T, DefaultBoolT]): ...

OneDefault[float] == OneDefault[float, bool]  # Valid
reveal_type(OneDefault)          # type is type[OneDefault[T, DefaultBoolT = bool]]
reveal_type(OneDefault[float]()) # type is OneDefault[float, bool]


class AllTheDefaults(Generic[T1, T2, DefaultStrT, DefaultIntT, DefaultBoolT]): ...

reveal_type(AllTheDefaults)                  # type is type[AllTheDefaults[T1, T2, DefaultStrT = str, DefaultIntT = int, DefaultBoolT = bool]]
reveal_type(AllTheDefaults[int, complex]())  # type is AllTheDefaults[int, complex, str, int, bool]
AllTheDefaults[int]  # Invalid: expected 2 arguments to AllTheDefaults
(
    AllTheDefaults[int, complex] ==
    AllTheDefaults[int, complex, str] ==
    AllTheDefaults[int, complex, str, int] ==
    AllTheDefaults[int, complex, str, int, bool]
)  # All valid

This cannot be enforced at runtime for functions, for now, but in the future, this might be possible (see Interaction with PEP 695).

`ParamSpec` Defaults

ParamSpec defaults are defined using the same syntax as TypeVar s but use a list of types or an ellipsis literal “...” or another in-scope ParamSpec (see Scoping Rules).

DefaultP = ParamSpec("DefaultP", default=[str, int])

class Foo(Generic[DefaultP]): ...

reveal_type(Foo)                  # type is type[Foo[DefaultP = [str, int]]]
reveal_type(Foo())                # type is Foo[[str, int]]
reveal_type(Foo[[bool, bool]]())  # type is Foo[[bool, bool]]

`TypeVarTuple` Defaults

TypeVarTuple defaults are defined using the same syntax as TypeVar s but use an unpacked tuple of types instead of a single type or another in-scope TypeVarTuple (see Scoping Rules).

DefaultTs = TypeVarTuple("DefaultTs", default=Unpack[tuple[str, int]])

class Foo(Generic[*DefaultTs]): ...

reveal_type(Foo)               # type is type[Foo[DefaultTs = *tuple[str, int]]]
reveal_type(Foo())             # type is Foo[str, int]
reveal_type(Foo[int, bool]())  # type is Foo[int, bool]

Using Another `TypeVarLike` as `default`

This allows for a value to be used again when the constraints solver fails to solve a constraint for a type, or the type parameter to a generic is missing but another type parameter is specified.

To use another TypeVarLike as a default the default and the TypeVarLike must be the same type (a TypeVar’s default must be a TypeVar, etc.).

This could be used on builtins.slice where the start parameter should default to int, stop default to the type of start and step default to int | None.

StartT = TypeVar("StartT", default=int)
StopT = TypeVar("StopT", default=StartT)
StepT = TypeVar("StepT", default=int | None)

class slice(Generic[StartT, StopT, StepT]): ...

reveal_type(slice)  # type is type[slice[StartT = int, StopT = StartT, StepT = int | None]]
reveal_type(slice())                        # type is slice[int, int, int | None]
reveal_type(slice[str]())                   # type is slice[str, str, int | None]
reveal_type(slice[str, bool, timedelta]())  # type is slice[str, bool, timedelta]

T2 = TypeVar("T2", default=DefaultStrT)

class Foo(Generic[DefaultStrT, T2]):
    def __init__(self, a: DefaultStrT, b: T2) -> None: ...

reveal_type(Foo(1, ""))  # type is Foo[int, str]
Foo[int](1, "")          # Invalid: Foo[int, str] cannot be assigned to self: Foo[int, int] in Foo.__init__
Foo[int]("", 1)          # Invalid: Foo[str, int] cannot be assigned to self: Foo[int, int] in Foo.__init__

When using a TypeVarLike as the default to another TypeVarLike. Where T1 is the default for T2 the following rules apply.

TypeVarTuples are not supported because:

Scoping Rules does not allow usage of TypeVarLikes from outer scopes.
Multiple TypeVarTuples cannot appear in the type parameter list for a single class, as specified in PEP 646.
TypeVarLike defaults in functions are not supported.

These reasons leave no current valid location where a TypeVarTuple could have a default.

Scoping Rules

T1 must be used before T2 in the parameter list of the generic.

DefaultT = TypeVar("DefaultT", default=T)

class Foo(Generic[T, DefaultT]): ...   # Valid
class Foo(Generic[T]):
    class Bar(Generic[DefaultT]): ...  # Valid

StartT = TypeVar("StartT", default="StopT")  # Swapped defaults around from previous example
StopT = TypeVar("StopT", default=int)
class slice(Generic[StartT, StopT, StepT]): ...
                  # ^^^^^^ Invalid: ordering does not allow StopT to be bound

Using a TypeVarLike from an outer scope as a default is not supported.

Bound Rules

T2’s bound must be a subtype of T1’s bound.

T = TypeVar("T", bound=float)
TypeVar("Ok", default=T, bound=int)        # Valid
TypeVar("AlsoOk", default=T, bound=float)  # Valid
TypeVar("Invalid", default=T, bound=str)   # Invalid: str is not a subtype of float

Constraint Rules

The constraints of T2 must be a superset of the constraints of T1.

T1 = TypeVar("T1", bound=int)
TypeVar("Invalid", float, str, default=T1)         # Invalid: upper bound int is incompatible with constraints float or str

T1 = TypeVar("T1", int, str)
TypeVar("AlsoOk", int, str, bool, default=T1)      # Valid
TypeVar("AlsoInvalid", bool, complex, default=T1)  # Invalid: {bool, complex} is not a superset of {int, str}

`TypeVarLike`s as Parameters to Generics

TypeVarLikes are valid as parameters to generics inside of a default when the first parameter is in scope as determined by the previous section.

T = TypeVar("T")
ListDefaultT = TypeVar("ListDefaultT", default=list[T])

class Bar(Generic[T, ListDefaultT]):
    def __init__(self, x: T, y: ListDefaultT): ...

reveal_type(Bar)                    # type is type[Bar[T, ListDefaultT = list[T]]]
reveal_type(Bar[int])               # type is type[Bar[int, list[int]]]
reveal_type(Bar[int]())             # type is Bar[int, list[int]]
reveal_type(Bar[int, list[str]]())  # type is Bar[int, list[str]]
reveal_type(Bar[int, str]())        # type is Bar[int, str]

Specialisation Rules

TypeVarLikes currently cannot be further subscripted. This might change if Higher Kinded TypeVars are implemented.

`Generic` `TypeAlias`es

Generic TypeAliases should be able to be further subscripted following normal subscription rules. If a TypeVarLike has a default that hasn’t been overridden it should be treated like it was substituted into the TypeAlias. However, it can be specialised further down the line.

class SomethingWithNoDefaults(Generic[T, T2]): ...

MyAlias: TypeAlias = SomethingWithNoDefaults[int, DefaultStrT]  # Valid
reveal_type(MyAlias)          # type is type[SomethingWithNoDefaults[int, DefaultStrT]]
reveal_type(MyAlias[bool]())  # type is SomethingWithNoDefaults[int, bool]

MyAlias[bool, int]  # Invalid: too many arguments passed to MyAlias

Subclassing

Subclasses of Generics with TypeVarLikes that have defaults behave similarly to Generic TypeAliases.

class SubclassMe(Generic[T, DefaultStrT]):
    x: DefaultStrT

class Bar(SubclassMe[int, DefaultStrT]): ...
reveal_type(Bar)          # type is type[Bar[DefaultStrT = str]]
reveal_type(Bar())        # type is Bar[str]
reveal_type(Bar[bool]())  # type is Bar[bool]

class Foo(SubclassMe[float]): ...

reveal_type(Foo().x)  # type is str

Foo[str]  # Invalid: Foo cannot be further subscripted

class Baz(Generic[DefaultIntT, DefaultStrT]): ...

class Spam(Baz): ...
reveal_type(Spam())  # type is <subclass of Baz[int, str]>

Using `bound` and `default`

If both bound and default are passed default must be a subtype of bound. Otherwise the type checker should generate an error.

TypeVar("Ok", bound=float, default=int)     # Valid
TypeVar("Invalid", bound=str, default=int)  # Invalid: the bound and default are incompatible

Constraints

For constrained TypeVars, the default needs to be one of the constraints. A type checker should generate an error even if it is a subtype of one of the constraints.

TypeVar("Ok", float, str, default=float)     # Valid
TypeVar("Invalid", float, str, default=int)  # Invalid: expected one of float or str got int

Function Defaults

TypeVarLikes currently are not supported in the signatures of functions as ensuring the default is returned in every code path where the TypeVarLike can go unsolved is too hard to implement.

Binding rules

TypeVarLikes defaults should be bound by attribute access (including call and subscript).

class Foo[T = int]:
    def meth(self) -> Self:
        return self

reveal_type(Foo.meth)  # type is (self: Foo[int]) -> Foo[int]

Implementation

At runtime, this would involve the following changes to the typing module.

The classes TypeVar, ParamSpec, and TypeVarTuple should expose the type passed to default. This would be available as a __default__ attribute, which would be None if no argument is passed and NoneType if default=None.

The following changes would be required to both GenericAliases:

logic to determine the defaults required for a subscription.
ideally, logic to determine if subscription (like Generic[T, DefaultT]) would be valid.

A reference implementation of the runtime changes can be found at https://github.com/Gobot1234/cpython/tree/pep-696

A reference implementation of the type checker can be found at https://github.com/Gobot1234/mypy/tree/TypeVar-defaults

Pyright currently supports this functionality.

Interaction with PEP 695

The syntax proposed in PEP 695 will be extended to introduce a way to specify defaults for type parameters using the “=” operator inside of the square brackets like so:

# TypeVars
class Foo[T = str]: ...

# ParamSpecs
class Baz[**P = [int, str]]: ...

# TypeVarTuples
class Qux[*Ts = *tuple[int, bool]]: ...

# TypeAliases
type Foo[T, U = str] = Bar[T, U]
type Baz[**P = [int, str]] = Spam[**P]
type Qux[*Ts = *tuple[str]] = Ham[*Ts]
type Rab[U, T = str] = Bar[T, U]

Similarly to the bound for a type parameter, defaults should be lazily evaluated, with the same scoping rules to avoid the unnecessary usage of quotes around them.

This functionality was included in the initial draft of PEP 695 but was removed due to scope creep.

Grammar Changes

type_param:
    | a=NAME b=[type_param_bound] d=[type_param_default]
    | a=NAME c=[type_param_constraint] d=[type_param_default]
    | '*' a=NAME d=[type_param_default]
    | '**' a=NAME d=[type_param_default]

type_param_default:
    | '=' e=expression
    | '=' e=starred_expression

This would mean that TypeVarLikes with defaults proceeding those with non-defaults can be checked at compile time.

Rejected Alternatives

Allowing the `TypeVarLike`s Defaults to Be Passed to `type.new`’s `**kwargs`

T = TypeVar("T")

@dataclass
class Box(Generic[T], T=int):
    value: T | None = None

While this is much easier to read and follows a similar rationale to the TypeVar unary syntax, it would not be backwards compatible as T might already be passed to a metaclass/superclass or support classes that don’t subclass Generic at runtime.

Ideally, if PEP 637 wasn’t rejected, the following would be acceptable:

T = TypeVar("T")

@dataclass
class Box(Generic[T = int]):
    value: T | None = None

Allowing Non-defaults to Follow Defaults

YieldT = TypeVar("YieldT", default=Any)
SendT = TypeVar("SendT", default=Any)
ReturnT = TypeVar("ReturnT")

class Coroutine(Generic[YieldT, SendT, ReturnT]): ...

Coroutine[int] == Coroutine[Any, Any, int]

Allowing non-defaults to follow defaults would alleviate the issues with returning types like Coroutine from functions where the most used type argument is the last (the return). Allowing non-defaults to follow defaults is too confusing and potentially ambiguous, even if only the above two forms were valid. Changing the argument order now would also break a lot of codebases. This is also solvable in most cases using a TypeAlias.

Coro: TypeAlias = Coroutine[Any, Any, T]
Coro[int] == Coroutine[Any, Any, int]

Having `default` Implicitly Be `bound`

In an earlier version of this PEP, the default was implicitly set to bound if no value was passed for default. This while convenient, could have a TypeVarLike with no default follow a TypeVarLike with a default. Consider:

T = TypeVar("T", bound=int)  # default is implicitly int
U = TypeVar("U")

class Foo(Generic[T, U]):
    ...

# would expand to

T = TypeVar("T", bound=int, default=int)
U = TypeVar("U")

class Foo(Generic[T, U]):
    ...

This would have also been a breaking change for a small number of cases where the code relied on Any being the implicit default.

Allowing `TypeVarLike`s with defaults to be used in function signatures

A previous version of this PEP allowed TypeVarLikes with defaults to be used in function signatures. This was removed for the reasons described in Function Defaults. Hopefully, this can be added in the future if a way to get the runtime value of a type parameter is added.

Last modified: 2023-09-14 13:33:43 GMT

Python Enhancement Proposals