Following system colour scheme Selected dark colour scheme Selected light colour scheme

Python Enhancement Proposals

PEP 712 – Adding a “converter” parameter to dataclasses.field

Author:
Joshua Cannon <joshdcannon at gmail.com>
Sponsor:
Eric V. Smith <eric at trueblade.com>
Discussions-To:
Discourse thread
Status:
Draft
Type:
Standards Track
Created:
01-Jan-2023
Python-Version:
3.13
Post-History:
27-Dec-2022, 19-Jan-2023, 23-Apr-2023

Table of Contents

Abstract

PEP 557 added dataclasses to the Python stdlib. PEP 681 added dataclass_transform() to help type checkers understand several common dataclass-like libraries, such as attrs, Pydantic, and object relational mapper (ORM) packages such as SQLAlchemy and Django.

A common feature these libraries provide over the standard library implementation is the ability for the library to convert arguments given at initialization time into the types expected for each field using a user-provided conversion function.

Therefore, this PEP adds a converter parameter to dataclasses.field() (along with the requisite changes to dataclasses.Field and dataclass_transform()) to specify the function to use to convert the input value for each field to the representation to be stored in the dataclass.

Motivation

There is no existing, standard way for dataclasses or third-party dataclass-like libraries to support argument conversion in a type-checkable way. To work around this limitation, library authors/users are forced to choose to:

  • Opt-in to a custom Mypy plugin. These plugins help Mypy understand the conversion semantics, but not other tools.
  • Shift conversion responsibility onto the caller of the dataclass constructor. This can make constructing certain dataclasses unnecessarily verbose and repetitive.
  • Provide a custom __init__ which declares “wider” parameter types and converts them when setting the appropriate attribute. This not only duplicates the typing annotations between the converter and __init__, but also opts the user out of many of the features dataclasses provides.
  • Provide a custom __init__ but without meaningful type annotations for the parameter types requiring conversion.

None of these choices are ideal.

Rationale

Adding argument conversion semantics is useful and beneficial enough that most dataclass-like libraries provide support. Adding this feature to the standard library means more users are able to opt-in to these benefits without requiring third-party libraries. Additionally third-party libraries are able to clue type-checkers into their own conversion semantics through added support in dataclass_transform(), meaning users of those libraries benefit as well.

Specification

New converter parameter

This specification introduces a new parameter named converter to the dataclasses.field() function. When an __init__ method is synthesized by dataclass-like semantics, if an argument is provided for the field, the dataclass object’s attribute will be assigned the result of calling the converter on the provided argument. If no argument is given and the field was constructed with a default value, the dataclass object’s attribute will be assigned the result of calling the converter on the provided default.

Adding this parameter also implies the following changes:

Example

@dataclasses.dataclass
class InventoryItem:
    # `converter` as a type
    id: int = dataclasses.field(converter=int)
    skus: tuple[int, ...] = dataclasses.field(converter=tuple[int, ...])
    # `converter` as a callable
    names: tuple[str, ...] = dataclasses.field(
      converter=lambda names: tuple(map(str.lower, names))
    )

    # The default value is also converted; therefore the following is not a
    # type error.
    stock_image_path: pathlib.PurePosixPath = dataclasses.field(
      converter=pathlib.PurePosixPath, default="assets/unknown.png"
    )

item1 = InventoryItem("1", [234, 765], ["PYTHON PLUSHIE", "FLUFFY SNAKE"])
# item1 would have the following values:
#   id=1
#   skus=(234, 765)
#   names=('python plushie', 'fluffy snake')
#   stock_image_path=pathlib.PurePosixPath("assets/unknown.png")

Impact on typing

A converter must be a callable that accepts a single positional argument, and the parameter type corresponding to this positional argument provides the type of the the synthesized __init__ parameter associated with the field.

In other words, the argument provided for the converter parameter must be compatible with Callable[[T], X] where T is the input type for the converter and X is the output type of the converter.

Type-checking the default value

Because the default value is unconditionally converted using converter, if arguments for both converter and default are provided to dataclasses.field(), the default argument’s type should be checked using the type of the single argument to the converter callable.

Converter return type

The return type of the callable must be a type that’s compatible with the field’s declared type. This includes the field’s type exactly, but can also be a type that’s more specialized (such as a converter returning a list[int] for a field annotated as list, or a converter returning an int for a field annotated as int | str).

Example

@dataclasses.dataclass
class Example:
    my_int: int = dataclasses.field(converter=int)
    my_tuple: tuple[int, ...] = dataclasses.field(converter=tuple[int, ...])
    my_cheese: Cheese = dataclasses.field(converter=make_cheese)

    # Although the default value is of type `str` and the field is declared to
    # be of type `pathlib.Path`, this is not a type error because the default
    # value will be converted.
    tmpdir: pathlib.Path = dataclasses.field(default="/tmp", converter=pathlib.Path)

Backward Compatibility

These changes don’t introduce any compatibility problems since they only introduce opt-in new features.

Security Implications

There are no direct security concerns with these changes.

How to Teach This

Documentation and examples explaining the new parameter and behavior will be added to the relevant sections of the docs site (primarily on dataclasses) and linked from the What’s New document.

The added documentation/examples will also cover the “common pitfalls” that users of converters are likely to encounter. Such pitfalls include:

  • Needing to handle None/sentinel values.
  • Needing to handle values that are already of the correct type.
  • Avoiding lambdas for converters, as the synthesized __init__ parameter’s type will become Any.

Reference Implementation

The attrs library already includes a converter parameter containing converter semantics.

CPython support is implemented on a branch in the author’s fork.

Rejected Ideas

Just adding “converter” to typing.dataclass_transform’s field_specifiers

The idea of isolating this addition to dataclass_transform() was briefly discussed on Typing-SIG where it was suggested to broaden this to dataclasses more generally.

Additionally, adding this to dataclasses ensures anyone can reap the benefits without requiring additional libraries.

Not converting default values

There are pros and cons with both converting and not converting default values. Leaving default values as-is allows type-checkers and dataclass authors to expect that the type of the default matches the type of the field. However, converting default values has two large advantages:

  1. Compatibility with attrs. Attrs unconditionally uses the converter to convert the default value.
  2. Simpler defaults. Allowing the default value to have the same type as user-provided values means dataclass authors get the same conveniences as their callers.

Automatic conversion using the field’s type

One idea could be to allow the type of the field specified (e.g. str or int) to be used as a converter for each argument provided. Pydantic’s data conversion has semantics which appear to be similar to this approach.

This works well for fairly simple types, but leads to ambiguity in expected behavior for complex types such as generics. E.g. For tuple[int, ...] it is ambiguous if the converter is supposed to simply convert an iterable to a tuple, or if it is additionally supposed to convert each element type to int.


Source: https://github.com/python/peps/blob/main/pep-0712.rst

Last modified: 2023-04-23 20:24:27+00:00 GMT