PEPs are the only specification

The Python type system was initially created by a PEP (PEP 484), and changes to the type system are still made by PEPs. The specification for the Python type system, to the extent there is one, consists of this series of PEPs. But Standards Track PEPs aren’t meant to be living documents or specifications; they are change proposals.

An example may illustrate the problem here. Around the same time as the introduction of the typing module by PEP 484, PEP 3156 introduced the asyncio module, another major new feature that has been instrumental to the success of Python 3. Both modules have evolved greatly since their initial creation and inspired changes to the core language.

However, asyncio and typing are different in an essential aspect: a user who uses asyncio interacts only with the standard library itself, while a user of typing has to also think about an external tool, the type checker. The Python language reference covers the symbols in the typing module, but does not (and should not) go into detail on how the full type system should be interpreted by type checkers. That material currently exists only in the PEPs.

This problem is shared by the packaging ecosystem, which attempts to solve it by maintaining a separate set of PyPA specifications.

It’s hard to evolve the specification

Because the PEPs are the only specification we have, anything that could be seen as a change to the specification would theoretically require a new PEP. But that is often too heavy a process for a small change. Sometimes changes are made directly to old PEPs instead, but that goes against the idea that accepted and implemented PEPs become historical documents that should no longer be changed.

Some concrete examples include:

• PEP 484 explicitly says that typing.NoReturn cannot be used in argument annotations. Nevertheless, type checkers have long accepted such usage.
• A 2023 discussion noted that PEP 561’s description of partial stubs is unclear, and major type checkers did not implement it exactly as specified.
• The widely used third-party typing_extensions package provides backports of new type system features. Type checkers are expected to treat symbols in this module the same as symbols in typing, but this is not explicitly specified in any of the PEPs.

The type system is underspecified

While the PEPs provide a specification, they are often not sufficiently precise (sometimes intentionally so). This is especially true as the combinatorial complexity of the type system has grown.

It ends up falling to individual type checkers to decide how to navigate underspecified areas. In cases where type checkers informally coordinate, this results in de facto standards that aren’t clearly recorded anywhere, making the type system less accessible to newcomers. For example:

• How @overload matching works
• How ParamSpec should work with methods
• The concept of recursive aliases
• Semantics of variable initialization
• Reachability semantics of annotations on __exit__
• Symbol visibility
• Use of NoReturn for exhaustiveness checking

The Steering Council is not well-placed to solve the above problems

The SC has the entire language in its remit, and is not well-placed to make decisions that are purely about the type system – if only because they don’t have the time to deal with type system arcana alongside their other responsibilities. This is similar in spirit to the reasons why the Steering Council sometimes uses PEP delegation.

Endorsements

This PEP was endorsed by maintainers of all major type checkers, including Rebecca Chen (pytype), Eric Traut (Pyright), and privately by maintainers of mypy and Pyre.

Mandate

The Typing Council’s mandate is to ensure that the Python type system is:

• Useful: The type system should serve common use cases. As identified by PEP 484, the primary use case is static analysis, but there are others, such as runtime type checking, static compilation, IDE support, and documentation. The Typing Council should consider all of these use cases when making decisions, and be open to supporting additional use cases as they come up.
• Usable: The type system should be easy to use for Python developers. It should be ergonomic to write well-typed Python code that is accepted by type checkers. There should be good documentation for the type system.
• Stable: As the type system matures, users should be able to rely on their typed code continuing to work and be able to trust their mental model for the type system. Changes should be made with care and in a way that minimizes disruption. Nevertheless, the type system should be able to evolve, and it does not make sense to use the same compatibility guidelines for type checker behavior as for Python itself. Of course, the existence and runtime behavior of objects in the typing module does follow Python’s standard compatibility policy in PEP 387.

Operations and process

The council would have three to five members, comprised of prominent community members, such as Python core developers and maintainers of major type checkers. The members should include people affiliated with a variety of projects related to type checking, which may include type checkers, CPython, typeshed, or other projects.

The initial members of the council are:

• Eric Traut (Pyright; author of PEP 647, PEP 681, and PEP 695)
• Guido van Rossum (core developer; author of PEP 484 and PEP 526)
• Jelle Zijlstra (core developer; typeshed; pyanalyze; author of PEP 688 and PEP 702)
• Rebecca Chen (pytype)
• Shantanu Jain (core developer; typeshed; mypy)

There is no term limit for council members. Council members may resign their position at any time. There is an expectation that each member serves at most five consecutive years before resigning.

If there is a vacancy and there are three or more remaining members, it is up to the Council to decide whether to appoint a new member. To determine replacements, nominations will be collected from the typing community. Self-nominations are allowed. The existing Typing Council will then decide the replacement member(s) from the nominees. The expectation is that this would be done by fiat, but the Typing Council can choose a replacement by any means they see fit, including a vote.

The Typing Council remains accountable to the Steering Council. At any point, for any reason, the Steering Council could (publicly or privately) make a specific change or request a non-specific change to the composition of the Typing Council.

We acknowledge that this is a not particularly democratic structure and puts a lot of faith in the Typing Council. However, the Python community has a long history of success with not particularly democratic structures! We believe self-governance, cycling of membership, and accountability to the Steering Council will be sufficient to ensure that the Typing Council is meeting the needs of the community.

The council would operate primarily through reviews of GitHub PRs. Regular meetings are likely not necessary, but the council may set up video calls, a private chat, or whatever other mechanism they decide upon internally.

The council should aim for transparency, posting all decisions publicly on discuss.python.org, with a rationale if possible. Before making a decision, the council should give all interested community members a chance to weigh in. There should be at least a week between the start of a discussion and the council’s decision.

Members of the council will be eligible to sponsor PEPs. If this PEP is accepted, PEP 1 should be amended to note this fact.

Projects

Here are some efforts a Typing Council would be responsible for.

Amendments

This PEP serves as a charter for the Typing Council. Changes to its operation can be made either through a new PEP or through a change to this PEP. In either case, the change would be decided upon by the Steering Council after discussion in the community.

Writing the specification from scratch

This PEP proposes creating the typing specification by starting from the existing PEPs, then clarifying and improving the specification as necessary. Some members of the community prefer to start from scratch, writing a new, more formal specification covering the entire type system. This could provide a more solid basis for the specification.

However, this would be a much larger undertaking. The existing formalization effort by Kevin Millikin is a good start, but so far covers only a subset of PEP 484. Covering the rest of the type system would likely require several times more effort when we consider that major type system features such as typing.Protocol, typing.Literal, and typing.TypedDict were introduced only after PEP 484. It is not clear that there is even energy in the community for such a huge undertaking. Even if someone steps up to do all the work of putting together a specification, lots of effort would be required from community members and type checker maintainers to consider whether the specification accurately reflects current behavior, and if not, whether the specification or the type checkers should change.

Starting with the existing PEPs creates a lower-quality specification, but it means that the Typing Council can immediately start making a difference anywhere in the type system by improving and clarifying the specification. A formalization effort can still proceed by gradually replacing sections of the specification.

Alternate governance mechanisms

An earlier draft of this PEP suggested that the Steering Council appoint members of the Typing Council each year. The current Steering Council suggested that it would be better to have the Typing Council self-organise and avoid the need for the Steering Council to continuously supervise the Typing Council.

Alternate governance mechanisms are possible, including more democratic ones, but these typically raise several thorny questions, require much heavier amounts of process and are potentially more divisive. For example, see the PEP 8000 series, or recent discussions about alternative governance in other Python subcommunities. Ultimately, the Typing Council exists under the authority of the Steering Council, and so can rely on it to bootstrap governance and serve as an accountability mechanism.

Do nothing

We are hopeful substantial progress will be made on projects that improve the type system regardless of whether this PEP is accepted. We anticipate projects like specification or the potential for PEP delegation would benefit more from a Typing Council, and projects like end user documentation would benefit less. Certainly the bottleneck is likely to be contributor effort, not governance.

However, currently the tools available to the community to resolve potential contention are either establishment of approximate consensus or the exercise of power by individual projects or contributors. While very valuable, the former is a slow process that can often end in inaction. The latter can result in a less consistent ecosystem. Finally, easily legible governance structures make the community more accessible and equitable.