Abstract

This PEP proposes introducing JSON encoded core metadata and wheel file format metadata files in Python packages. Python package metadata (“core metadata”) was first defined in PEP 241 to use RFC 822 email headers to encode information about packages. This was reasonable in 2001; email messages were the only widely used, standardized text format that had a parser in the standard library. However, issues with handling different encodings, differing handling of line breaks, and other differences between implementations have caused numerous packaging bugs. Using the JSON format for encoding metadata files would eliminate a wide range of these potential issues.

Motivation

The email message format has a number of complexities and limitations which reduce its utility as a portable textual interchange format for packaging metadata. Due to the email parser requiring configuration changes to properly generate valid core metadata, many projects do not use the email module and instead generate core metadata in a custom manner. There are many pitfalls with generating email headers that can be encountered by such custom generators. First, core metadata fields may contain newlines in the value of fields. These newlines must be handled properly to “unfolded” multiple lines per RFC 822. One particularly difficult to encode field is the Description field, which may contain newlines and indentation. To encode the field in email headers, CRLF line breaks must be followed by seven (7) spaces and a pipe (’|’) character. While Description may now be encoded in the message body, similar escaping issues occur for the Author and Maintainer fields. Improperly escaped newlines can lead to missing, partial, or invalid core metadata. Second, as discussed in the core metadata specifications:

The standard file format for metadata (including in wheels and installed projects) is based on the format of email headers. However, email formats have been revised several times, and exactly which email RFC applies to packaging metadata is not specified. In the absence of a precise definition, the practical standard is set by what the standard library email.parser module can parse using the email.policy.compat32 policy.

Since no specific email RFC is selected, the current core metadata specification is ambiguous whether a given core metadata document is valid. RFC 822 is the only email standard to be explicitly listed in a PEP. However, the core metadata specifications also requires that core metadata is encoded using UTF-8 when written to a file. This de-facto makes the core metadata follow RFC 6532, which specifies internationalization of email headers. This has practical interoperability concerns. Until a few years ago, it was unspecified how to properly encode non-ASCII emails in core metadata, making parsing ambiguous. Third, the current format is difficult to properly validate and parse. Many tools do not check for issues with the output of the email parser. If a document is malformed, it may still parse without error by the email module as a valid email message. Furthermore, due to limitations in the email format, fields like Project-Url must create custom encodings of nested key-value items, further complicating parsing and validation. Finally, the lack of a schema makes it difficult to validate the contents of email message encoded metadata. While introducing a specification for the current format has been discussed previously, no progress had been made, and converting to JSON was a suggested resolution to the issues raised.

The WHEEL file format is currently encoded in a custom key-value format. While this format is easy to parse and write, it requires manual parsing and validation to ensure that the contents are valid. Moving to a JSON encoded format will allow for easier parsing and validation of the contents, and simplify packaging tools and services by using a consistent format for distribution metadata.

Rationale

Introducing a new core metadata file with a well-specified format will greatly ease generating, parsing, and validating metadata. JSON is a natural choice for storing package core metadata. It is easily machine readable and writable, is understandable to humans, and is well supported across many languages. Furthermore, PEP 566 already specifies a canonicalization of email formatted core metadata to JSON. JSON is also a frequently used format for data interchange on the web. For discussion of other formats considered, please refer to the rejected ideas section.

To maintain backwards compatibility, the JSON metadata file MUST be generated alongside the existing email formatted metadata file. This ensures that tools that do not support the new format can still read package metadata for new packages.

The JSON formatted metadata file must be semantically equivalent to the email encoded file. This ensures that the metadata is unambiguous between the two formats, and tools may read either when both are present. To maintain performance, this equivalence is not required to be verified by installers, though other tools may do so. Some tools may choose to make the check dependent on a configuration flag.

Package indexes SHOULD check that the metadata files are semantically equivalent when the package is added to the index. This is a low-cost, one-time check that ensures users of the index are served valid packages.

Specification

Backwards Compatibility

The specification for METADATA.json and WHEEL.json is designed such that the new format is completely backwards compatible. Existing tools may read metadata from the existing email formatted files, and new tools may take advantage of the new format.

A future major revision of the wheel specification may make the METADATA, PKG-INFO, and WHEEL files optional and make the METADATA.json and WHEEL.json files required.

Note that tools will need to maintain parsing of email metadata and the key-value formatted WHEEL file indefinitely to support parsing metadata for old packages which only have the METADATA, PKG-INFO, or WHEEL files.

Security Implications

One attack vector with JSON encoded core metadata is if the JSON payload is designed to consume excessive memory or CPU resources in a denial of service (DoS) attack. While this attack is not likely to affect users whom can cancel resource-intensive interactive operations, it may be an issue for package indexes.

There are several mitigations that can be made to prevent this:

1. The length of the JSON payload can be restricted to a reasonable size.
2. The reader may use a JSONDecoder to omit parsing int and float values to avoid quadratic number parsing time complexity attacks.
3. I plan to contribute a change to JSONDecoder in Python 3.15+ that will allow it to be configured to restrict the nesting of JSON payloads to a reasonable depth. Core metadata currently has a maximum depth of 2 to encode mapping and list fields.

With these mitigations in place, concerns about denial of service attacks with JSON encoded core metadata are minimal.

Reference Implementation

A reference implementation of the JSON schema for JSON core metadata is available in Appendix: JSON Schema for Core Metadata.

Furthermore, a reference implementation in the packaging library is available.

A reference implementation generating both METADATA.json and WHEEL.json in the uv build backend is also available.

Rejected Ideas

Open Issues

Acknowledgements

Thanks to Konstantin Schütze for implementing the reference implementation of this PEP in the uv build backend and for providing valuable feedback on the specification.

Copyright

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.