Abstract

This PEP proposes extending the core metadata specification for Python packaging to include two new, repeatable fields named Import-Name and Import-Namespace to record the import names that a project provides once installed. New keys named import-names and import-namespaces will be added to the [project] table in pyproject.toml for providing the values for the new core metadata field. This also leads to the introduction of core metadata version 2.5.

Motivation

In Python packaging there is no requirement that a project name match the name(s) that you can import for that project. As such, there is no clean, easy, accurate way to go from import name to project name and vice-versa. This can make it difficult for tools that try to help people in discovering the right project to install when they know the import name or knowing what import names a project will provide once installed.

As an example, a code editor may detect a user has an unsatisfied import in a selected virtual environment. But with no way to reliably know what import names various projects provide, the code editor cannot accurately provide a user with a list of potential projects to install to satisfy that import requirement (e.g. it is not obvious that import PIL very likely implies the user wants the Pillow project installed). This also applies to when a user vaguely remembers the project name but does not remember the import name(s) and would have their memory jogged when seeing a list of import names a package provides. Finally, tools would be able to notify users what import names will become available once they install a project.

There is also no easy way to know whether installing two projects will conflict with one another based on the import names they provide. For instance, if two different projects have a _utils module, installing both project will lead to a clash as one project’s _utils module would take precedence over the other project’s version. This issue has been seen in the wild.

It may also help with spam detection. If a project specifies the same import names as a very popular project it can act as a signal to take a closer look at the validity of the less popular project. A project found to be lying about what import names it provides would be another signal.

Rationale

This PEP proposes extending the packaging Core metadata specifications so that project owners can specify the highest-level import names that a project provides if installed on some platform.

Putting this metadata in the core metadata means the data is (potentially) served by an index server, independent of any sdist or wheel. That negates needing to come up with a way to expose the metadata to tools to avoid having to download an entire e.g. wheel.

Having this metadata be the same across all release artifacts would allow for projects to only have to check a single file’s core metadata to get all possible import names instead of checking all the released files. This also means one does not need to worry if a file is missing when reading the core metadata or one can work solely from an sdist if the metadata is provided. As well, it simplifies having project.import-names and project.import-namespaces keys in pyproject.toml by having it be consistent for the entire project version and not unique per released file for the same version.

This PEP is not overly strict on what to (not) list in the proposed metadata on purpose. Having build back-ends verify that a project is accurately following a specification that is somehow strict about what can be listed would be near impossible to get right due to how flexible Python’s import system is. As such, this PEP only requires that valid import names be used and that projects don’t lie (and it is acknowledged the latter requirements cannot be validated programmatically). Project do, though, need to account for all levels of the names they list (e.g. you can’t list a.b.c and not account for a and a.b).

Various other attempts have been made to solve this, but they all have to make various trade-offs. For instance, one could download every wheel for every project release and look at what files are provided via the Binary distribution format, but that’s a lot of CPU and bandwidth for something that is static information (although tricks can be used to lessen the data requests such as using HTTP range requests to only read the table of contents of the zip file). This sort of calculation is also currently repeated by everyone independently instead of having the metadata hosted by a central index server like PyPI. It also doesn’t work for sdists as the structure of the wheel isn’t known yet, and so inferring the structure of the code installed isn’t known yet. As well, these solutions are not necessarily accurate as it is based on inference instead of being explicitly provided by the project owners. All of these accuracy issues affect even having an index hosting the information to avoid the compute costs of gathering the information.

Specification

Because this PEP introduces a new field to the core metadata, it bumps the latest core metadata version to 2.5.

The Import-Name and Import-Namespace fields are “multiple uses” fields. Each entry of both fields MUST be a valid import name. The names specified MUST be importable when the project is installed on some platform for the same version of the project (e.g. the metadata MUST be consistent across all sdists and wheels for a project release). This does imply that the information isn’t specific to the distribution artifact it is found in, but for the release version the distribution artifact belongs to.

Import-Name lists import names which a project, when installed, would exclusively provide (i.e. if two projects were installed with the same import names listed in Import-Name, then one of the projects would shadow the name for the other). Import-Namespace lists import names that, when installed, would be provided by the project, but not exclusively (i.e. projects all listing the same import name in Import-Namespace being installed together would not shadow those shared names).

The pyproject.toml specification will gain an import-names key. It will be an array of strings that stores what will be written out to Import-Name. Build back-ends MAY support dynamically calculating the value on the user’s behalf if desired, if the user declares the key in project.dynamic. The same applies to import-namespaces for Import-Namespace.

Projects SHOULD list all the shortest import names that are exclusively provided by a project which would cover all import name scenarios. If any of the shortest names are dotted names, all intervening names from that name to the top-level name should also be listed appropriately in Import-Namespace. For instance, a project which is a single package named spam with multiple submodules would only list project.import-names = ["spam"]. A project that provides spam.bacon.eggs which is exclusively from the project while the intervening names are namespaces would have project-names = ["spam.bacon.eggs"] and project-namespaces = ["spam", "spam.bacon"]. Listing all names acts as a check that the intent of the import names is as expected.

Tools SHOULD raise an error when two projects that are to be installed list names that overlap in each others’ Import-Name entries. This is to avoid projects unexpectedly shadowing another project’s code. The same applies to when a project has an entry in Import-Name that overlaps with another project’s Import-Namespace entries.

Tools SHOULD raise an error when an entry in Import-Name is higher than Import-Namespace in the same project, e.g. project.import-names = ["spam"] and project.import-namespaces = ["spam.bacon"]. This is because if a project exclusively owns a higher import name then that would mean it is impossible for another project to install with the same import name found in Import-Name in order to contribute to the namespace listed in Import-Namespace.

Projects MAY leave Import-Name and Import-Namespace empty. In that instance, tools SHOULD assume that the normalized project name when converted to an import name would be an entry in Import-Name (i.e. - substituted for - in the normalized project name).

[project]
import-names = ["sklearn"]

[project]
import-names = ["_pytest", "py", "pytest"]

[project]
import-names = ["azure.mgmt.search"]
import-namespaces = ["azure", "azure.mgmt"]

Backwards Compatibility

As this is a new field for the core metadata and a new core metadata version, there should be no backwards compatibility concerns.

Security Implications

Tools should treat the metadata as potentially inaccurate. As such, any decisions made based on the provided metadata should be assumed to be malicious in some way.

How to Teach This

Project owners should be taught that they can now record what names their projects provide for importing. If their project name matches the module or package name their project provides they don’t have to do anything. If there is a difference, though, they should record all the import names their project provides, using the shortest names possible. If any of the names are implicit namespaces, those go into project.import-namespaces in pyproject.toml, otherwise the name goes into project.import-names.

Users of projects don’t necessarily need to know about this new metadata. While they may be exposed to it via tooling, the details of where that data came from isn’t critical. It’s possible they may come across it if an index server exposed it (e.g., listed the values from Import-Name and marked whether the file structure backed up the claims the metadata makes), but that still wouldn’t require users to know the technical details of this PEP.

Reference Implementation

https://github.com/brettcannon/packaging/tree/pep-794 is a branch to update ‘packaging’ to support this PEP.

Rejected Ideas

Open Issues

N/A

Acknowledgments

Thanks to HeeJae Chang for ~~complaining about~~ bringing up regularly the usefulness that this metadata would provide. Thanks to Josh Cannon (no relation) for reviewing drafts of this PEP and providing feedback. Also, thanks to everyone who participated in a previous discussion on this topic.

Copyright

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.