Abstract

This PEP proposes extending the core metadata specification for Python packaging to include two new, repeatable fields named Import-Name and Import-Namespace to record the import names that a project provides once installed. New keys named import-names and import-namespaces will be added to the [project] table in pyproject.toml for providing the values for the new core metadata fields. This also leads to the introduction of core metadata version 2.5.

Motivation

In Python packaging there is no requirement that a project name match the name(s) that you can import for that project. As such, there is no clean, easy, accurate way to go from import name to project name and vice versa. This can make it difficult for tools that try to help people in discovering the right project to install when they know the import name or knowing what import names a project will provide once installed.

As an example, a code editor may detect a user has an unsatisfied import in a selected virtual environment. But with no way to reliably know what import names various projects provide, the code editor cannot accurately provide a user with a list of potential projects to install to satisfy that import requirement (e.g. it is not obvious that import PIL very likely implies the user wants the Pillow project installed). This also applies to when a user vaguely remembers the project name but does not remember the import name(s) and would have their memory jogged when seeing a list of import names a package provides. Finally, tools would be able to notify users what import names will become available once they install a project.

There is also no easy way to know whether installing two projects will conflict with one another based on the import names they provide. For instance, if two different projects have a _utils module, installing both projects will lead to a clash as one project’s _utils module would take precedence over the other’s, by overwriting the other project’s file; this issue has been seen in the wild.

It may also help with spam detection. If a project specifies the same import names as a very popular project it can act as a signal to take a closer look at the validity of the less popular project. A project found to be lying about what import names it provides would be another signal.

Rationale

This PEP proposes extending the packaging Core metadata specifications so that project owners can specify the highest-level import names that a project provides if installed on some platform.

Putting this metadata in the core metadata means the data is (potentially) served by an index server, independent of any sdist or wheel. That negates needing to come up with a way to expose the metadata to tools to avoid having to download an entire e.g. wheel.

Having this metadata be the same across all release artifacts would allow for projects to only have to check a single file’s core metadata to get all possible import names instead of checking all the released files. This also means one does not need to worry if a file is missing when reading the core metadata or one can work solely from an sdist if the metadata is provided. As well, it simplifies having project.import-names and project.import-namespaces keys in pyproject.toml by having it be consistent for the entire project version and not unique per released file for the same version.

A distribution file containing modules and packages can have any combination of public and private APIs at the module/package level. Distribution files can also contain no modules or packages of any kind. Being able to distinguish between the situations all have various tool uses that could be beneficial to users. For instance, knowing all import names regardless of whether they are public or private helps detect clashes at install time. But knowing what is explicitly public or private allows tools such as editors to not suggest private import names as part of auto-complete.

This PEP is not overly strict on what to (not) list in the proposed metadata on purpose. Having build back-ends verify that a project is accurately following a specification that is somehow strict about what can be listed would be nearly impossible to get right due to how flexible Python’s import system is. As such, this PEP only requires that valid import names be used and that projects don’t lie (and it is acknowledged the latter requirement cannot be validated programmatically). Projects do, though, need to account for all levels of the names they list (e.g. you can’t list a.b.c and not account for a and a.b).

Various other attempts have been made to solve this, but they all have to make trade-offs. For instance, one could download every wheel for every project release and look at what files are provided via the Binary distribution format, but that’s a lot of CPU and bandwidth for something that is static information (although tricks can be used to lessen the data requests such as using HTTP range requests to only read the table of contents of the zip file). This sort of calculation is also currently repeated by everyone independently instead of having the metadata hosted by a central index server like PyPI. It also doesn’t work for sdists as the structure of the wheel isn’t known yet, and so inferring the structure of the code installed isn’t possible. As well, these solutions are not necessarily accurate as they are based on inference instead of being explicitly provided by the project owners. All of these accuracy issues affect even having an index host the information to avoid the compute costs of gathering it.

Specification

Because this PEP introduces a new field to the core metadata, it bumps the latest core metadata version to 2.5.

The Import-Name and Import-Namespace fields are “multiple uses” fields. Each entry of both fields MUST be a valid import name or can be empty in the case of Import-Name. Any names specified MUST be importable when the project is installed on some platform for the same version of the project (e.g. the metadata MUST be consistent across all sdists and wheels for a project release). This does imply that the information isn’t specific to the distribution artifact it is found in, but to the release version the distribution artifact belongs to.

An import name MAY be followed by a semicolon and the term “private” (e.g. ; private). This signals to tools that the import name is not part of the public API for the project. Any number of spaces surrounding the ; is allowed.

Import-Name lists import names which a project, when installed, would exclusively provide (i.e. if two projects were installed with the same import names listed in Import-Name, then one of the projects would shadow the name for the other). Import-Namespace lists import names that, when installed, would be provided by the project, but not exclusively (i.e. projects all listing the same import name in Import-Namespace being installed together would not shadow those shared names).

The pyproject.toml specification will gain an import-names key. It will be an array of strings that stores what will be written out to Import-Name. Build back-ends MAY support dynamically calculating the value on the user’s behalf if desired, if the user declares the key in project.dynamic. The same applies to import-namespaces for Import-Namespace.

Projects SHOULD list all the shortest import names that are exclusively provided by a project which would cover all import name scenarios. If any of the shortest names are dotted names, all intervening names from that name to the top-level name should also be listed appropriately in Import-Namespace and/or Import-Name. For instance, a project which is a single package named spam with multiple submodules would only list project.import-names = ["spam"]. A project that lists spam.bacon.eggs would also need to account for spam and spam.bacon appropriately in import-names and import-namespaces. Listing all names acts as a check that the intent of the import names is as expected. As well, projects SHOULD list all import names, public or private, using the ; private modifier as appropriate.

If a project lists the same name in both Import-Name and Import-Namespace, then tools MUST raise an error due to ambiguity; this also applies to import-names and import-namespaces, respectively.

Tools SHOULD raise an error when two projects that are about to be installed by a tool list names that overlap in each other’s Import-Name entries (i.e. installed in the same command/action). This is to avoid projects unexpectedly shadowing another project’s code. The same applies to when a project has an entry in Import-Name that overlaps with another project’s Import-Namespace entries. This does not apply to overlapping Import-Namespace entries as that’s the purpose of namespace packages. Tools MAY warn or raise an error when installing a project into a preexisting environment where there is import name overlap with a project that is already installed. This is a “MAY” and not a “SHOULD” due to some users purposefully overwriting import names when installation is done in multiple steps (e.g. using different installers with the same environment).

Tools MAY leave Import-Name and Import-Namespace out of the core metadata for a project. In that instance, tools consuming such metadata SHOULD assume that when the core metadata is 2.5 or newer, the normalized project name, when converted to an import name, would be an entry in Import-Name (i.e. - replaced with _ in the normalized project name). This is deemed reasonable as this will only occur for projects that make a new release once their build back-end supports core metadata 2.5 or newer as proposed by this PEP.

Projects MAY set import-names or import-namespaces – as well as Import-Name or Import-Namespace, respectively – to the normalized import name of the project to explicitly declare that the project’s name is also the import name.

Projects MAY set import-names an empty array and not set import-namespaces at all in a pyproject.toml file (e.g. import-names = []). To match this, projects MAY have an empty Import-Name field in their metadata. This represents a project with NO import names, public or private (i.e. there are no Python modules of any kind in the distribution file).

[project]
import-names = ["sklearn"]

[project]
# The pytest docs list code out of all of these modules, so it isn't
# obvious whether they would mark any as private.
import-names = ["_pytest", "py", "pytest"]

[project]
import-names = ["azure.mgmt.search"]
import-namespaces = ["azure", "azure.mgmt"]

Backwards Compatibility

As this is a new field for the core metadata and a new core metadata version, there should be no backwards compatibility concerns.

Security Implications

Tools should treat the metadata as potentially inaccurate. As such, any decisions made based on the provided metadata should be assumed to be malicious in some way.

How to Teach This

Project owners should be taught that they can now record what names their projects provide for importing. If their project name matches the module or package name their project provides they don’t have to do anything. If there is a difference, though, they should record all the import names their project provides, using the shortest names possible. If any of the names are implicit namespaces, those go into project.import-namespaces in pyproject.toml, otherwise the name goes into project.import-names.

Users of projects don’t necessarily need to know about this new metadata. While they may be exposed to it via tooling, the details of where that data came from isn’t critical. It’s possible they may come across it if an index server exposes it (e.g., lists the values from Import-Name and marks whether the file structure backs up the claims the metadata makes), but that still wouldn’t require users to know the technical details of this PEP.

Reference Implementation

https://github.com/brettcannon/packaging/tree/pep-794 is a branch to update ‘packaging’ to support this PEP.

Rejected Ideas

Open Issues

N/A

Acknowledgments

Thanks to HeeJae Chang for ~~complaining about~~ regularly bringing up the usefulness that this metadata would provide. Thanks to Josh Cannon (no relation) for reviewing drafts of this PEP and providing feedback. Also, thanks to everyone who participated in a previous discussion on this topic.

Copyright

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.