PEP 627 – Recording installed projects
- Petr Viktorin <encukou at gmail.com>
- Paul Moore <p.f.moore at gmail.com>
- Discourse thread
- Standards Track
- Discourse post
Table of Contents
- Rationale Change
- Standard and Changes Process
- Changes and their Rationale
- Deferred Ideas
This PEP clarifies and updates PEP 376 (Database of Installed Python Distributions), rewriting it as an interoperability standard. It moves the canonical location of the standard to the Python Packaging Authority (PyPA) standards repository, and sets up guidelines for changing it.
Two files in installed
.dist-info directories are made optional:
RECORD (which PEP 376 lists as mandatory, but suggests it can be left out
for “system packages”), and
Python packaging is moving from relying on specific tools (Setuptools and pip) toward an ecosystem of tools and tool-agnostic interoperability standards.
PEP 376 is not written as an interoperability standard. It describes implementation details of specific tools and libraries, and is underspecified, leaving much room for implementation-defined behavior.
This is a proposal to “distill” the standard from PEP 376, clarify it, and rewrite it to be tool-agnostic.
The aim of this PEP is to have a better standard, not necessarily a perfect one. Some issues are left to later clarification.
PEP 376’s rationale focuses on two problems:
- There are too many ways to install projects and this makes interoperation difficult.
- There is no API to get information on installed distributions.
The new document focuses only the on-disk format of information about installed projects. Providing API to install, uninstall or query this information is left to be implemented by tools.
Standard and Changes Process
The canonical standard for Recording installed projects (previously known as
Database of Installed Python Distributions) is the documentation at
Any changes to the document (except trivial language or typography fixes) must
be made through the PEP process.
The document is normative (with examples to aid understanding). PEPs that change it, such as this one, contain additional information that is expected to get out of date, such as rationales and compatibility considerations.
The proposed standard is submitted together with this PEP as a pull request to
Changes and their Rationale
Renaming to “Recording installed projects”
The standard is renamed from Database of Installed Python Distributions to Recording installed projects.
While putting files in known locations on disk may be thought of as a “database”, it’s not what most people think about when they hear the term. The PyPA links to PEP 376 under the heading Recording installed distributions.
The PyPA glossary defines “Distribution” (or, “Distribution Package” to prevent confusion with e.g. Linux distributions) as “A versioned archive file […]”. Since there may be other ways to install Python code than from archive files, the document uses “installed project” rather than “installed distribution”.
Removal of Implementation Details
All tool- and library-specific details are removed.
The mechanisms of how a project is installed are also left out: the document
focuses on the end state.
One exception is a sketch of an uninstallation algorithm,
which is given to better explain the purpose of the
formats specific to
are left out.
Explicitly Allowing Additional Files
.dist-info directory is allowed to contain files not specified in
The current tools already do this.
A note in the specification mentions files in the
.dist-info directory of wheels.
Current tools copy these files to the installed
to keep in mind for further standardization efforts.
Clarifications in the
The CSV dialect is specified to be the default of Python’s
This resolves edge cases around handling double-quotes and line terminators
in file names.
The “base” of relative paths in
RECORD is specified relative to the
.dist-info directory, rather than tool-specific
Both hash and size fields are now optional (for any file, not just
RECORD). Leavng them out is discouraged,
(Note that PEP 376 is unclear on what was optional; when taken literally,
its text and examples contradict. Despite that, “both fields are optional“ is a
reasonable interpretation of PEP 376.
The alternative would be to mandate—rather than recommend—which files can be
recorded without hash and size, and to update that list over time as new use
cases come up.)
The new spec explicitly says that the
RECORD file must now include all
files of the installed project (the exception for
.pyc files remains).
Since tools use
RECORD for uninstallation, incomplete file lists could
introduce orphaned files to users’ environments.
On the other hand, this means that there is no way to record hashes of some
any files if the full list of files is unknown.
A sketch of an uninstallation algorithm is included to clarify the file’s primary purpose and contents.
Tools must not uninstall/remove projects that lack a
(unless they have external information, such as in system package
managers of Linux distros).
On Windows, files in
RECORD may be separated by either
PEP 376 was unclear on this: it mandates forward slashes in one place, but
shows backslackes in a Windows-specific example.
RECORD file is made optional.
Not all tools can easily generate a list of installed files in a
RECORD file is unnecessary when projects are installed
by a Linux system packaging system, which has its own ways to keep track of
files, uninstall them or check their integrity.
Having to keep a
RECORD file in sync with the disk and the system package
database would be unreasonably fragile, and no
RECORD file is better
than one that does not correspond to reality.
(Full disclosure: The author of this PEP is an RPM packager active in the Fedora Linux distro.)
INSTALLER file is also made optional, and specified to be used for
informational purposes only.
It is still a single-line text file containing the name of the installer.
This file was originally added to distinguish projects installed by the Python
pip) from ones installed by other package managers
There were attempts to use this file to prevent
pip from updating or
uninstalling packages it didn’t install.
Our goal is supporting interoperating tools, and basing any action on which tool happened to install a package runs counter to that goal.
Instead of relying on the installer name, tools should use feature detection.
The current document offers a crude way of making a project untouchable by
Python tooling: omitting
On the other hand, the installer name may be useful in hints to the user.
To align with this new purpose of the file, the new specification allows
any ASCII string in
INSTALLER, rather than a lowercase identifier.
It also suggests using the command-line command, if available.
REQUESTED File: Removed from Spec
REQUESTED file is now considered a tool-specific extension.
Per PEP 376,
REQUESTED was to be written when a project was installed
by direct user request, as opposed to automatically to satisfy dependencies
of another project. Projects without this marker file could be uninstalled
when no longer needed.
Despite the standard, many existing installers (including older versions of
pip) never write this file. There is no distinction between projects
that are “OK to remove when no longer needed” and ones simply installed by
a tool that ignores
REQUESTED. So, the file is currently not usable for its
intended purpose (unless a tool can use additional, non-standard information).
When possible, terms (such as
version) are qualified by
references to existing specs.
To limit the scope of this PEP, some improvements are explicitly left to future PEPs:
- Encoding of the
- Limiting or namespacing files that can appear in
- Marking the difference between projects installed directly by user request versus those installed to satisfy dependencies, so that the latter can be removed when no longer needed.
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
Last modified: 2022-06-21 21:47:58 GMT