Terminology

This PEP uses the following terminology:

• build machine: the machine on which the package build process is being executed
• host machine: the machine on which the produced artifact will be installed and run
• build dependency: dependency for building the package that needs to be present at build time and itself was built for the build machine’s OS and architecture
• host dependency: dependency for building the package that needs to be present at build time and itself was built for the host machine’s OS and architecture

Note that this terminology is not consistent across build and packaging tools, so care must be taken when comparing build/host dependencies in pyproject.toml to dependencies from other package managers.

Note that “target machine” or “target dependency” is not used in this PEP. That is typically only relevant for cross-compiling compilers or other such advanced scenarios [3], [4] - this is out of scope for this PEP.

Finally, note that while “dependency” is the term most widely used for packages needed at build time, the existing key in pyproject.toml for PyPI build-time dependencies is build-requires. Hence this PEP uses the keys build-requires and host-requires under [external] for consistency.

Build and host dependencies

Clear separation of metadata associated with the definition of build and target platforms, rather than assuming that build and target platform will always be the same, is important [5].

Build dependencies are typically run during the build process - they may be compilers, code generators, or other such tools. In case the use of a build dependency implies a runtime dependency, that runtime dependency does not have to be declared explicitly. For example, when compiling Fortran code with gfortran into a Python extension module, the package likely incurs a dependency on the libgfortran runtime library. The rationale for not explicitly listing such runtime dependencies is two-fold: (1) it may depend on compiler/linker flags or details of the build environment whether the dependency is present, and (2) these runtime dependencies can be detected and handled automatically by tools like auditwheel.

Host dependencies are typically not run during the build process, but only used for linking against. This is not a rule though – it may be possible or necessary to run a host dependency under an emulator, or through a custom tool like crossenv. When host dependencies imply a runtime dependency, that runtime dependency also does not have to be declared, just like for build dependencies.

When host dependencies are declared and a tool is not cross-compilation aware and has to do something with external dependencies, the tool MAY merge the host-requires list into build-requires. This may for example happen if an installer like pip starts reporting external dependencies as a likely cause of a build failure when a package fails to build from an sdist.

Concrete package specification through PURL

The two types of concrete packages are supported by PURL (Package URL), which implements a scheme for identifying packages that is meant to be portable across packaging ecosystems. Its design is:

scheme:type/namespace/name@version?qualifiers#subpath

The scheme component is a fixed string, pkg, and of the other components only type and name are required. As an example, a package URL for the requests package on PyPI would be:

pkg:pypi/requests

Adopting PURL to specify external dependencies in pyproject.toml solves a number of problems at once - and there are already implementations of the specification in Python and multiple languages. PURL is also already supported by dependency-related tooling like SPDX (see External Repository Identifiers in the SPDX 2.3 spec), the Open Source Vulnerability format, and the Sonatype OSS Index; not having to wait years before support in such tooling arrives is valuable.

For concrete packages without a canonical package manager to refer to, either pkg:generic/pkg-name can be used, or a direct reference to the VCS system that the package is maintained in (e.g., pkg:github/user-or-org-name/pkg-name). Which of these is more appropriate is situation-dependent. This PEP recommends using pkg:generic when the package name is unambiguous and well-known (e.g., pkg:generic/git or pkg:generic/openblas), and using the VCS as the PURL type otherwise.

Virtual package specification

There is no ready-made support for virtual packages in PURL or another standard. There are a relatively limited number of such dependencies though, and adopting a scheme similar to PURL but with the virtual: rather than pkg: scheme seems like it will be understandable and map well to Linux distros with virtual packages and to the likes of Conda and Spack.

The two known virtual package types are compiler and interface.

Versioning

Support in PURL for version expressions and ranges beyond a fixed version is still pending, see the Open Issues section.

Dependency specifiers

Regular Python dependency specifiers (as originally defined in PEP 508) may be used behind PURLs. PURL qualifiers, which use ? followed by a package type-specific dependency specifier component, must not be used. The reason for this is pragmatic: dependency specifiers are already used for other metadata in pyproject.toml, any tooling that is used with pyproject.toml is likely to already have a robust implementation to parse it. And we do not expect to need the extra possibilities that PURL qualifiers provide (e.g. to specify a Conan or Conda channel, or a RubyGems platform).

Differences between sdist and wheel metadata

A wheel may vendor its external dependencies. This happens in particular when distributing wheels on PyPI or other Python package indexes - and tools like auditwheel, delvewheel and delocate automate this process. As a result, a Requires-External entry in an sdist may disappear from a wheel built from that sdist. It is also possible that a Requires-External entry remains in a wheel, either unchanged or with narrower constraints. auditwheel does not vendor certain allow-listed dependencies, such as OpenGL, by default. In addition, auditwheel and delvewheel allow a user to manually exclude dependencies via a --exclude or --no-dll command-line flag. This is used to avoid vendoring large shared libraries, for example those from CUDA.

Requires-External entries generated from external dependencies in pyproject.toml in a wheel are therefore allowed to be narrower than those for the corresponding sdist. They must not be wider, i.e. constraints must not allow a version of a dependency for a wheel that isn’t allowed for an sdist, nor contain new dependencies that are not listed in the sdist’s metadata at all.

Canonical names of dependencies and -dev(el) split packages

It is fairly common for distros to split a package into two or more packages. In particular, runtime components are often separately installable from development components (headers, pkg-config and CMake files, etc.). The latter then typically has a name with -dev or -devel appended to the project/library name. This split is the responsibility of each distro to maintain, and should not be reflected in the [external] table. It is not possible to specify this in a reasonable way that works across distros, hence only the canonical name should be used in [external].

The intended meaning of using a PURL or virtual dependency is “the full package with the name specified”. It will depend on the context in which the metadata is used whether the split is relevant. For example, if libffi is a host dependency and a tool wants to prepare an environment for building a wheel, then if a distro has split off the headers for libffi into a libffi-devel package then the tool has to install both libffi and libffi-devel.

Python development headers

Python headers and other build support files may also be split. This is the same situation as in the section above (because Python is simply a regular package in distros). However, a python-dev|devel dependency is special because in pyproject.toml Python itself is an implicit rather than an explicit dependency. Hence a choice needs to be made here - add python-dev implicitly, or make each package author add it explicitly under [external]. For consistency between Python dependencies and external dependencies, we choose to add it implicitly. Python development headers must be assumed to be necessary when an [external] table contains one or more compiler packages.

Table name

Tools MUST specify fields defined by this PEP in a table named [external]. No tools may add fields to this table which are not defined by this PEP or subsequent PEPs. The lack of an [external] table means the package either does not have any external dependencies, or the ones it does have are assumed to be present on the system already.

build-requires/optional-build-requires

• Format: Array of PURL strings (build-requires) and a table with values of arrays of PURL strings (optional-build-requires)
• Core metadata: N/A

The (optional) external build requirements needed to build the project.

For build-requires, it is a key whose value is an array of strings. Each string represents a build requirement of the project and MUST be formatted as either a valid PURL string or a virtual: string.

For optional-build-requires, it is a table where each key specifies an extra set of build requirements and whose value is an array of strings. The strings of the arrays MUST be valid PURL strings.

host-requires/optional-host-requires

• Format: Array of PURL strings (host-requires) and a table with values of arrays of PURL strings (optional-host-requires)
• Core metadata: N/A

The (optional) external host requirements needed to build the project.

For host-requires, it is a key whose value is an array of strings. Each string represents a host requirement of the project and MUST be formatted as either a valid PURL string or a virtual: string.

For optional-host-requires, it is a table where each key specifies an extra set of host requirements and whose value is an array of strings. The strings of the arrays MUST be valid PURL strings.

dependencies/optional-dependencies

• Format: Array of PURL strings (dependencies) and a table with values of arrays of PURL strings (optional-dependencies)
• Core metadata: Requires-External, N/A

The (optional) runtime dependencies of the project.

For dependencies, it is a key whose value is an array of strings. Each string represents a dependency of the project and MUST be formatted as either a valid PURL string or a virtual: string. Each string maps directly to a Requires-External entry in the core metadata.

For optional-dependencies, it is a table where each key specifies an extra and whose value is an array of strings. The strings of the arrays MUST be valid PURL strings. Optional dependencies do not map to a core metadata field.