Terminology

This PEP uses the following terminology:

• build machine: the machine on which the package build process is being executed.
• host machine: the machine on which the produced artifact will be installed and run.
• build dependency: package required only during the build process. It must be available at build time and is built for the build machine’s OS and architecture. Typical examples include compilers, code generators, and build tools.
• host dependency: package needed during the build and often also at runtime. It must be available during the build and is built for the host machine’s OS and architecture. These are usually libraries the project links against.
• runtime dependency: package required only when the package is used after installation. It is not required at build time but must be available on the host machine at runtime.

Note that this terminology is not consistent across build and packaging tools, so care must be taken when comparing build/host dependencies in pyproject.toml to dependencies from other package managers.

Note that “target machine” or “target dependency” are not used in this PEP. That is typically only relevant for cross-compiling a compiler or other such advanced scenarios [3], [4] – this is out of scope for this PEP.

Finally, note that while “dependency” is the term most widely used for packages needed at build time, the existing key in pyproject.toml for PyPI build-time dependencies is build-requires. Hence this PEP uses the keys build-requires and host-requires under [external] for consistency.

Build and host dependencies

Clear separation of metadata associated with the definition of build and host platforms, rather than assuming that build and host platform will always be the same, is important [5].

Build dependencies are typically run during the build process - they may be compilers, code generators, or other such tools. In case the use of a build dependency implies a runtime dependency, that runtime dependency does not have to be declared explicitly. For example, when compiling Fortran code with gfortran into a Python extension module, the package likely incurs a dependency on the libgfortran runtime library. The rationale for not explicitly listing such runtime dependencies is two-fold: (1) it may depend on compiler/linker flags or details of the build environment whether the dependency is present, and (2) these runtime dependencies can be detected and handled automatically by tools like auditwheel.

Host dependencies are typically not run during the build process, but only used for linking against. This is not a rule though – it may be possible or necessary to run a host dependency under an emulator, or through a custom tool like crossenv. When host dependencies imply a runtime dependency, that runtime dependency also does not have to be declared, just like for build dependencies.

When host dependencies are declared and a tool which is executing an action unrelated to cross-compiling, it may decide to merge the host-requires list into build-requires - whether this is useful is context-dependent.

Concrete package specification

A “Package URL” or PURL is a widely used URL string for identifying packages that is meant to be portable across packaging ecosystems. Its design is:

scheme:type/namespace/name@version?qualifiers#subpath

The scheme component is a fixed string, pkg, and of the other components only type and name are required.

Since external dependencies are likely to be typed by hand, we propose a PURL derivative that, in the name of ergonomics and user-friendliness, introduces a number of changes (further discussed below):

• Support for virtual packages via a new virtual type.
• Allow version ranges (and not just literals) in the version field.

In this derivative, we replace the pkg scheme with dep. Hence, we will refer to them as DepURLs.

As an example, a DepURL for the requests package on PyPI would be:

dep:pypi/requests
# equivalent to pkg:pypi/requests

Adopting PURL-compatible strings to specify external dependencies in pyproject.toml solves a number of problems at once, and there are already implementations of the specification in Python and multiple other languages. PURL is also already supported by dependency-related tooling like SPDX (see External Repository Identifiers in the SPDX 2.3 spec), the Open Source Vulnerability format, and the Sonatype OSS Index; not having to wait years before support in such tooling arrives is valuable. DepURLs are very easily transformed into PURLs, with the exception of dep:virtual which doesn’t have an equivalent in PURL.

For concrete packages without a canonical package manager to refer to, either dep:generic/dep-name can be used, or a direct reference to the VCS system that the package is maintained in (e.g., dep:github/user-or-org-name/dep-name). Which of these is more appropriate is situation-dependent. This PEP recommends using dep:generic when the package name is unambiguous and well-known (e.g., dep:generic/git or dep:generic/openblas), and using the VCS as the type otherwise. Which name is chosen as canonical for any given package, as well as the process to make and record such choices, is the topic of PEP 804.

Virtual package specification

PURL does not offer support for virtual or virtual dependency specification yet. A proposal to add a virtual type is being discussed for revision 1.1.

In the meantime, we propose adding a new type to our dep: derivative, the virtual type, which can take two namespaces (extensible through the process given in PEP 804):

• interface: for components such as BLAS or MPI.
• compiler: for compiled languages like C or Rust.

The name should be the most common name for the interface or language, lowercased. Some examples include:

dep:virtual/compiler/c
dep:virtual/compiler/cxx
dep:virtual/compiler/rust
dep:virtual/interface/blas
dep:virtual/interface/lapack

Since there are a limited number of such dependencies, it seems like it will be understandable and map well to Linux distros with virtual packages and to the likes of conda and Spack.

Versioning

PURLs support fixed versions via the @ component of the URL. For example, numpy===2.0 can be expressed as pkg:pypi/numpy@2.0.

Support in PURL for version expressions and ranges beyond a fixed version is available via vers URIs (see specification):

vers:type/version-constraint|version-constraint|...

Users are supposed to couple a pkg: URL with a vers: URL. For example, to express numpy>=2.0, the PURL equivalent would be pkg:pypi/numpy plus vers:pypi/>=2.0. This can be done with:

• A two-item list: ["pkg:pypi/numpy", "vers:pypi/>=2.0"].
• A percent-encoded URL qualifier: pkg:pypi/numpy?vers=vers:pypi%2F%3E%3D2.0.

Since none of these options are very ergonomic, we chose instead for DepURLs to accept version range specifiers too with semantics that are a subset of PEP 440 semantics. The allowed operators are those that are widely available across package managers (e.g., ==, > and >= are common, while ~= isn’t).

Some examples:

• dep:pypi/numpy@2.0: numpy pinned at exactly version 2.0.
• dep:pypi/numpy@>=2.0: numpy with version greater or equal than 2.0.
• dep:virtual/interface/lapack@>=3.7.1: any package implementing the LAPACK interface for version greater or equal than 3.7.1.

The versioning scheme for particular virtual packages, in case that isn’t unambiguously defined by an upstream project or standard, will be defined in the Central Registry (see PEP 804).

Environment markers

Regular environment markers (as originally defined in PEP 508) may be used behind DepURLs. PURL qualifiers, which use ? followed by a package type-specific dependency specifier component, should not be used for the purposes for which environment markers suffice. The reason for this is pragmatic: environment markers are already used for other metadata in pyproject.toml, hence any tooling that is used with pyproject.toml is likely to already have a robust implementation to parse it. And we do not expect to need the extra possibilities that PURL qualifiers provide (e.g., to specify a Conan or conda channel, or a RubyGems platform).

We name the combination of a DepURL and environment markers as “external dependency specifiers”, analogously to the existing dependency specifiers.

Canonical names of dependencies and -dev(el) split packages

It is fairly common, but far from universal, for distros to split a package into two or more packages. In particular, runtime components are often separately installable from development components (headers, pkg-config and CMake files, etc.). The latter then typically has a name with -dev or -devel appended to the project/library name. Also, larger packages are sometimes split into multiple separate packages to keep install sizes manageable. More often than not, such package splits are not defined or recognized by the maintainers of a package, and it’s therefore ambiguous what any split would mean. Hence, such splits should not be reflected in the [external] table. It is not possible to specify this in a reasonable way that works across distros, hence only the canonical name should be used in [external].

The intended meaning of using a DepURL is “the full package with the name specified”. I.e., including all installable artifacts that are part of the package. It will depend on the context in which the metadata is used whether a package split is relevant. For example, if libffi is a host dependency and a tool wants to prepare an environment for building a wheel, then if a distro has split off the headers for libffi into a libffi-devel package then the tool has to install both libffi and libffi-devel.

For defining what canonical package names are and how package splits are handled in practice when tools attempt to use [external] for installation purposes, we refer to PEP 804.

Python development headers

Python headers and other build support files may also be split. This is the same situation as in the section above (because Python is simply a regular package in distros). However, a python-dev|devel dependency is special because in pyproject.toml Python itself is an implicit rather than an explicit dependency. Hence a choice needs to be made here - add python-dev implicitly, or make each package author add it explicitly under [external]. For consistency between Python dependencies and external dependencies, we choose to add it implicitly. Python development headers must be assumed to be necessary when an [external] table contains one or more compiler packages.

Effect of vendoring shared libraries on wheel metadata

A wheel may vendor its external dependencies. This happens in particular when distributing wheels on PyPI or other Python package indexes – and tools like auditwheel, delvewheel and delocate automate this process. As a result, a Requires-External-Dep entry in an sdist may disappear from a wheel built from that sdist with a tool like cibuildwheel. It is also possible that a Requires-External-Dep entry remains in a wheel, either unchanged or with narrower constraints. auditwheel does not vendor certain allow-listed dependencies, such as OpenGL, by default. In addition, auditwheel and delvewheel allow a user to manually exclude dependencies via a --exclude or --no-dll command-line flag. This is used to avoid vendoring large shared libraries, for example those from CUDA.

Requires-External-Dep entries generated from external dependencies in pyproject.toml can therefore differ between an sdist and its corresponding wheel(s) depending on the build/distribution process.

Note that this does not imply that the field must be marked as Dynamic, since this distinction only applies to wheels built from an sdist by a build backend. In particular, wheels built from other wheels do not need to satisfy this constraint.

Optional dependencies versus dependency groups

The rationale for having external.dependency-groups is identical for the rationale given in PEP 735 for introducing [dependency-groups]. The intended usage and semantics of inclusion/exclusion into Core Metadata is thus identical to [dependency-groups].

external.optional-dependencies will show up in Core Metadata. external.dependency-groups will not.

Deprecations

The External-Requires Core Metadata field will be marked as obsolete and its usage will be discouraged.

Additions

Two new fields are added to Core Metadata:

• Requires-External-Dep. An external requirement expressed as an external dependency specifier string.
• Provides-External-Extra. An extra group that carries external dependencies (as found in Requires-External-Dep) only.

Version bump

Given that the proposed changes are purely additive, the Core Metadata version will be bumped to 2.6.

This will only impact PyPI and tools that want to support external runtime dependencies, and require no changes otherwise.

Table name

Tools MUST specify fields defined by this PEP in a table named [external]. No tools may add fields to this table which are not defined by this PEP or subsequent PEPs. The lack of an [external] table means the package either does not have any external dependencies, or the ones it does have are assumed to be present on the system already.

build-requires/optional-build-requires

• Format: Array of external dependency specifiers (build-requires) and a table with values of arrays of external dependency specifiers (optional-build-requires)
• Core metadata: N/A

The (optional) external build requirements needed to build the project.

For build-requires, it is a key whose value is an array of strings. Each string represents a build requirement of the project and MUST be formatted as a valid external dependency specifier.

For optional-build-requires, it is a table where each key specifies an extra set of build requirements and whose value is an array of strings. The strings of the arrays MUST be valid external dependency specifiers.

host-requires/optional-host-requires

• Format: Array of external dependency specifiers (host-requires) and a table with values of arrays of external dependency specifiers (optional-host-requires) - Core metadata: N/A

The (optional) external host requirements needed to build the project.

For host-requires, it is a key whose value is an array of strings. Each string represents a host requirement of the project and MUST be formatted as a valid external dependency specifier.

For optional-host-requires, it is a table where each key specifies an extra set of host requirements and whose value is an array of strings. The strings of the arrays MUST be valid external dependency specifiers.

dependencies/optional-dependencies

• Format: Array of external dependency specifiers (dependencies) and a table with values of arrays of external dependency specifiers (optional-dependencies)
• Core metadata: Requires-External-Dep, Provides-External-Extra

The (optional) runtime dependencies of the project.

For dependencies, it is a key whose value is an array of strings. Each string represents a dependency of the project and MUST be formatted as a valid external dependency specifier. Each string must be added to Core Metadata as a Requires-External-Dep field.

For optional-dependencies, it is a table where each key specifies an extra and whose value is an array of strings. The strings of the arrays MUST be valid external dependency specifiers. For each optional-dependencies group:

• The name of the group MUST be added to Core Metadata as a Provides-External-Extra field.
• The external dependency specifiers in that group MUST be added to Core Metadata as a Requires-External-Dep field, with the corresponding ; extra == 'name' environment marker.

dependency-groups

• Format: A table where each key is the name of the group, and the values are arrays of external dependency specifiers, tables, or a mix of both.
• Core metadata: N/A

PEP 735 -style dependency groups, but using external dependency specifiers instead of PEP 508 strings. Every other detail (e.g. group inclusion, name normalization) follows the official dependency groups specification.

cryptography 39.0

pyproject.toml content:

[external]
build-requires = [
  "dep:virtual/compiler/c",
  "dep:virtual/compiler/rust",
  "dep:generic/pkg-config",
]
host-requires = [
  "dep:generic/openssl",
  "dep:generic/libffi",
]

PKG-INFO / METADATA content: N/A.

SciPy 1.10

pyproject.toml content:

[external]
build-requires = [
  "dep:virtual/compiler/c",
  "dep:virtual/compiler/cpp",
  "dep:virtual/compiler/fortran",
  "dep:generic/ninja",
  "dep:generic/pkg-config",
]
host-requires = [
  "dep:virtual/interface/blas",
  "dep:virtual/interface/lapack@>=3.7.1",
]

PKG-INFO / METADATA content: N/A.

Pillow 10.1.0

pyproject.toml content:

[external]
build-requires = [
  "dep:virtual/compiler/c",
]
host-requires = [
  "dep:generic/libjpeg",
  "dep:generic/zlib",
]

[external.optional-host-requires]
extra = [
  "dep:generic/lcms2",
  "dep:generic/freetype",
  "dep:generic/libimagequant",
  "dep:generic/libraqm",
  "dep:generic/libtiff",
  "dep:generic/libxcb",
  "dep:generic/libwebp",
  "dep:generic/openjpeg@>=2.0",
  "dep:generic/tk",
]

PKG-INFO / METADATA content: N/A.

NAVis 1.4.0

pyproject.toml content:

[project.optional-dependencies]
r = ["rpy2"]

[external]
build-requires = [
  "dep:generic/XCB; platform_system=='Linux'",
]

[external.optional-dependencies]
nat = [
  "dep:cran/nat",
  "dep:cran/nat.nblast",
]

PKG-INFO / METADATA content:

Provides-External-Extra: nat
Requires-External-Dep: dep:cran/nat; extra == 'nat'
Requires-External-Dep: dep:cran/nat.nblast; extra == 'nat'

Spyder 6.0

pyproject.toml content:

[external]
dependencies = [
  "dep:cargo/ripgrep",
  "dep:cargo/tree-sitter-cli",
  "dep:golang/github.com/junegunn/fzf",
]

PKG-INFO / METADATA content:

Requires-External-Dep: dep:cargo/ripgrep
Requires-External-Dep: dep:cargo/tree-sitter-cli
Requires-External-Dep: dep:golang/github.com/junegunn/fzf

jupyterlab-git 0.41.0

pyproject.toml content:

[external]
dependencies = [
  "dep:generic/git",
]

[external.optional-build-requires]
dev = [
  "dep:generic/nodejs",
]

PKG-INFO / METADATA content:

Requires-External-Dep: dep:generic/git

PyEnchant 3.2.2

pyproject.toml content:

[external]
dependencies = [
  # libenchant is needed on all platforms but vendored into wheels
  # distributed on PyPI for Windows. Hence choose to encode that in
  # the metadata. Note: there is no completely unambiguous way to do
  # this; another choice is to leave out the environment marker in the
  # source distribution and either live with the unnecessary ``METADATA``
  # entry in the distributed Windows wheels, or to apply a patch to this
  # metadata when building those wheels.
  "dep:github/AbiWord/enchant; platform_system!='Windows'",
]

PKG-INFO / METADATA content:

Requires-External-Dep: dep:github/AbiWord/enchant; platform_system!="Windows"

With dependency groups

pyproject.toml content:

[external.dependency-groups]
dev = [
  "dep:generic/catch2",
  "dep:generic/valgrind",
]

PKG-INFO / METADATA content: N/A.