PEP 735 – Dependency Groups in pyproject.toml
- Author:
- Stephen Rosen <sirosen0 at gmail.com>
- Sponsor:
- Brett Cannon <brett at python.org>
- PEP-Delegate:
- Paul Moore <p.f.moore at gmail.com>
- Discussions-To:
- Discourse thread
- Status:
- Draft
- Type:
- Standards Track
- Topic:
- Packaging
- Created:
- 20-Nov-2023
- Post-History:
- 14-Nov-2023, 20-Nov-2023
Table of Contents
- Abstract
- Motivation
- Rationale
- Specification
- Reference Implementation
- Backwards Compatibility
- Security Implications
- How to Teach This
- Rejected Ideas
- Why not define each Dependency Group as a table?
- Why not define a special string syntax to extend Dependency Specifiers?
- Why not allow for more non-PEP 508 dependency specifiers?
- Why is the table not named
[run]
,[project.dependency-groups]
, …? - Why is pip’s planned implementation of
--only-deps
not sufficient? - Why isn’t <environment manager> a solution?
- Deferred Ideas
- Appendix A: Prior Art in Non-Python Languages
- Appendix B: Prior Art in Python
- Appendix C: Use Cases
- Copyright
Abstract
This PEP specifies a mechanism for storing package requirements in
pyproject.toml
files such that they are not included in any built distribution of
the project.
This is suitable for creating named groups of dependencies, similar to
requirements.txt
files, which launchers, IDEs, and other tools can find and
identify by name.
The feature defined here is referred to as “Dependency Groups”.
Motivation
There are two major use cases for which the Python community has no standardized answer:
- How should development dependencies be defined for packages?
- How should dependencies be defined for projects which do not build distributions (non-package projects)?
In support of these two needs, there are two common solutions which are similar to this proposal:
requirements.txt
files- package extras
Both requirements.txt
files and extras
have limitations which this
standard seeks to overcome.
Note that the two use cases above describe two different types of projects which this PEP seeks to support:
- Python packages, such as libraries
- non-package projects, such as data science projects
Several motivating use cases are defined in detail in the Use Cases Appendix.
Limitations of requirements.txt
files
Many projects may define one or more requirements.txt
iles,
and may arrange them either at the project root (e.g. requirements.txt
and
test-requirements.txt
) or else in a directory (e.g.
requirements/base.txt
and requirements/test.txt
). However, there are
major issues with the use of requirements files in this way:
- There is no standardized naming convention such that tools can discover or use these files by name.
requirements.txt
files are not standardized, but instead provide options topip
.
As a result, it is difficult to define tool behaviors based on
requirements.txt
files. They are not trivial to discover or identify by
name, and their contents may contain a mix of package specifiers and additional
pip
options.
The lack of a standard for requirements.txt
contents also means they are
not portable to any alternative tools which wish to process them other than
pip
.
Additionally, requirements.txt
files require a file per dependency list.
For some use-cases, this makes the marginal cost of dependency groupings high,
relative to their benefit.
A terser declaration is beneficial to projects with a number of small groups of
dependencies.
In contrast with this, Dependency Groups are defined at a well known location
in pyproject.toml
with fully standardized contents. Not only will they have
immediate utility, but they will also serve as a starting point for future
standards.
Limitations of extras
extras
are additional package metadata declared in the
[project.optional-dependencies]
table. They provide names for lists of
package specifiers which are published as part of a package’s metadata, and
which a user can request under that name, as in pip install 'foo[bar]'
to
install foo
with the bar
extra.
Because extras
are package metadata, they are not guaranteed to be
statically defined and may require a build system to resolve.
Furthermore, definition of a [project.optional-dependencies]
indicates to
many tools that a project is a package, and may drive tool behaviors such as
validation of the [project]
table.
For projects which are packages, extras
are a common solution for defining
development dependencies, but even under these circumstances they have
downsides:
- Because an
extra
defines optional additional dependencies, it is not possible to install anextra
without installing the current package and its dependencies. - Because they are user-installable,
extras
are part of the public interface for packages. Becauseextras
are published, package developers often are concerned about ensuring that their development extras are not confused with user-facing extras.
Rationale
This PEP defines the storage of requirements data in lists within a
[dependency-groups]
table.
This name was chosen to match the canonical name of the feature
(“Dependency Groups”).
This format should be as simple and learnable as possible, having a format
very similar to existing requirements.txt
files for many cases. Each list
in [dependency-groups]
is defined as a list of package specifiers. For
example:
[dependency-groups]
test = ["pytest>7", "coverage"]
There are a number of use cases for requirements.txt
files which require
data which cannot be expressed in PEP 508 dependency specifiers. Such
fields are not valid in Dependency Groups. Including many of the data and
fields which pip
supports, such as index servers, hashes, and path
dependencies, requires new standards. This standard leaves room for new
standards and developments, but does not attempt to support all valid
requirements.txt
contents.
The only exception to this is the -r
flag which requirements.txt
files
use to include one file in another. Dependency Groups support an “include”
mechanism which is similar in meaning, allowing one dependency group to extend
another.
Dependency Groups have two additional features which are similar to
requirements.txt
files:
- they are not published as distinct metadata in any built distribution
- installation of a dependency group does not imply installation of a package’s dependencies or the package itself
Use Cases
The following use cases are considered important targets for this PEP. They are defined in greater detail in the Use Cases Appendix.
- Web Applications deployed via a non-python-packaging build process
- Libraries with unpublished dev dependency groups
- Data science projects with groups of dependencies but no core package
- Input data to lockfile generation (Dependency Groups should generally not be used as a location for locked dependency data)
- Input data to an environment manager, such as tox, Nox, or Hatch
- Configurable IDE discovery of test and linter requirements
Regarding Poetry and PDM Dependency Groups
The existing Poetry and PDM tools already offer a feature which each calls
“Dependency Groups”. However, absent any standard for specifying collections
of dependencies, each tool defines these in a tool-specific way, in the
relevant sections of the [tool]
table.
(PDM also uses extras for some Dependency Groups, and overlaps the notion heavily with extras.)
This PEP does not support all of the features of Poetry and PDM, which, like
requirements.txt
files for pip
, support several non-standard extensions
to common dependency specifiers.
It should be possible for such tools to use standardized Dependency Groups as extensions of their own Dependency Group mechanisms. However, defining a new data format which replaces the existing Poetry and PDM solutions is a non-goal. Doing so would require standardizing several additional features, such as path dependencies, which are supported by these tools.
Future Compatibility & Invalid Data
Dependency Groups are intended to be extensible in future PEPs. However, Dependency Groups should also be usable by multiple tools in a single Python project. With multiple tools using the same data, it is possible that one implements a future PEP which extends Dependency Groups, while another does not.
To support users in this case, this PEP defines and recommends validation
behaviors in which tools only examine Dependency Groups which they are using.
This allows multiple tools, using different versions of Dependency Groups data,
to share a single table in pyproject.toml
.
Specification
This PEP defines a new section (table) in pyproject.toml
files named
dependency-groups
. The dependency-groups
table contains an arbitrary
number of user-defined keys, each of which has, as its value, a list of
requirements (defined below). These keys must be
valid non-normalized names,
and must be
normalized
before comparisons.
Tools SHOULD prefer to present the original, non-normalized name to users by default. If duplicate names, after normalization, are encountered, tools SHOULD emit an error.
Requirement lists under dependency-groups
may contain strings, tables
(“dicts” in Python), or a mix of strings and tables.
Strings in requirement lists must be valid Dependency Specifiers, as defined in PEP 508.
Tables in requirement lists must be valid Dependency Object Specifiers.
Dependency Object Specifiers
Dependency Object Specifiers are tables which define zero or more dependencies.
This PEP standardizes only one type of Dependency Object Specifier, a “Dependency Group Include”. Other types may be added in future standards.
Dependency Group Include
A Dependency Group Include includes the dependencies of another Dependency Group in the current Dependency Group.
An include is defined as a table with exactly one key, "include-group"
,
whose value is a string, the name of another Dependency Group.
For example, {include-group = "test"}
is an include which expands to the
contents of the test
Dependency Group.
Includes are defined to be exactly equivalent to the contents of the named
Dependency Group, inserted into the current group at the location of the include.
For example, if foo = ["a", "b"]
is one group, and
bar = ["c", {include-group = "foo"}, "d"]
is another, then bar
should
evaluate to ["c", "a", "b", "d"]
when Dependency Group Includes are expanded.
Dependency Group Includes may specify the same package multiple times. Tools SHOULD NOT deduplicate or otherwise alter the list contents produced by the include. For example, given the following table:
[dependency-groups]
group-a = ["foo"]
group-b = ["foo>1.0"]
group-c = ["foo<1.0"]
all = ["foo", {include-group = "group-a"}, {include-group = "group-b"}, {include-group = "group-c"}]
The resolved value of all
SHOULD be ["foo", "foo", "foo>1.0", "foo<1.0"]
.
Tools should handle such a list exactly as they would handle any other case in
which they are asked to process the same requirement multiple times with
different version constraints.
Dependency Group Includes may include lists containing Dependency Group Includes, in which case those includes should be expanded as well. Dependency Group Includes MUST NOT include cycles, and tools SHOULD report an error if they detect a cycle.
Example Dependency Groups Table
The following is an example of a partial pyproject.toml
which uses this to
define four Dependency Groups: test
, docs
, typing
, and
typing-test
:
[dependency-groups]
test = ["pytest", "coverage"]
docs = ["sphinx", "sphinx-rtd-theme"]
typing = ["mypy", "types-requests"]
typing-test = [{include-group = "typing"}, {include-group = "test"}, "useful-types"]
Note that none of these Dependency Group declarations implicitly install the
current package, its dependencies, or any optional dependencies.
Use of a Dependency Group like test
to test a package requires that the
user’s configuration or toolchain also installs .
. For example,
$TOOL install-dependency-group test
pip install -e .
could be used (supposing $TOOL
is a tool which supports installing
Dependency Groups) to build a testing environment.
This also allows for the docs
dependency group to be used without
installing the project as a package:
$TOOL install-dependency-group docs
Package Building
Build backends MUST NOT include Dependency Group data in built distributions as package metadata. This means that PKG-INFO in sdists and METADATA in wheels do not include any referencable fields containing Dependency Groups.
It is valid to use Dependency Groups in the evaluation of dynamic metadata, and
pyproject.toml
files included in sdists will naturally still contain the
[dependency-groups]
table. However, the table contents are not part of a
published package’s interfaces.
Installing Dependency Groups
Tools which support Dependency Groups are expected to provide new options and interfaces to allow users to install from Dependency Groups.
No syntax is defined for expressing the Dependency Group of a package, for two reasons:
- it would not be valid to refer to the Dependency Groups of a third-party package from PyPI (because the data is defined to be unpublished)
- there is not guaranteed to be a current package for Dependency Groups – part of their purpose is to support non-package projects
For example, a possible pip interface for installing Dependency Groups would be:
pip install --dependency-groups=test,typing
Note that this is only an example. This PEP does not declare any requirements for how tools support the installation of Dependency Groups.
Overlapping Install UX with Extras
Tools MAY choose to provide the same interfaces for installing Dependency Groups as they do for installing extras.
Note that this specification does not forbid having an extra whose name matches a Dependency Group.
Users are advised to avoid creating Dependency Groups whose names match extras. Tools MAY treat such matching as an error.
Validation and Compatibility
Tools supporting Dependency Groups may want to validate data before using it. However, tools implementing such validation behavior should be careful to allow for future expansions to this spec, so that they do not unnecessarily emit errors or warnings in the presence of new syntax.
Tools SHOULD error when evaluating or processing unrecognized data in Dependency Groups.
Tools SHOULD NOT eagerly validate the list contents of all Dependency Groups.
This means that in the presence of the following data, most tools will allow
the foo
group to be used, and will only error when the bar
group is
used:
[dependency-groups]
foo = ["pyparsing"]
bar = [{set-phasers-to = "stun"}]
Linters and Validators may be stricter
Eager validation is discouraged for tools which primarily install or resolve Dependency Groups. Linters and validation tools may have good cause to ignore this recommendation.
Reference Implementation
The following Reference Implementation prints the contents of a Dependency
Group to stdout, newline delimited.
The output is therefore valid requirements.txt
data.
import re
import sys
import tomllib
from collections import defaultdict
from packaging.requirements import Requirement
def _normalize_name(name: str) -> str:
return re.sub(r"[-_.]+", "-", name).lower()
def _normalize_group_names(dependency_groups: dict) -> dict:
original_names = defaultdict(list)
normalized_groups = {}
for group_name, value in dependency_groups.items():
normed_group_name = _normalize_name(group_name)
original_names[normed_group_name].append(group_name)
normalized_groups[normed_group_name] = value
errors = []
for normed_name, names in original_names.items():
if len(names) > 1:
errors.append(f"{normed_name} ({', '.join(names)})")
if errors:
raise ValueError(f"Duplicate dependency group names: {', '.join(errors)}")
return normalized_groups
def _resolve_dependency_group(
dependency_groups: dict, group: str, past_groups: tuple[str, ...] = ()
) -> list[str]:
if group in past_groups:
raise ValueError(f"Cyclic dependency group include: {group} -> {past_groups}")
if group not in dependency_groups:
raise LookupError(f"Dependency group '{group}' not found")
raw_group = dependency_groups[group]
if not isinstance(raw_group, list):
raise ValueError(f"Dependency group '{group}' is not a list")
realized_group = []
for item in raw_group:
if isinstance(item, str):
# packaging.requirements.Requirement parsing ensures that this is a valid
# PEP 508 Dependency Specifier
# raises InvalidRequirement on failure
Requirement(item)
realized_group.append(item)
elif isinstance(item, dict):
if tuple(item.keys()) != ("include-group",):
raise ValueError(f"Invalid dependency group item: {item}")
include_group = _normalize_name(next(iter(item.values())))
realized_group.extend(
_resolve_dependency_group(
dependency_groups, include_group, past_groups + (group,)
)
)
else:
raise ValueError(f"Invalid dependency group item: {item}")
return realized_group
def resolve(dependency_groups: dict, group: str) -> list[str]:
if not isinstance(dependency_groups, dict):
raise TypeError("Dependency Groups table is not a dict")
if not isinstance(group, str):
raise TypeError("Dependency group name is not a str")
return _resolve_dependency_group(dependency_groups, group)
if __name__ == "__main__":
with open("pyproject.toml", "rb") as fp:
pyproject = tomllib.load(fp)
dependency_groups_raw = pyproject["dependency-groups"]
dependency_groups = _normalize_group_names(dependency_groups_raw)
print("\n".join(resolve(pyproject["dependency-groups"], sys.argv[1])))
Backwards Compatibility
At time of writing, the dependency-groups
namespace within a
pyproject.toml
file is unused. Since the top-level namespace is
reserved for use only by standards specified at packaging.python.org,
there are no direct backwards compatibility concerns.
However, the introduction of the feature has implications for a
number of ecosystem tools, especially those which attempt to support
examination of data in setup.py
and requirements.txt
.
Audit and Update Tools
A wide range of tools understand Python dependency data as expressed in
requirements.txt
files. (e.g., Dependabot, Tidelift, etc)
Such tools inspect dependency data and, in some cases, offer tool-assisted or fully automated updates. It is our expectation that no such tools would support the new Dependency Groups at first, and broad ecosystem support could take many months or even some number of years to arrive.
As a result, users of Dependency Groups would experience a degradation in their workflows and tool support at the time that they start using Dependency Groups. This is true of any new standard for where and how dependency data are encoded.
Security Implications
This PEP introduces new syntaxes and data formats for specifying dependency information in projects. However, it does not introduce newly specified mechanisms for handling or resolving dependencies.
It therefore does not carry security concerns other than those inherent in any
tools which may already be used to install dependencies – i.e. malicious
dependencies may be specified here, just as they may be specified in
requirements.txt
files.
How to Teach This
This feature should be referred to by its canonical name, “Dependency Groups”.
The basic form of usage should be taught as a variant on typical
requirements.txt
data. Standard dependency specifiers (PEP 508) can be
added to a named list. Rather than asking pip to install from a
requirements.txt
file, either pip or a relevant workflow tool will install
from a named Dependency Group.
For new Python users, they may be taught directly to create a section in
pyproject.toml
containing their Dependency Groups, similarly to how they
are currently taught to use requirements.txt
files.
This also allows new Python users to learn about pyproject.toml
files
without needing to learn about package building.
A pyproject.toml
file with only [dependency-groups]
and no other tables
is valid.
For both new and experienced users, the Dependency Group Includes will need to
be explained. For users with experience using requirements.txt
, this can be
described as an analogue for -r
. For new users, they should be taught that
an include allows one Dependency Group to extend another. Similar configuration
interfaces and the Python list.extend
method may be used to explain the
idea by analogy.
Python users who have used setup.py
packaging may be familiar with common
practices which predate pyproject.toml
, in which package metadata is
defined dynamically. Requirements loaded from requirements.txt
files and
definitions of static lists prior to setup()
invocation readily analogize
with Dependency Groups.
Interfaces for Use of Dependency Groups
This specificaion provides no universal interface for interacting with
Dependency Groups, other than inclusion in a built package via the project
table. This has implications both for tool authors and for users.
Tool authors should determine how or if Dependency Groups are relevant to their
user stories, and build their own interfaces to fit.
For environment managers, resolvers, installers, and related non-build tools,
they will be able to document that they support “PEP 735 Dependency Groups”,
but they will be responsible for documenting their usage modes.
For build backends, supporting Dependency Groups will require support for
inclusion from the project
table, but set no other strict requirements.
For users, the primary consequence is that they must consult relevant tool documentation whenever they wish to use Dependency Groups outside of package builds. Users should be advised by tools, either through documentation or runtime warnings or errors, about usages which are disrecommended or not supported. For example, if a tool wishes to require that all Dependency Groups are mutually compatible, containing no contradictory package specifiers, it should document that restriction and advise users on how to appropriately leverage Dependency Groups for its purposes.
Rejected Ideas
Why not define each Dependency Group as a table?
If our goal is to allow for future expansion, then defining each Dependency Group as a subtable, thus enabling us to attach future keys to each group, allows for the greatest future flexibility.
However, it also makes the structure nested more deeply, and therefore harder
to teach and learn. One of the goals of this PEP is to be an easy replacement
for many requirements.txt
use-cases.
Why not define a special string syntax to extend Dependency Specifiers?
Earlier drafts of this specification defined syntactic forms for Dependency Group Includes and Path Dependencies.
However, there were three major issues with this approach:
- it complicates the string syntax which must be taught, beyond PEP 508
- the resulting strings would always need to be disambiguated from PEP 508 specifiers, complicating implementations
Why not allow for more non-PEP 508 dependency specifiers?
Several use cases surfaced during discussion which need more expressive specifiers than are possible with PEP 508.
“Path Dependencies”, referring to local paths, and references to
[project.dependencies]
were of particular interest.
However, there are no existing standards for these features (excepting the
de-facto standard of pip
’s implementation details).
As a result, attempting to include these features in this PEP results in a
significant growth in scope, to attempt to standardize these various features
and pip
behaviors.
Special attention was devoted to attempting to standardize the expression of
editable installations, as expressed by pip install -e
and PEP 660.
However, although the creation of editable installs is standardized for build
backends, the behavior of editables is not standardized for installers.
Inclusion of editables in this PEP requires that any supporting tool allows for
the installation of editables.
Therefore, although Poetry and PDM provide syntaxes for some of these features, they are considered insufficiently standardized at present for inclusion in Dependency Groups.
Why is the table not named [run]
, [project.dependency-groups]
, …?
There are many possible names for this concept.
It will have to live alongside the already existing [project.dependencies]
and [project.optional-dependencies]
tables, and possibly a new
[external]
dependency table as well (at time of writing, PEP 725, which
defines the [external]
table, is in progress).
[run]
was a leading proposal in earlier discussions, but its proposed usage
centered around a single set of runtime dependencies. This PEP explicitly
outlines multiple groups of dependencies, which makes [run]
a less
appropriate fit – this is not just dependency data for a specific runtime
context, but for multiple contexts.
[project.dependency-groups]
would offer a nice parallel with
[project.dependencies]
and [project.optional-dependencies]
, but has
major downsides for non-package projects.
[project]
requires several keys to be defined, such as name
and
version
. Using this name would either require redefining the [project]
table to allow for these keys to be absent, or else would impose a requirement
on non-package projects to define and use these keys. By extension, it would
effectively require any non-package project allow itself to be treated as a
package.
Why is pip’s planned implementation of --only-deps
not sufficient?
pip currently has a feature on the roadmap to add an –only-deps flag. This flag is intended to allow users to install package dependencies and extras without installing the current package.
It does not address the needs of non-package projects, nor does it allow for the installation of an extra without the package dependencies.
Why isn’t <environment manager> a solution?
Existing environment managers like tox, Nox, and Hatch already have the ability to list inlined dependencies as part of their configuration data. This meets many development dependency needs, and clearly associates dependency groups with relevant tasks which can be run. These mechanisms are good but they are not sufficient.
First, they do not address the needs of non-package projects.
Second, there is no standard for other tools to use to access these data. This
has impacts on high-level tools like IDEs and Dependabot, which cannot support
deep integration with these Dependency Groups. (For example, at time of writing
Dependabot will not flag dependencies which are pinned in tox.ini
files.)
Deferred Ideas
Why not support Dependency Group Includes in [project.dependencies]
or [project.optional-dependencies]
?
Earlier drafts of this specification allowed Dependency Group Includes to be
used in the [project]
table.
However, there were several issues raised during community feedback which led
to its removal.
Only a small number of additional use cases would be addressed by the inclusion
of Dependency Groups, and it increased the scope of the specification
significantly. In particular, this inclusion would increase the number of parties
impacted by the addition. Many readers of the [project]
table, including build
backends, SBOM generators, and dependency analyzers are implicated by a change to
[project]
but may continue to operate as-is in the presence of a new (but
unconnected) [dependency-groups]
table.
Separately from the above concern, allowing inclusion of dependency groups from the
[project]
table encourages package maintainers to move dependency metadata out
of the current standard location.
This complicates static pyproject.toml
metadata and conflicts with the goal of
PEP 621 to store dependency metadata in a single location.
Finally, exclusion of [project]
support from this PEP is not final. The
use of includes from that table, or an inclusion syntax from
[dependency-groups]
into [project]
, could be introduced by a future
PEP and considered on its own merits.
Use Cases for Dependency Group Includes From [project]
Although deferred in this PEP, allowing includes from the [project]
table would address several use cases.
In particular, there are cases in which package developers would like to install only the dependencies of a package, without the package itself.
For example:
- Specify different environment variables or options when building dependencies vs when building the package itself
- Creating layered container images in which the dependencies are isolated from the package being installed
- Providing the dependencies to analysis environments (e.g., type checking) without having to build and install the package itself
For an example of the last case, consider the following sample
pyproject.toml
:
[project]
dependencies = [{include = "runtime"}]
[optional-dependencies]
foo = [{include = "foo"}]
[dependency-groups]
runtime = ["a", "b"]
foo = ["c", "d"]
typing = ["mypy", {include = "runtime"}, {include = "foo"}]
In this case, a typing
group can be defined, with all of the package’s
runtime dependencies, but without the package itself. This allows uses of the
typing
Dependency Group to skip installation of the package – not only is
this more efficient, but it may reduce the requirements for testing systems.
Why not support Dependency Group Includes in [build-system.requires]
?
Given that we will not allow for [project]
usage of Dependency Groups,
[build-system.requires]
can be considered in comparison with
[project.dependencies]
.
There are fewer theoretical usages for build requirements specified in a group than package requirements. Additionally, the impact of such a change implicates PEP 517 frontend, which would need to support Dependency Groups in order to prepare a build environment.
Compared with changes to [project.dependencies]
and
[project.optional-dependencies]
, changing the behaviors of
[build-system.requires]
is higher impact and has fewer potential uses.
Therefore, given that this PEP declines to make changes to the [project]
table, changing [build-system]
is also deferred.
Why not support a Dependency Group which includes the current project?
Several usage scenarios for dependency groups revolve around installing a
dependency group alongside a package defined in the [project]
table.
For example, testing a package involves installing testing dependencies and the
package itself. Additionally, the compatibility of a dependency group with the
main package is a valuable input to lockfile generators.
In such cases, it is desirable for a Dependency Group to declare that it
depends upon the project itself. Example syntaxes from discussions included
{include-project = true}
and {include-group = ":project:"}
.
However, if a specification is established to extend PEP 508 with Path
Dependencies, this would result in Dependency Groups having two ways of
specifying the main package. For example, if .
becomes formally supported,
and {include-project = true}
is included in this PEP, then dependency
groups may specify any of the following groups
[dependency-groups]
case1 = [{include-project = true}]
case2 = ["."]
case3 = [{include-project = true}, "."]
case4 = [{include-project = false}, "."]
In order to avoid a confusing future in which multiple different options
specify the package defined in pyproject.toml
, any syntax for declaring
this relationship is omitted from this PEP.
Appendix A: Prior Art in Non-Python Languages
This section is primarily informational and serves to document how other language ecosystems solve similar problems.
JavaScript and package.json
In the JavaScript community, packages contain a canonical configuration and
data file, similar in scope to pyproject.toml
, at package.json
.
Two keys in package.json
control dependency data: "dependencies"
and
"devDependencies"
. The role of "dependencies"
is effectively the same
as that of [project.dependencies]
in pyproject.toml
, declaring the
direct dependencies of a package.
"dependencies"
data
Dependency data is declared in package.json
as a mapping from package names
to version specifiers.
Version specifiers support a small grammar of possible versions, ranges, and other values, similar to Python’s PEP 440 version specifiers.
For example, here is a partial package.json
file declaring a few
dependencies:
{
"dependencies": {
"@angular/compiler": "^17.0.2",
"camelcase": "8.0.0",
"diff": ">=5.1.0 <6.0.0"
}
}
The use of the @
symbol is a scope which declares the package
owner, for organizationally owned packages.
"@angular/compiler"
therefore declares a package named compiler
grouped
under angular
ownership.
Dependencies Referencing URLs and Local Paths
Dependency specifiers support a syntax for URLs and Git repositories, similar to the provisions in Python packaging.
URLs may be used in lieu of version numbers. When used, they implicitly refer to tarballs of package source code.
Git repositories may be similarly used, including support for committish specifiers.
Unlike PEP 440, NPM allows for the use of local paths to package source code
directories for dependencies. When these data are added to package.json
via
the standard npm install --save
command, the path is normalized to a
relative path, from the directory containing package.json
, and prefixed
with file:
. For example, the following partial package.json
contains a
reference to a sibling of the current directory:
{
"dependencies": {
"my-package": "file:../foo"
}
}
The official NPM documentation states that local path dependencies “should not” be published to public package repositories, but makes no statement about the inherent validity or invalidity of such dependency data in published packages.
"devDependencies"
data
package.json
is permitted to contain a second section named
"devDependencies"
, in the same format as "dependencies"
.
The dependencies declared in "devDependencies"
are not installed by default
when a package is installed from the package repository (e.g. as part of a
dependency being resolved) but are installed when npm install
is run in the
source tree containing package.json
.
Just as "dependencies"
supports URLs and local paths, so does
"devDependencies"
.
"peerDependencies"
and "optionalDependencies"
There are two additional, related sections in package.json
which have
relevance.
"peerDependencies"
declares a list of dependencies in the same format as
"dependencies"
, but with the meaning that these are a compatibility
declaration.
For example, the following data declares compatibility with package foo
version 2:
{
"peerDependencies": {
"foo": "2.x"
}
}
"optionalDependencies"
declares a list of dependencies which should be
installed if possible, but which should not be treated as failures if they are
unavailable. It also uses the same mapping format as "dependencies"
.
"peerDependenciesMeta"
"peerDependenciesMeta"
is a section which allows for additional control
over how "peerDependencies"
are treated.
Warnings about missing dependencies can be disabled by setting packages to
optional
in this section, as in the following sample:
{
"peerDependencies": {
"foo": "2.x"
},
"peerDependenciesMeta": {
"foo": {
"optional": true
}
}
}
--omit
and --include
The npm install
command supports two options, --omit
and --include
,
which can control whether “prod”, “dev”, “optional”, or “peer” dependencies are installed.
The “prod” name refers to dependencies listed under "dependencies"
.
By default, all four groups are installed when npm install
is executed
against a source tree, but these options can be used to control installation
behavior more precisely.
Furthermore, these values can be declared in .npmrc
files, allowing
per-user and per-project configurations to control installation behaviors.
Ruby & Ruby Gems
Ruby projects may or may not be intended to produce packages (“gems”) in the Ruby ecosystem. In fact, the expectation is that most users of the language do not want to produce gems and have no interest in producing their own packages. Many tutorials do not touch on how to produce packages, and the toolchain never requires user code to be packaged for supported use-cases.
Ruby splits requirement specification into two separate files.
Gemfile
: a dedicated file which only supports requirement data in the form of dependency groups<package>.gemspec
: a dedicated file for declaring package (gem) metadata
The bundler
tool, providing the bundle
command, is the primary interface
for using Gemfile
data.
The gem
tool is responsible for building gems from .gemspec
data, via the
gem build
command.
Gemfiles & bundle
A Gemfile is a Ruby file
containing gem
directives enclosed in any number of group
declarations.
gem
directives may also be used outside of the group
declaration, in which
case they form an implicitly unnamed group of dependencies.
For example, the following Gemfile
lists rails
as a project dependency.
All other dependencies are listed under groups:
source 'https://rubygems.org'
gem 'rails'
group :test do
gem 'rspec'
end
group :lint do
gem 'rubocop'
end
group :docs do
gem 'kramdown'
gem 'nokogiri'
end
If a user executes bundle install
with these data, all groups are
installed. Users can deselect groups by creating or modifying a bundler config
in .bundle/config
, either manually or via the CLI. For example, bundle
config set --local without 'lint:docs'
.
It is not possible, with the above data, to exclude the top-level use of the
'rails'
gem or to refer to that implicit grouping by name.
gemspec and packaged dependency data
A gemspec file is a ruby file containing a Gem::Specification instance declaration.
Only two fields in a Gem::Specification
pertain to package dependency data.
These are add_development_dependency
and add_runtime_dependency
.
A Gem::Specification
object also provides methods for adding dependencies
dynamically, including add_dependency
(which adds a runtime dependency).
Here is a variant of the rails.gemspec
file, with many fields removed or
shortened to simplify:
version = '7.1.2'
Gem::Specification.new do |s|
s.platform = Gem::Platform::RUBY
s.name = "rails"
s.version = version
s.summary = "Full-stack web application framework."
s.license = "MIT"
s.author = "David Heinemeier Hansson"
s.files = ["README.md", "MIT-LICENSE"]
# shortened from the real 'rails' project
s.add_dependency "activesupport", version
s.add_dependency "activerecord", version
s.add_dependency "actionmailer", version
s.add_dependency "activestorage", version
s.add_dependency "railties", version
end
Note that there is no use of add_development_dependency
.
Some other mainstream, major packages (e.g. rubocop
) do not use development
dependencies in their gems.
Other projects do use this feature. For example, kramdown
makes use of
development dependencies, containing the following specification in its
Rakefile
:
s.add_dependency "rexml"
s.add_development_dependency 'minitest', '~> 5.0'
s.add_development_dependency 'rouge', '~> 3.0', '>= 3.26.0'
s.add_development_dependency 'stringex', '~> 1.5.1'
The purpose of development dependencies is only to declare an implicit group,
as part of the .gemspec
, which can then be used by bundler
.
For full details, see the gemspec
directive in bundler
's
documentation on Gemfiles.
However, the integration between .gemspec
development dependencies and
Gemfile
/bundle
usage is best understood via an example.
gemspec development dependency example
Consider the following simple project in the form of a Gemfile
and .gemspec
.
The cool-gem.gemspec
file:
Gem::Specification.new do |s|
s.author = 'Stephen Rosen'
s.name = 'cool-gem'
s.version = '0.0.1'
s.summary = 'A very cool gem that does cool stuff'
s.license = 'MIT'
s.files = []
s.add_dependency 'rails'
s.add_development_dependency 'kramdown'
end
and the Gemfile
:
source 'https://rubygems.org'
gemspec
The gemspec
directive in Gemfile
declares a dependency on the local
package, cool-gem
, defined in the locally available cool-gem.gemspec
file. It also implicitly adds all development dependencies to a dependency
group named development
.
Therefore, in this case, the gemspec
directive is equivalent to the
following Gemfile
content:
gem 'cool-gem', :path => '.'
group :development do
gem 'kramdown'
end
Appendix B: Prior Art in Python
In the absence of any prior standard for Dependency Groups, two known workflow tools, PDM and Poetry, have defined their own solutions.
This section will primarily focus on these two tools as cases of prior art regarding the definition and use of Dependency Groups in Python.
Projects are Packages
Both PDM and Poetry treat the projects they support as packages.
This allows them to use and interact with standard pyproject.toml
metadata
for some of their needs, and allows them to support installation of the
“current project” by doing a build and install using their build backends.
Effectively, this means that neither Poetry nor PDM supports non-package projects.
Non-Standard Dependency Specifiers
PDM and Poetry extend PEP 508 dependency specifiers with additional features which are not part of any shared standard. The two tools use slightly different approaches to these problems, however.
PDM supports specifying local paths, and editable installs, via a syntax which
looks like a set of arguments to pip install
. For example, the following
dependency group includes a local package in editable mode:
[tool.pdm.dev-dependencies]
mygroup = ["-e file:///${PROJECT_ROOT}/foo"]
This declares a dependency group mygroup
which includes a local editable
install from the foo
directory.
Poetry describes dependency groups as tables, mapping package names to
specifiers. For example, the same configuration as the above mygroup
example might appear as follows under Poetry:
[tool.poetry.group.mygroup]
foo = { path = "foo", editable = true }
PDM restricts itself to a string syntax, and Poetry introduces tables which describe dependencies.
Installing and Referring to Dependency Groups
Both PDM and Poetry have tool-specific support for installing dependency groups. Because both projects support their own lockfile formats, they also both have the capability to transparently use a dependency group name to refer to the locked dependency data for that group.
However, neither tool’s dependency groups can be referenced natively from other
tools like tox
, nox
, or pip
.
Attempting to install a dependency group under tox
, for example, requires
an explicit call to PDM or Poetry to parse their dependency data and do the
relevant installation step.
Appendix C: Use Cases
Web Applications
A web application (e.g. a Django or Flask app) often does not need to build a distribution, but bundles and ships its source to a deployment toolchain.
For example, a source code repository may define Python packaging metadata as
well as containerization or other build pipeline metadata (Dockerfile
,
etc).
The Python application is built by copying the entire repository into a
build context, installing dependencies, and bundling the result as a machine
image or container.
Such applications have dependency groups for the build, but also for linting,
testing, etc. In practice, today, these applications often define themselves as
packages to be able to use packaging tools and mechanisms like extras
to
manage their dependency groups. However, they are not conceptually packages,
meant for distribution in sdist or wheel format.
Dependency Groups allow these applications to define their various dependencies without relying on packaging metadata, and without trying to express their needs in packaging terms.
Libraries
Libraries are Python packages which build distributions (sdist and wheel) and publish them to PyPI.
For libraries, Dependency Groups represent an alternative to extras
for
defining groups of development dependencies, with the important advantages
noted above.
A library may define groups for test
and typing
which allow testing and
type-checking, and therefore rely on the library’s own dependencies (as
specified in [project.dependencies]
).
Other development needs may not require installation of the package at all. For
example, a lint
Dependency Group may be valid and faster to install without
the library, as it only installs tools like black
, ruff
, or flake8
.
lint
and test
environments may also be valuable locations to hook in
IDE or editor support. See the case below for a fuller description of such
usage.
Here’s an example Dependency Groups table which might be suitable for a library:
[dependency-groups]
test = ["pytest<8", "coverage"]
typing = ["mypy==1.7.1", "types-requests"]
lint = ["black", "flake8"]
typing-test = [{include-group = "typing"}, "pytest<8"]
Note that none of these implicitly install the library itself.
It is therefore the responsibility of any environment management toolchain to
install the appropriate Dependency Groups along with the library when needed,
as in the case of test
.
Data Science Projects
Data Science Projects typically take the form of a logical collection of scripts and utilities for processing and analyzing data, using a common toolchain. Components may be defined in the Jupyter Notebook format (ipynb), but rely on the same common core set of utilities.
In such a project, there is no package to build or install. Therefore,
pyproject.toml
currently does not offer any solution for dependency
management or declaration.
It is valuable for such a project to be able to define at least one major grouping of dependencies. For example:
[dependency-groups]
main = ["numpy", "pandas", "matplotlib"]
However, it may also be necessary for various scripts to have additional supporting tools. Projects may even have conflicting or incompatible tools or tool versions for different components, as they evolve over time.
Consider the following more elaborate configuration:
[dependency-groups]
main = ["numpy", "pandas", "matplotlib"]
scikit = [{include-group = "main"}, "scikit-learn==1.3.2"]
scikit-old = [{include-group = "main"}, "scikit-learn==0.24.2"]
This defines scikit
and scikit-old
as two similar variants of the
common suite of dependencies, pulling in different versions of scikit-learn
to suit different scripts.
This PEP only defines these data. It does not formalize any mechanism for a Data Science Project (or any other type of project) to install the dependencies into known environments or associate those environments with the various scripts. Such combinations of data are left as a problem for tool authors to solve, and perhaps eventually standardize.
Lockfile Generation
There are a number of tools which generate lockfiles in the Python ecosystem
today. PDM and Poetry each use their own lockfile formats, and pip-tools
generates requirements.txt
files with version pins and hashes.
Dependency Groups are not an appropriate place to store lockfiles, as they lack many of the necessary features. Most notably, they cannot store hashes, which most lockfile users consider essential.
However, Dependency Groups are a valid input to tools which generate lockfiles. Furthermore, PDM and Poetry both allow a Dependency Group name (under their notions of Dependency Groups) to be used to refer to its locked variant.
Therefore, consider a tool which produces lockfiles, here called $TOOL
.
It might be used as follows:
$TOOL lock --dependency-group=test
$TOOL install --dependency-group=test --use-locked
All that such a tool needs to do is to ensure that its lockfile data records
the name test
in order to support such usage.
The mutual compatibility of Dependency Groups is not guaranteed. For example,
the Data Science example above shows conflicting versions of scikit-learn
.
Therefore, installing multiple locked dependency groups in tandem may require
that tools apply additional constraints or generate additional lockfile data.
These problems are considered out of scope for this PEP.
As two examples of how combinations might be locked:
- A tool might require that lockfile data be explicitly generated for any combination to be considered valid
- Poetry implements the requirement that all Dependency Groups be mutually compatible, and generates only one locked version. (Meaning it finds a single solution, rather than a set or matrix of solutions.)
Environment Manager Inputs
A common usage in tox, Nox, and Hatch is to install a set of dependencies into a testing environment.
For example, under tox.ini
, type checking dependencies may be defined
inline:
[testenv:typing]
deps =
pyright
useful-types
commands = pyright src/
This combination provides a desirable developer experience within a limited
context. Under the relevant environment manager, the dependencies which are
needed for the test environment are declared alongside the commands which need
those dependencies. They are not published in package metadata, as extras
would be, and they are discoverable for the tool which needs them to build the
relevant environment.
Dependency Groups apply to such usages by effectively “lifting” these
requirements data from a tool-specific location into a more broadly available
one. In the example above, only tox
has access to the declared list of
dependencies. Under an implementation supporting dependency groups, the same
data might be available in a Dependency Group:
[dependency-groups]
typing = ["pyright", "useful-types"]
The data can then be used under multiple tools. For example, tox
might
implement support as dependency_groups = typing
, replacing the deps
usage above.
In order for Dependency Groups to be a viable alternative for users of environment managers, the environment managers will need to support processing Dependency Groups similarly to how they support inline dependency declaration.
IDE and Editor Use of Requirements Data
IDE and editor integrations may benefit from conventional or configurable name definitions for Dependency Groups which are used for integrations.
There are at least two known scenarios in which it is valuable for an editor or IDE to be capable of discovering the non-published dependencies of a project:
- testing: IDEs such as VS Code support GUI interfaces for running particular tests
- linting: editors and IDEs often support linting and autoformatting integrations which highlight or autocorrect errors
These cases could be handled by defining conventional group names like
test
, lint
, and fix
, or by defining configuration mechanisms which
allow the selection of Dependency Groups.
For example, the following pyproject.toml
declares the three aforementioned
groups:
[dependency-groups]
test = ["pytest", "pytest-timeout"]
lint = ["flake8", "mypy"]
fix = ["black", "isort", "pyupgrade"]
This PEP makes no attempt to standardize such names or reserve them for such uses. IDEs may standardize or may allow users to configure the group names used for various purposes.
This declaration allows the project author’s knowledge of the appropriate tools for the project to be shared with all editors of that project.
Copyright
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
Source: https://github.com/python/peps/blob/main/peps/pep-0735.rst
Last modified: 2024-10-02 21:05:48 GMT