Recommended but not required dependencies

Package maintainers often use extras to declare optional dependencies that extend the functionality or performance of a package. In some cases, it can be difficult to determine which dependencies should be required and which should be categorized as extras. A balance must be struck between the needs of typical users, who may prefer most features to be available by default, and users who want minimal installations without large, optional dependencies. One solution with existing Python packaging infrastructure is for package maintainers to define an extra called, for example, recommended, which includes all non-essential but suggested dependencies. Users are then instructed to install the package using package[recommended], while those who prefer more control can use package. However, in practice, many users are unaware of the [recommended] syntax, placing the burden on them to know this for a typical installation. Having a way to have recommended dependencies be installed by default while providing a way for users to request a more minimal installation would satisfy this use case, and this PEP describes a solution to this.

Examples of packages that demonstrate this pattern by encouraging users to include extra dependencies by default include:

• astropy: astropy[recommended]
• sunpy: sunpy[all]
• fastapi: fastapi[standard]

Packages supporting multiple backends or frontends

Another common use case for using extras is to define different backends or frontends and dependencies that need to be installed for each backend or frontend. A package might need at least one backend or frontend to be installed in order to be functional, but may be flexible on which backend or frontend this is. Concrete examples of such frontends or backends include:

• The Qt frontend library, which can be provided by PyQt5, PyQt6, PySide2, or PySide6
• Backend databases, such as MySQL, PostgreSQL, SQLite

With current packaging standards, maintainers have to either require one of the backends or frontends or require users to always specify extras, e.g., package[backend], and therefore risk users having an unusable installation if they only install package. Having a way to specify one or more default backend or frontend and providing a way to override these defaults would provide a much better experience for users, and the approach described in this PEP will allow this.

Note that this PEP does not aim to address the issue of disallowing conflicting or incompatible extras, for example if a package requires exactly one frontend or backend package. There is currently no mechanism in Python packaging infrastructure to disallow conflicting or incompatible extras to be installed, and this PEP does not change that.

Examples of packages that require at least one backend or frontend to work and recommend a default extra to install a backend or frontend include:

• kivy: kivy[base]
• napari: napari[all]
• glueviz: glueviz[qt]

In all three cases, installing the package without any extras results in a broken installation, and this is a commonly reported support issue for some of these packages.

Default-Extra Metadata Field

A new multiple-use metadata field, Default-Extra, will be added to the core package metadata. For this field, each entry must be a string specifying an extra that will be automatically included when the package is installed without any extras being specified explicitly.

Only entries already specified in a Provides-Extra entry can be used in a Default-Extra entry.

Examples:

Default-Extra: recommended
Default-Extra: backend1
Default-Extra: backend2
Default-Extra: backend3

Since this introduces a new field in the core package metadata, this will require Metadata-Version to be bumped to the next minor version (2.5 at the time of writing).

New key in [project] metadata table

A new key will be added to the [project] table in project metadata as originally defined in PEP 621 and now defined in the PyPA specifications. This key will be named default-optional-dependency-keys with the following description:

• TOML type: Array of strings
• Corresponding core metadata field: Default-Extra

Each string in default-optional-dependency-keys must be the name of an extra defined in optional-dependencies, and each extra in this array will be converted to a matching Default-Extra entry in the core package metadata. Examples of valid usage which would produce the example Default-Extra entries presented in the previous section are:

[project]
default-optional-dependency-keys = [
    "recommended",
]

and:

[project]
default-optional-dependency-keys = [
    "backend1",
    "backend2",
    "backend3"
]

Overriding default extras

If extras are explicitly given in a dependency specification, the default extras are ignored. Otherwise, the default extras are installed.

For example, if a package defines an extra1 default extra as well as a non-default extra2 extra, then if a user were to install the package with:

$ pip install package

the default extra1 dependency would be included. If the user instead installs the package with:

$ pip install package[extra2]

then the extra2 extra would be installed but the default extra1 extra would be ignored.

If the same package is specified multiple times in an installation command or dependency tree, the default extras must be installed if any of the instances of the package are specified without extras. For instance, if one installs a package spam where package appears several times in the dependency tree:

spam
├── tomato
│   ├── package[extra2]
└── egg
    └── package

then the default extra should be installed because package appears at least once with no extras specified.

An empty set of extras, such as package[] should be interpreted as meaning that the package should be installed without any default extras (unless package appears elsewhere in the dependency tree, in which case, the default extra would be installed as mentioned above). This would provide a universal way of obtaining a minimal installation of a package.

We also note that some tools such as pip currently ignore unrecognized extras, and emit a warning to the user to indicate that the extra has not been recognized, e.g:

$ pip install package[non-existent-extra]
WARNING: package 3.0.0 does not provide the extra 'non-existent-extra'
...

For tools that behave like this (rather than raising an error), if an extra is recognized as invalid in a dependency specification, it should be ignored, and if all specified extras are invalid, then this should be considered equivalent to package[] (rather than package) and not install any default extras.

Finally, we note (as also discussed in Relying on tooling to deselect any default extras) that package installers are allowed to implement their own options to control the above behavior, for example implementing an option that disables default extras for some or all packages regardless of where these packages appear in the dependency tree. If such tool-specific options are implemented, tool developers should make these opt-in, and users should experience the above PEP 771 behavior as default.

Examples

In this section we take a look at the use cases described in the Motivation section and how these can now be addressed by using the specification outlined above.

[project]
default-optional-dependency-keys = [
    "recommended"
]

[project.optional-dependencies]
recommended = [
    "package1",
    "package2"
]

[project]
default-optional-dependency-keys = [
    "recommended"
]

[project.optional-dependencies]
recommended = [
    "scipy",
    "..."
]

$ pip install astropy

$ pip install astropy[]

[project]
default-optional-dependency-keys = [
    "backend1"
]

[project.optional-dependencies]
backend1 = [
    "package1",
    "package2"
]
backend2 = [
    "package3"
]

[project]
default-optional-dependency-keys = [
    "pyqt5"
]

[project.optional-dependencies]
pyqt5 = [
    "PyQt5",
    "..."
]
pyside2 = [
    "PySide2",
    "..."
]
pyqt6 = [
    "PyQt6",
    "..."
]
pyside6 = [
    "PySide6",
    "..."
]

$ pip install napari

$ pip install napari[pyside6]

[project]
default-optional-dependency-keys = [
    "recommended"
]

[project.optional-dependencies]
recommended = [
    "package1",
    "package2"
]
alternative = [
    "package3"
]
additional = [
    "package[recommended]",
    "package4"
]

[project]
default-optional-dependency-keys = [
    "recommended"
]

[project.optional-dependencies]
recommended = [
    "scipy",
    "..."
]
jupyter = [
    "astropy[recommended]",
    "ipywidgets",
    "..."
]

$ pip install astropy[jupyter]

$ pip install astropy[recommended, jupyter]

$ pip install package  # installs default backend and frontend
$ pip install package[]  # installs no backends or frontends
$ pip install package[backend1]  # installs backend1 and default frontend
$ pip install package[frontend2]  # installs frontend2 and default backend
$ pip install package[backend1, frontend2]  # installs backend1 and frontend2

$ pip install package  # installs default backend and frontend
$ pip install package[]  # installs no backends or frontends
$ pip install package[backend1, defaultfrontend]  # installs backend1 and default frontend
$ pip install package[defaultbackend, frontend2]  # installs frontend2 and default backend
$ pip install package[backend1, frontend2]  # installs backend1 and frontend2

Packages not using default extras

Once support for this PEP is added to tools in the packaging ecosystem, packages that do not make use of default extras will continue to work as-is and there should be no break in compatibility.

Packages using default extras

Once packages start defining default extras, those defaults will only be honored with recent versions of packaging tools which implement this PEP, but those packages will remain installable with older packaging tools – with the main difference being that the default extras will not be installed automatically when older packaging tools are used.

As described in How to teach this, package authors need to carefully evaluate when and how they adopt the default extra feature depending on their user base, as some actions (such as moving a required dependency to a default extra) will likely result in breakage for users if a significant fraction of them are still using older package installers that do not support default extras. In this sense, package authors should be aware that this feature, if used in certain ways, can cause backward-compatibility issues for users, and they are thus responsible for ensuring that they minimize the impact to users.

Packaging-related tools

The most significant backward-compatibility aspect is related to assumptions packaging tools make about extras – specifically, this PEP changes the assumption that extras are no longer exclusively additive in terms of adding dependencies to the dependency tree, and specifying some extras can result in fewer dependencies being installed.

A specific example of change in behavior can be seen with pip: consider a package package which has a required dependency of numpy, and a default extra called recommended which includes scipy. If a user installs package[], only package and numpy will be installed. If a user then does:

$ pip freeze > requirements.txt

then requirements.txt will contain e.g.:

package==1.0.2
numpy==2.1.0

However, if the user then installs the requirements from this file using:

$ pip install -r requirements.txt

then pip will install package (which will include the default extra) as well as numpy, so the final environment will contain scipy. A solution in this specific case is for the user to pass --no-deps to pip install to avoid resolving the dependency tree, but the point here is to illustrate that there may be changes in behavior in packaging tools due to the change in the assumption about what impact an extra can have.

It is worth noting that the recently-accepted PEP 751 defines a new file format which is intended to replace alternatives such as the pip freeze output and other tools in future. The new file format is designed so that the packages in the file are installed without resolving dependencies, which means that it will be fully compatible with default extras as specified in this PEP, and will avoid the issue with pip freeze/pip install -r mentioned above.

Package end users

Package users should be provided with clear installation instructions that show what extras are available for packages and how they behave, for example explaining that by default some recommended dependencies or a given frontend or backend will be installed, and how to opt out of this or override defaults, depending what is available.

Package authors

While the mechanism used to define extras and the associated rule about when to use it are clear, package authors need to carefully consider several points before adopting this capability in their packages, to avoid inadvertently breaking backward-compatibility.

$ pip install package  # this installs package[A]
$ pip install package[B]

package1
└── package2
    └── package1

$ pip install package1[]

package1
└── package2
    └── package1[]

Packaging repository maintainers

The impact on individuals who repackage Python libraries for different distributions, such as conda, Homebrew, linux package installers (such as apt and yum) and so on, needs to be considered. Not all package distributions have mechanisms that would line up with the approach described. In fact, some distributions such as conda, do not even have the concept of extras.

There are two cases to consider here:

• In cases where the repackaging is done by hand, such as for a number of conda-forge recipes, and especially where there is no equivalent to extras, the introduction of default extras should not have a large impact since manual decisions already have to be made as to which dependencies to include (for example, the conda-forge recipe for the astropy package mentioned in the Motivation includes all the recommended dependencies by default since there is no way for users to explicitly request them otherwise).
• In cases where the repackaging is done in an automated, way, distribution maintainers will need to carefully consider how to treat default extras, and this may imply a non-negligible amount of work and discussion.

It is impossible for a PEP such as this to exhaustively consider each of the different package distributions. However, ultimately, default extras should be understood as how package authors would like their package to be installed for the majority of users, and this should inform decisions about how default extras should be handled, whether manually or automatically.

Using a meta-package for recommended installations

Using existing packaging tools and infrastructure, package maintainers who want to provide a minimal installation for some users and a default non-minimal installation for regular users (e.g. with recommended dependencies or a default backend) can technically already achieve this if they are willing to distribute two packages instead of one – for example package-core which would be the main package with minimal dependencies, and package which would be a metapackage that would depend on package-core with optional dependencies enabled.

Taking once again a concrete example from the Motivation section, the astropy package defines a recommended extra that users are currently instructed to install in the default installation instructions. In principle, one could rename the existing astropy package to e.g. astropy-core and then create a new astropy package which would be a metapackage that would contain the following dependencies section:

dependencies = [
    "astropy-core[recommended]"
]

Since users may want to pin or place version constraints on the astropy meta-package (e.g. astropy>5.0), the metapackage would need to follow the same versions as the core package, and would need to use strict pinning in the dependency section, e.g.:

version = "7.1.0"
dependencies = [
    "astropy-core[recommended]==7.1.0"
]

This idea may seem appealing because it is technically already feasible. However, in practice, many projects have opted not to do this, for a number of reasons, which we now take a look at. Some of these may not be applicable to future new projects, but some of them apply to all projects, old and new.

$ pip uninstall package

Syntax for deselecting extras

One of the main competing approaches was as follows: instead of having defaults be unselected if any extras were explicitly provided, default extras would need to be explicitly unselected.

In this picture, a new syntax for unselecting extras would be introduced as an extension of the mini-language defined in PEP 508. If a package defined default extras, users could opt out of these defaults by using a minus sign (-) before the extra name. The proposed syntax update would have been as follows:

extras_list   = (-)?identifier (wsp* ',' wsp* (-)?identifier)*

Valid examples of this new syntax would have included, e.g.:

• package[-recommended]
• package[-backend1, backend2]
• package[pdf, -svg]

However, there are two main issues with this approach:

• One would need to define a number of rules for how to interpret corner cases such as if an extra and its negated version were both present in the same dependency specification (e.g. package[pdf, -pdf]) or if a dependency tree included both package[pdf] and package[-pdf], and the rules would not be intuitive to users.
• More critically, this would introduce new syntax into dependency specification, which means that if any package defined a dependency using the new syntax, it and any other package depending on it would no longer be installable by existing packaging tools, so this would be a major backward compatibility break.

For these reasons, this alternative was not included in the final proposal.

Adding a special entry in extras_require

A potential solution that has been explored as an alternative to introducing the new Default-Extra metadata field would be to make use of an extra with a ‘special’ name.

One example would be to use an empty string:

Provides-Extra:
Requires-Dist: numpy ; extra == ''

The idea would be that dependencies installed as part of the ‘empty’ extras would only get installed if another extra was not specified. An implementation of this was proposed in https://github.com/pypa/setuptools/pull/1503, but it was found that there would be no way to make this work without breaking compatibility with existing usage. For example, packages using Setuptools via a setup.py file can do:

setup(
    ...
    extras_require={'': ['package_a']},
)

which is valid and equivalent to having package_a being defined in install_requires, so changing the meaning of the empty string would break compatibility.

In addition, no other string (such as 'default') can be used as a special string since all strings that would be a backward-compatible valid extras name may already be used in existing packages.

There have been suggestions of using the special None Python variable, but again this is not possible, because even though one can use None in a setup.py file, this is not possible in declarative files such as setup.cfg or pyproject.toml, and furthermore ultimately extras names have to be converted to strings in the package metadata. Having:

Provides-Extra: None

would be indistinguishable from the string ‘None’ which may already be used as an extra name in a Python package. If we were to modify the core metadata syntax to allow non-string ‘special’ extras names, then we would be back to modifying the core metadata specification, which is no better than introducing Default-Extra.

Relying on tooling to deselect any default extras

Another option to unselect extras would be to implement this at the level of packaging tools. For instance, pip could include an option such as:

$ pip install package --no-default-extras

This option could apply to all or specific packages, similar to the --no-binary option, e.g.,:

$ pip install package --no-default-extras :all:

The advantage of this approach is that tools supporting default extras could also support unselecting them. This approach would be similar to the --no-install-recommends option for the apt tool.

However, this solution is not ideal on its own because it would not allow packages to specify themselves that they do not need some of the default extras of a dependency. It would also carry risks for users who might disable all default extras in a big dependency tree, potentially breaking packages in the tree that rely on default extras at any point.

Nevertheless, this PEP does not disallow this approach and it is up to the maintainers of different packaging tools to decide if they want to support this kind of option. It is a flag that could at the very least be useful for package maintainers who want to identify places in dependency trees where default extras are being relied on. However, if it is supported, it should be made clear that using this flag does not guarantee a functional environment.