PEP 771 – Default Extras for Python Software Packages
- Author:
- Thomas Robitaille <thomas.robitaille at gmail.com>, Jonathan Dekhtiar <jonathan at dekhtiar.com>
- Sponsor:
- Pradyun Gedam <pradyunsg at gmail.com>
- Discussions-To:
- Discourse thread
- Status:
- Draft
- Type:
- Standards Track
- Topic:
- Packaging
- Created:
- 13-Jan-2025
- Post-History:
- 15-Jan-2025, 06-Feb-2025
Table of Contents
- Abstract
- Motivation
- Rationale
- Specification
- Backward Compatibility
- Security Implications
- How to teach this
- Reference Implementation
- Rejected Ideas
- Open issues
- Copyright
Abstract
PEP 508 specifies a mini-language for declaring package dependencies. One feature of this language is the ability to specify extras, which are optional components of a distribution that, when used, install additional dependencies. This PEP proposes a mechanism to allow one or more extras to be installed by default if none are provided explicitly.
Motivation
Various use cases for default extras and possible solutions in this PEP were discussed extensively on this DPO thread. These fall into two broad cases that that provide the motivation for the present PEP.
Recommended but not required dependencies
Package maintainers often use extras to declare optional dependencies that
extend the functionality or performance of a package. In some cases, it can be
difficult to determine which dependencies should be required and which should be
categorized as extras. A balance must be struck between the needs of typical
users (who may prefer most features to be available ‘by default’) and users who
want minimal installations without large, optional dependencies. One solution
with existing Python packaging infrastructure is for package maintainers to
define an extra called, for example, recommended
, which
includes all non-essential but suggested dependencies. Users are then instructed to
install the package using package[recommended]
, while those who prefer more
control can use package
. However, in practice, many users are unaware
of the [recommended]
syntax, placing the burden on them to know this for a
typical installation. Having a way to have recommended dependencies be installed
by default while providing a way for users to request a more minimal installation
would satisfy this use case, and this PEP describes a solution to this.
Examples of packages that demonstrate this pattern by encouraging users to include extra dependencies by default include:
- astropy:
astropy[recommended]
- fastapi:
fastapi[standard]
- tensorflow:
tensorflow[and-cuda]
Packages supporting multiple backends or frontends
Another common use case for using extras is to define different backends or frontends and dependencies that need to be installed for each backend or frontend. A package might need at least one backend or frontend to be installed in order to be functional, but may be flexible on which backend or frontend this is. Concrete examples of such frontends or backends include:
- The Qt frontend library, which can be provided by PyQt5, PyQt6, PySide2, or PySide6
- BLAS/LAPACK, which have different possible implementations (e.g. OpenBLAS, and MKL)
- FFT libraries, which also have different implementations (e.g.
scipy.fft
and pyFFTW)
With current packaging standards, maintainers have to either
require one of the backends or frontends, or require users
to always specify extras, e.g. package[backend]
and therefore risk users
having an unusable installation if they only install package
. Having a
way to specify one or more default backend or frontend and providing a way to
override these defaults would provide a much better experience for users, and
the approach described in this PEP will allow this.
Note that this PEP does not aim to address the issue of disallowing conflicting or incompatible extras - for example if a package requires exactly one frontend or backend package. There is currently no mechanism in Python packaging infrastructure to disallow conflicting or incompatible extras to be installed, and this PEP does not change that.
Examples of packages that require at least one backend or frontend to work and recommend a default extra to install a backend or frontend include:
In all three cases, installing the package without any extras results in a broken installation (and this is a commonly reported issue for some of these packages).
Rationale
A number of possible solutions have been extensively and vigorously discussed by the community for several years, including in this DPO thread as well as in numerous issues and pull requests. The solution that is presented below:
- is an opt-in solution which means that package maintainers can choose whether or not to use it
- is flexible enough to accommodate both of the major use cases described in Motivation
- re-uses the syntax from PEP 508
It is the only solution out of all those discussed that meets all three criteria.
Specification
Default-Extra
Metadata Field
A new multiple-use metadata field, Default-Extra
, will be added to the core package
metadata.
For this field, each entry must be a string specifying an extra that will be
automatically included when the package is installed without any extras specified explicitly.
Only entries already specified in a Provides-Extra
entry can be used in a Default-Extra
entry.
Examples:
Default-Extra: recommended
Default-Extra: backend1
Default-Extra: backend2
Default-Extra: backend3
Since this introduces a new field in the core package metadata, this will require Metadata-Version to be bumped to the next minor version (2.5 at the time of writing).
New key in [project]
metadata table
A new key will be added to the [project]
table in project metadata as
originally defined in PEP 621 and now defined in the PyPA specifications. This key will be named
default-optional-dependency-keys
with the following description:
- TOML type: Array of strings
- Corresponding core metadata field:
Default-Extra
Each string in default-optional-dependency-keys
must be the name of an extra
defined in optional-dependencies,
and each extra in this array will be converted to a matching Default-Extra
entry in the core package metadata. Examples of valid usage which would
produce the example Default-Extra
entries presented in the previous section are:
[project]
default-optional-dependency-keys = [
"recommended",
]
and:
[project]
default-optional-dependency-keys = [
"backend1",
"backend2",
"backend3"
]
Overriding default extras
If extras are explicitly given in a dependency specification, the default extras are ignored. Otherwise, the default extras are installed.
For example, if a package defines an extra1
default extra as well as a
non-default extra2
extra, then if a user were to install the package with:
$ pip install package
the default extra1
dependency would be included. If the user instead
installs the package with:
$ pip install package[extra2]
then the extra2
extra would be installed but the default extra1
extra
would be ignored.
If the same package is specified multiple times in an installation command or
dependency tree, the default extras must be installed if any of the instances of
the package are specified without extras. For instance, if one installs a
package spam
where package
appears several times in the dependency
tree:
spam
├── tomato
│ ├── package[extra2]
└── egg
└── package
then the default extra should be installed because package
appears at least
once with no extras specified.
Note that package[]
would continue to be equivalent to package
and would
not be provided as a way to install without default extras (see the Rejected
Ideas section for the rationale).
We also note that some tools such as pip currently ignore unrecognized extras, and emit a warning to the user to indicate that the extra has not been recognized, e.g:
$ pip install package[non-existent-extra]
WARNING: package 3.0.0 does not provide the extra 'non-existent-extra'
...
For tools that behave like this (rather than raising an error), if an extra is recognized as invalid in a dependency specification, it should be ignored and treated as if the user has not passed an explicit extra. If none of the provided extras are valid, default extras should be installed.
Installing without default extras
In some cases, package maintainers may want to facilitate installing packages
without any default extras. In this case, as will be shown in more detail in
Examples, the best approach is to define an extra which could be
called e.g. minimal
or nodefault
(the naming would be up to the package
maintainer) which would be an empty set of dependencies. If this extra is
specified, no default extras will be included, so that e.g. package[minimal]
would include only required dependencies and no extras. Note that this requires
no additional specification and is a natural consequence of the rule described
in Overriding default extras.
There are however valid use cases where package maintainers may not want to provide this. For example, in the case of the multiple possible frontends or backends, it may be that the package would not be functional without any of the options. To take a specific example, a package may need either PyQt or PySide to be installed but will not work if none are provided, so a package maintainer may therefore not want to provide an option to install the package without any extras.
Examples
In this section we take a look at the use cases described in the Motivation section and how these can now be addressed by using the specification outlined above.
Recommended dependencies and minimal installations
First, we consider the case of packages that want recommended but not strictly required dependencies installed by default, while also providing a way to only install the required dependencies.
In order to do this, a package maintainer would define an extra called
recommended
containing the recommended but not required dependencies, and
would choose to have this be included as a default extra:
[project]
default-optional-dependency-keys = [
"recommended"
]
[project.optional-dependencies]
recommended = [
"package1",
"package2"
]
In this specific case, a package maintainer may want to allow users to also install the package without the recommended dependencies, in which case they could define an empty extra:
[project.optional-dependencies]
minimal = []
recommended = [
"package1",
"package2"
]
This would then allow users to install package[minimal]
which, since
there would be an extra explicitly specified, would mean the default extra
does not get installed, and since the minimal
extra is empty, no
additional dependencies would be installed.
Maintainers would have the choice as to whether to offer the capability to do a minimal installation or not - in some cases, such as highlighted in the next section, this might not be desirable.
To take a one of the concrete examples of package from the Motivation
section, the astropy package defines a recommended
extra that users are
currently instructed to install in the default installation instructions.
With this PEP, the recommended
extra could be declared as being a default
extra, and a new minimal
extra could be provided for users wishing to
retain the ability to install only the strictly required dependencies:
[project]
default-optional-dependency-keys = [
"recommended"
]
[project.optional-dependencies]
minimal = []
recommended = [
"scipy",
"..."
]
meaning that installing:
$ pip install astropy
would then also install optional but important optional dependencies such as scipy. Advanced users who want a minimal install could then use:
$ pip install astropy[minimal]
Packages requiring at least one backend or frontend
As described in Motivation, some packages may support multiple backends and/or frontends, and in some cases it may be desirable to ensure that there is always at least one backend or frontend package installed, as the package would be unusable otherwise. Concrete examples of this might include a GUI application that needs a GUI library to be present to be usable but is able to support different ones, or a package that can rely on different computational backends but needs at least one to be installed.
In this case, package maintainers could make the choice to define an extra for each backend or frontend, and provide a default, e.g.:
[project]
default-optional-dependency-keys = [
"backend1"
]
[project.optional-dependencies]
backend1 = [
"package1",
"package2"
]
backend2 = [
"package3"
]
Unlike the previous example however, maintainers would not necessarily provide a way to do an installation without any extras since it might leave the package in an unusable state.
If packages can support e.g. multiple backends at the same time, and some of the backends should always be installed, then the dependencies for these must be given as required dependencies rather than using the default extras mechanism.
To take one of the concrete examples mentioned in Motivation, the napari package
can make use of one of PyQt5, PyQt6, PySide2, or PySide6, and users currently
need to explicitly specify napari[all]
in order to have one of these be installed,
or e.g., napari[pyqt5]
to explicitly specify one of the frontend packages. Installing
napari
with no extras results in a non-functional package. With this PEP, napari
could define the following configuration:
[project]
default-optional-dependency-keys = [
"pyqt5"
]
[project.optional-dependencies]
pyqt5 = [
"PyQt5",
"..."
]
pyside2 = [
"PySide2",
"..."
]
pyqt6 = [
"PyQt6",
"..."
]
pyside6 = [
"PySide6",
"..."
]
meaning that:
$ pip install napari
would work out-of-the-box, but there would still be a mechanism for users to explicitly specify a frontend, e.g.:
$ pip install napari[pyside6]
Supporting minimal installations while not always removing default extras
An additional case we consider here is where a package maintainer wants to support minimal installations without any extras, but also wants to support having users specify additional extras without removing the default one. Essentially, they would want:
package[minimal]
to give an installation without any extraspackage
to install recommended dependencies (in arecommended
extras)package[additional]
to install both recommended and additional dependencies (in anadditional
extras)
This could be achieved with e.g:
[project]
default-optional-dependency-keys = [
"recommended"
]
[project.optional-dependencies]
minimal = []
recommended = [
"package1",
"package2"
]
additional = [
"package[recommended]",
"package3"
]
The ability for a package to reference itself in the extras is supported by existing Python packaging tools.
Once again considering a concrete example, astropy could with this PEP define a
recommended
extra (as described in Recommended dependencies and minimal
installations). However, it also defines other extras, including for example
jupyter
, which adds packages that
enhance the user experience inside Jupyter-based
environments. It is possible that users opting in to this extra would still want
recommended dependencies to be installed. In this case, the following
configuration would solve this case:
[project]
default-optional-dependency-keys = [
"recommended"
]
[project.optional-dependencies]
minimal = []
recommended = [
"scipy",
"..."
]
jupyter = [
"astropy[recommended]",
"ipywidgets",
"..."
]
Users installing:
$ pip install astropy[jupyter]
would then get the same as:
$ pip install astropy[recommended, jupyter]
Backward Compatibility
Packages not using default extras
Once support for this PEP is added to tools in the packaging ecosystem, packages that do not make use of default extras will continue to work as-is and there should be no break in compatibility.
Packages using default extras
Once packages start defining default extras, those defaults will only be honored with recent versions of packaging tools which implement this PEP, but those packages will remain installable with older packaging tools – with the main difference being that the default extras will not be installed automatically when older packaging tools are used.
As described in How to teach this, package authors need to carefully evaluate when and how they adopt the default extra feature depending on their user base, as some actions (such as moving a required dependency to a default extra) will likely result in breakage for users if a significant fraction of them are still using older package installers that do not support default extras. In this sense, package authors should be aware that this feature, if used in certain ways, can cause backward-compatibility issues for users, and they are thus responsible for ensuring that they minimize the impact to users.
Security Implications
There are no known security implications for this PEP.
How to teach this
This section outlines information that should be made available to people in different groups in the community in relation to the implementation of this PEP. Some aspects described below will be relevant even before the PEP is fully implemented in packaging tools as there are some preparations that can be done in advance of this implementation to facilitate any potential transition later on. The groups covered below are:
Package end users
Package users should be provided with clear installation instructions that show what extras are available for packages and how they behave, for example explaining that by default some recommended dependencies or a given frontend or backend will be installed, and how to opt out of this or override defaults, depending what is available.
Packaging repository maintainers
The impact on individuals who repackage Python libraries for different
distributions, such as conda, Homebrew, linux package installers (such as apt
and yum
) and
so on, needs to be considered. Not all package distributions have mechanisms
that would line up with the approach described. In fact, some distributions such
as conda, do not even have the concept of extras.
There are two cases to consider here:
- In cases where the repackaging is done by hand, such as for a number of conda-forge
recipes, and especially where there is no equivalent to extras, the
introduction of default extras should not have a large impact since manual
decisions already have to be made as to which dependencies to include (for
example, the conda-forge recipe for the astropy package mentioned in the
Motivation includes all the
recommended
dependencies by default since there is no way for users to explicitly request them otherwise). - In cases where the repackaging is done in an automated, way, distribution maintainers will need to carefully consider how to treat default extras, and this may imply a non-negligible amount of work and discussion.
It is impossible for a PEP such as this to exhaustively consider each of the different package distributions. However, ultimately, default extras should be understood as how package authors would like their package to be installed for the majority of users, and this should inform decisions about how default extras should be handled, whether manually or automatically.
Reference Implementation
The following repository contains a fully functional demo package that makes use of default extras:
https://github.com/wheel-next/pep_771
This makes use of modified branches of several packages, and the following links are to these branches:
In addition, this branch contains a modified version of the Flit package.
The implementations above are proofs-of-concept at this time and the existing changes have not yet been reviewed by the relevant maintainers. Nevertheless, they are functional enough to allow for interested maintainers to try these out.
Rejected Ideas
Syntax for deselecting extras
One of the main competing approaches was as follows: instead of having defaults be unselected if any extras were explicitly provided, default extras would need to be explicitly unselected.
In this picture, a new syntax for unselecting extras would be introduced as an
extension of the mini-language defined in PEP 508. If a package defined
default extras, users could opt out of these defaults by using a minus sign
(-
) before the extra name. The proposed syntax update would have been as follows:
extras_list = (-)?identifier (wsp* ',' wsp* (-)?identifier)*
Valid examples of this new syntax would have included, e.g.:
package[-recommended]
package[-backend1, backend2]
package[pdf, -svg]
However, there are two main issues with this approach:
- One would need to define a number of rules for how to interpret corner cases
such as if an extra and its negated version were both present in the same
dependency specification (e.g.
package[pdf, -pdf]
) or if a dependency tree included bothpackage[pdf]
andpackage[-pdf]
, and the rules would not be intuitive to users. - More critically, this would introduce new syntax into dependency specification, which means that if any package defined a dependency using the new syntax, it and any other package depending on it would no longer be installable by existing packaging tools, so this would be a major backward compatibility break.
For these reasons, this alternative was not included in the final proposal.
Adding a special entry in extras_require
A potential solution that has been explored as an alternative to introducing the
new Default-Extra
metadata field would be to make use of an extra with a
‘special’ name.
One example would be to use an empty string:
Provides-Extra:
Requires-Dist: numpy ; extra == ''
The idea would be that dependencies installed as part of the ‘empty’ extras
would only get installed if another extra was not specified. An implementation
of this was proposed in https://github.com/pypa/setuptools/pull/1503, but it
was found that there would be no way to make this work without breaking
compatibility with existing usage. For example, packages using Setuptools via
a setup.py
file can do:
setup(
...
extras_require={'': ['package_a']},
)
which is valid and equivalent to having package_a
being defined in
install_requires
, so changing the meaning of the empty string would
break compatibility.
In addition, no other string (such as 'default'
) can be used as a special
string since all strings that would be a backward-compatible valid extras name
may already be used in existing packages.
There have been suggestions of using the special None
Python variable, but
again this is not possible, because even though one can use None
in a setup.py
file,
this is not possible in declarative files such as setup.cfg
or
pyproject.toml
, and furthermore ultimately extras names have to be converted
to strings in the package metadata. Having:
Provides-Extra: None
would be indistinguishable from the string ‘None’ which may already be used as
an extra name in a Python package. If we were to modify the core metadata
syntax to allow non-string ‘special’ extras names, then we would be back to
modifying the core metadata specification, which is no better than
introducing Default-Extra
.
Relying on tooling to deselect any default extras
Another option to unselect extras would be to implement this at the level of packaging tools. For instance, pip could include an option such as:
$ pip install package --no-default-extras
This option could apply to all or specific packages, similar to
the --no-binary
option, e.g.,:
$ pip install package --no-default-extras :all:
The advantage of this approach is that tools supporting default extras could
also support unselecting them. This approach would be similar to the --no-install-recommends
option for the apt
tool.
However, this solution is not ideal because it would not allow packages to specify themselves that they do not need some of the default extras of a dependency. It would also carry risks for users who might disable all default extras in a big dependency tree, potentially breaking packages in the tree that rely on default extras at any point. Nevertheless, this PEP does not disallow this approach and it is up to the maintainers of different packaging tools to decide if they want to support this kind of option.
package[]
disables default extras
Another way to specify not to install any extras, including default extras, would
be to use package[]
. However, this would break the current assumption in packaging tools that
package[]
is equivalent to package
, and may also result
in developers overusing []
by default even when it is not needed. As
highlighted in How to teach this, there may also be cases where package
maintainers do not actually want to support an installation without any extras,
for example in cases where at least one backend or frontend must be installed.
Open issues
Should package[]
disable default extras?
Currently, the PEP as written above does not allow package[]
to be
equivalent to installing the package with no extras, but there would be some
benefits to allowing this:
- It would avoid different packages using different names for a ‘no default’
extras (e.g.
minimal
,no-default
,no-defaults
) and reduce the burden for people who don’t want to have to scan through source code or documentation to figure out whether there is the equivalent of aminimal
extra. - It would allow people who want to use existing packages as-is and future
versions of those packages with no default extras to use
package[]
because that syntax works right now, so it would provide a consistent way over time to get a minimal install.
On the other hand, it is not clear at this point whether any tools are currently
relying on package[]
being identical to package
in a way that would
break compatibility if this was done, so this needs to be investigated.
Copyright
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
Source: https://github.com/python/peps/blob/main/peps/pep-0771.rst
Last modified: 2025-02-06 17:02:06 GMT