Following system colour scheme Selected dark colour scheme Selected light colour scheme

Python Enhancement Proposals

PEP 771 – Default Extras for Python Software Packages

Author:
Thomas Robitaille <thomas.robitaille at gmail.com>, Jonathan Dekhtiar <jonathan at dekhtiar.com>
Sponsor:
Pradyun Gedam <pradyunsg at gmail.com>
Discussions-To:
Discourse thread
Status:
Draft
Type:
Standards Track
Topic:
Packaging
Created:
13-Jan-2025
Post-History:
15-Jan-2025, 06-Feb-2025

Table of Contents

Abstract

PEP 508 specifies a mini-language for declaring package dependencies. One feature of this language is the ability to specify extras, which are optional components of a distribution that, when used, install additional dependencies. This PEP proposes a mechanism to allow one or more extras to be installed by default if none are provided explicitly.

Motivation

Various use cases for default extras and possible solutions in this PEP were discussed extensively on this DPO thread. These fall into two broad cases that that provide the motivation for the present PEP.

Packages supporting multiple backends or frontends

Another common use case for using extras is to define different backends or frontends and dependencies that need to be installed for each backend or frontend. A package might need at least one backend or frontend to be installed in order to be functional, but may be flexible on which backend or frontend this is. Concrete examples of such frontends or backends include:

  • The Qt frontend library, which can be provided by PyQt5, PyQt6, PySide2, or PySide6
  • BLAS/LAPACK, which have different possible implementations (e.g. OpenBLAS, and MKL)
  • FFT libraries, which also have different implementations (e.g. scipy.fft and pyFFTW)

With current packaging standards, maintainers have to either require one of the backends or frontends, or require users to always specify extras, e.g. package[backend] and therefore risk users having an unusable installation if they only install package. Having a way to specify one or more default backend or frontend and providing a way to override these defaults would provide a much better experience for users, and the approach described in this PEP will allow this.

Note that this PEP does not aim to address the issue of disallowing conflicting or incompatible extras - for example if a package requires exactly one frontend or backend package. There is currently no mechanism in Python packaging infrastructure to disallow conflicting or incompatible extras to be installed, and this PEP does not change that.

Examples of packages that require at least one backend or frontend to work and recommend a default extra to install a backend or frontend include:

In all three cases, installing the package without any extras results in a broken installation (and this is a commonly reported issue for some of these packages).

Rationale

A number of possible solutions have been extensively and vigorously discussed by the community for several years, including in this DPO thread as well as in numerous issues and pull requests. The solution that is presented below:

  • is an opt-in solution which means that package maintainers can choose whether or not to use it
  • is flexible enough to accommodate both of the major use cases described in Motivation
  • re-uses the syntax from PEP 508

It is the only solution out of all those discussed that meets all three criteria.

Specification

Default-Extra Metadata Field

A new multiple-use metadata field, Default-Extra, will be added to the core package metadata. For this field, each entry must be a string specifying an extra that will be automatically included when the package is installed without any extras specified explicitly.

Only entries already specified in a Provides-Extra entry can be used in a Default-Extra entry.

Examples:

Default-Extra: recommended
Default-Extra: backend1
Default-Extra: backend2
Default-Extra: backend3

Since this introduces a new field in the core package metadata, this will require Metadata-Version to be bumped to the next minor version (2.5 at the time of writing).

New key in [project] metadata table

A new key will be added to the [project] table in project metadata as originally defined in PEP 621 and now defined in the PyPA specifications. This key will be named default-optional-dependency-keys with the following description:

  • TOML type: Array of strings
  • Corresponding core metadata field: Default-Extra

Each string in default-optional-dependency-keys must be the name of an extra defined in optional-dependencies, and each extra in this array will be converted to a matching Default-Extra entry in the core package metadata. Examples of valid usage which would produce the example Default-Extra entries presented in the previous section are:

[project]
default-optional-dependency-keys = [
    "recommended",
]

and:

[project]
default-optional-dependency-keys = [
    "backend1",
    "backend2",
    "backend3"
]

Overriding default extras

If extras are explicitly given in a dependency specification, the default extras are ignored. Otherwise, the default extras are installed.

For example, if a package defines an extra1 default extra as well as a non-default extra2 extra, then if a user were to install the package with:

$ pip install package

the default extra1 dependency would be included. If the user instead installs the package with:

$ pip install package[extra2]

then the extra2 extra would be installed but the default extra1 extra would be ignored.

If the same package is specified multiple times in an installation command or dependency tree, the default extras must be installed if any of the instances of the package are specified without extras. For instance, if one installs a package spam where package appears several times in the dependency tree:

spam
├── tomato
│   ├── package[extra2]
└── egg
    └── package

then the default extra should be installed because package appears at least once with no extras specified.

Note that package[] would continue to be equivalent to package and would not be provided as a way to install without default extras (see the Rejected Ideas section for the rationale).

We also note that some tools such as pip currently ignore unrecognized extras, and emit a warning to the user to indicate that the extra has not been recognized, e.g:

$ pip install package[non-existent-extra]
WARNING: package 3.0.0 does not provide the extra 'non-existent-extra'
...

For tools that behave like this (rather than raising an error), if an extra is recognized as invalid in a dependency specification, it should be ignored and treated as if the user has not passed an explicit extra. If none of the provided extras are valid, default extras should be installed.

Installing without default extras

In some cases, package maintainers may want to facilitate installing packages without any default extras. In this case, as will be shown in more detail in Examples, the best approach is to define an extra which could be called e.g. minimal or nodefault (the naming would be up to the package maintainer) which would be an empty set of dependencies. If this extra is specified, no default extras will be included, so that e.g. package[minimal] would include only required dependencies and no extras. Note that this requires no additional specification and is a natural consequence of the rule described in Overriding default extras.

There are however valid use cases where package maintainers may not want to provide this. For example, in the case of the multiple possible frontends or backends, it may be that the package would not be functional without any of the options. To take a specific example, a package may need either PyQt or PySide to be installed but will not work if none are provided, so a package maintainer may therefore not want to provide an option to install the package without any extras.

Examples

In this section we take a look at the use cases described in the Motivation section and how these can now be addressed by using the specification outlined above.

Packages requiring at least one backend or frontend

As described in Motivation, some packages may support multiple backends and/or frontends, and in some cases it may be desirable to ensure that there is always at least one backend or frontend package installed, as the package would be unusable otherwise. Concrete examples of this might include a GUI application that needs a GUI library to be present to be usable but is able to support different ones, or a package that can rely on different computational backends but needs at least one to be installed.

In this case, package maintainers could make the choice to define an extra for each backend or frontend, and provide a default, e.g.:

[project]
default-optional-dependency-keys = [
    "backend1"
]

[project.optional-dependencies]
backend1 = [
    "package1",
    "package2"
]
backend2 = [
    "package3"
]

Unlike the previous example however, maintainers would not necessarily provide a way to do an installation without any extras since it might leave the package in an unusable state.

If packages can support e.g. multiple backends at the same time, and some of the backends should always be installed, then the dependencies for these must be given as required dependencies rather than using the default extras mechanism.

To take one of the concrete examples mentioned in Motivation, the napari package can make use of one of PyQt5, PyQt6, PySide2, or PySide6, and users currently need to explicitly specify napari[all] in order to have one of these be installed, or e.g., napari[pyqt5] to explicitly specify one of the frontend packages. Installing napari with no extras results in a non-functional package. With this PEP, napari could define the following configuration:

[project]
default-optional-dependency-keys = [
    "pyqt5"
]

[project.optional-dependencies]
pyqt5 = [
    "PyQt5",
    "..."
]
pyside2 = [
    "PySide2",
    "..."
]
pyqt6 = [
    "PyQt6",
    "..."
]
pyside6 = [
    "PySide6",
    "..."
]

meaning that:

$ pip install napari

would work out-of-the-box, but there would still be a mechanism for users to explicitly specify a frontend, e.g.:

$ pip install napari[pyside6]

Supporting minimal installations while not always removing default extras

An additional case we consider here is where a package maintainer wants to support minimal installations without any extras, but also wants to support having users specify additional extras without removing the default one. Essentially, they would want:

  • package[minimal] to give an installation without any extras
  • package to install recommended dependencies (in a recommended extras)
  • package[additional] to install both recommended and additional dependencies (in an additional extras)

This could be achieved with e.g:

[project]
default-optional-dependency-keys = [
    "recommended"
]

[project.optional-dependencies]
minimal = []
recommended = [
    "package1",
    "package2"
]
additional = [
    "package[recommended]",
    "package3"
]

The ability for a package to reference itself in the extras is supported by existing Python packaging tools.

Once again considering a concrete example, astropy could with this PEP define a recommended extra (as described in Recommended dependencies and minimal installations). However, it also defines other extras, including for example jupyter, which adds packages that enhance the user experience inside Jupyter-based environments. It is possible that users opting in to this extra would still want recommended dependencies to be installed. In this case, the following configuration would solve this case:

[project]
default-optional-dependency-keys = [
    "recommended"
]

[project.optional-dependencies]
minimal = []
recommended = [
    "scipy",
    "..."
]
jupyter = [
    "astropy[recommended]",
    "ipywidgets",
    "..."
]

Users installing:

$ pip install astropy[jupyter]

would then get the same as:

$ pip install astropy[recommended, jupyter]

Backward Compatibility

Packages not using default extras

Once support for this PEP is added to tools in the packaging ecosystem, packages that do not make use of default extras will continue to work as-is and there should be no break in compatibility.

Packages using default extras

Once packages start defining default extras, those defaults will only be honored with recent versions of packaging tools which implement this PEP, but those packages will remain installable with older packaging tools – with the main difference being that the default extras will not be installed automatically when older packaging tools are used.

As described in How to teach this, package authors need to carefully evaluate when and how they adopt the default extra feature depending on their user base, as some actions (such as moving a required dependency to a default extra) will likely result in breakage for users if a significant fraction of them are still using older package installers that do not support default extras. In this sense, package authors should be aware that this feature, if used in certain ways, can cause backward-compatibility issues for users, and they are thus responsible for ensuring that they minimize the impact to users.

Security Implications

There are no known security implications for this PEP.

How to teach this

This section outlines information that should be made available to people in different groups in the community in relation to the implementation of this PEP. Some aspects described below will be relevant even before the PEP is fully implemented in packaging tools as there are some preparations that can be done in advance of this implementation to facilitate any potential transition later on. The groups covered below are:

Package end users

Package users should be provided with clear installation instructions that show what extras are available for packages and how they behave, for example explaining that by default some recommended dependencies or a given frontend or backend will be installed, and how to opt out of this or override defaults, depending what is available.

Package authors

While the mechanism used to define extras and the associated rule about when to use it are clear, package authors need to carefully consider several points before adopting this capability in their packages, to avoid inadvertently breaking backward-compatibility.

Supporting older versions of package installers

Package installers such as pip or uv will not necessarily implement support for default extras at the same time, and once they do it is likely that package authors will want to keep supporting users who do not have the most recent version of a package installer. In this case, the following recommendations would apply:

  • Moving a package from being a required dependency to a default extra would be a breaking change, because older versions of package installers would not recognise the concept of default extras, and would then install the package with fewer dependencies, which could affect users that would have been relying on these. Therefore, changing dependencies from required to a default extra in an established package should only be done in future once the developers only want to support users with installers that implement this PEP.
  • Making an existing extra become a default should be safer, such as making recommended in astropy be a default extra, but in order to support users with older versions of package installers, the documentation should still mention the extra explicitly as long as possible (until it is clear that most/all users are using package installers that implement this PEP). There is no downside to keeping the extra be explicitly mentioned, but this will ensure that users with modern tooling who do not read documentation (which may be a non-negligeable fraction of the user community) will start getting the recommended dependencies by default.
  • Adding a new extra, whether it be minimal or another new extra that is to be the default, comes with the same caveats that it does prior to this PEP, which is that users will only be able to use this extra for releases that define this extra. This might seem obvious, but consider a package that has a version 1.0 prior to using default extras. Suppose that package now defines minimal in 2.0, then downstream users and packages that want to depend on a minimal version of the package cannot declare the following dependency:
    package[minimal]>=1.0
    

    because package[minimal]==1.0 does not exist (in practice, pip ignores unknown extras, so it might be possible to do this, but there is no guarantee that other tools won’t error on an unrecognized extra).

    The easiest solution to this problem is for package authors to define a no-op minimal extra as soon as possible, even if only planning to adopt default extras further down the road, as it will allow package[minimal] to work for versions prior to when defaults were adopted.

Avoiding the addition of many default dependencies

One temptation for authors might be to include many dependencies by default since they can provide a way to opt out from these. We recommend however that authors carefully consider what is included by default to avoid unecessarily bloating installations and complicating dependency trees. Using default extras does not mean that all extras need to be defaults, and there is still scope for users to explicitly opt in to non-default extras.

Inheriting from default extras

If package authors choose to make an extra be installed by default, it is important that they are aware that if users explicitly specify another extra, the default may not be installed, unless they use the approach described in Supporting minimal installations while not always removing default extras.

There are cases, such as the interchangeable backends, or the minimal extras, where ignoring the default if an extra is explicitly specified is the right thing to do. However, for other cases, such as using default extras to include recommended dependencies while still providing a way to do minimal installs, it may be that many of the other extras should explicitly ‘inherit’ the default extra(s), so package authors should carefully consider in which cases they want the default extras to be installed.

Incompatible extras

In some cases, it may be that packages have extras that are mutually incompatible. In this case, we recommend against using the default extra feature for any extra that contains a dependency that could be incompatible with another.

Consider a package that has extras package[A] and package[B]. Users could already currently try and install package[A] and package[B] or package[A,B] which would result in a broken installation, however it would at least be explicit that both extras were being installed. Making A into a default extra however could lead to unintuitive issues. A user could do:

$ pip install package  # this installs package[A]
$ pip install package[B]

and end up with a broken installation, even though A and B were never explicitly both installed. For this reason, we recommend against using default extras for dependencies where this is likely to be an issue.

Circular dependencies

Authors need to take special care when circular dependencies are present. For instance, consider the following dependency tree:

package1
└── package2
    └── package1

If package1 has a default extra named recommended and a minimal extra which is empty, then:

$ pip install package1[minimal]

will still result in the recommended extra being installed if package2 continues to depend on package1 (with no extras specified). If the dependency tree was updated to instead be:

package1
└── package2
    └── package1[minimal]

Then package1 would no longer be installable with tools that do not yet implement this PEP (if those tools would fail on unrecognized extras). Authors therefore need to carefully consider a migration plan, coordinating with the authors of package2.

Packaging repository maintainers

The impact on individuals who repackage Python libraries for different distributions, such as conda, Homebrew, linux package installers (such as apt and yum) and so on, needs to be considered. Not all package distributions have mechanisms that would line up with the approach described. In fact, some distributions such as conda, do not even have the concept of extras.

There are two cases to consider here:

  • In cases where the repackaging is done by hand, such as for a number of conda-forge recipes, and especially where there is no equivalent to extras, the introduction of default extras should not have a large impact since manual decisions already have to be made as to which dependencies to include (for example, the conda-forge recipe for the astropy package mentioned in the Motivation includes all the recommended dependencies by default since there is no way for users to explicitly request them otherwise).
  • In cases where the repackaging is done in an automated, way, distribution maintainers will need to carefully consider how to treat default extras, and this may imply a non-negligible amount of work and discussion.

It is impossible for a PEP such as this to exhaustively consider each of the different package distributions. However, ultimately, default extras should be understood as how package authors would like their package to be installed for the majority of users, and this should inform decisions about how default extras should be handled, whether manually or automatically.

Reference Implementation

The following repository contains a fully functional demo package that makes use of default extras:

https://github.com/wheel-next/pep_771

This makes use of modified branches of several packages, and the following links are to these branches:

In addition, this branch contains a modified version of the Flit package.

The implementations above are proofs-of-concept at this time and the existing changes have not yet been reviewed by the relevant maintainers. Nevertheless, they are functional enough to allow for interested maintainers to try these out.

Rejected Ideas

Syntax for deselecting extras

One of the main competing approaches was as follows: instead of having defaults be unselected if any extras were explicitly provided, default extras would need to be explicitly unselected.

In this picture, a new syntax for unselecting extras would be introduced as an extension of the mini-language defined in PEP 508. If a package defined default extras, users could opt out of these defaults by using a minus sign (-) before the extra name. The proposed syntax update would have been as follows:

extras_list   = (-)?identifier (wsp* ',' wsp* (-)?identifier)*

Valid examples of this new syntax would have included, e.g.:

  • package[-recommended]
  • package[-backend1, backend2]
  • package[pdf, -svg]

However, there are two main issues with this approach:

  • One would need to define a number of rules for how to interpret corner cases such as if an extra and its negated version were both present in the same dependency specification (e.g. package[pdf, -pdf]) or if a dependency tree included both package[pdf] and package[-pdf], and the rules would not be intuitive to users.
  • More critically, this would introduce new syntax into dependency specification, which means that if any package defined a dependency using the new syntax, it and any other package depending on it would no longer be installable by existing packaging tools, so this would be a major backward compatibility break.

For these reasons, this alternative was not included in the final proposal.

Adding a special entry in extras_require

A potential solution that has been explored as an alternative to introducing the new Default-Extra metadata field would be to make use of an extra with a ‘special’ name.

One example would be to use an empty string:

Provides-Extra:
Requires-Dist: numpy ; extra == ''

The idea would be that dependencies installed as part of the ‘empty’ extras would only get installed if another extra was not specified. An implementation of this was proposed in https://github.com/pypa/setuptools/pull/1503, but it was found that there would be no way to make this work without breaking compatibility with existing usage. For example, packages using Setuptools via a setup.py file can do:

setup(
    ...
    extras_require={'': ['package_a']},
)

which is valid and equivalent to having package_a being defined in install_requires, so changing the meaning of the empty string would break compatibility.

In addition, no other string (such as 'default') can be used as a special string since all strings that would be a backward-compatible valid extras name may already be used in existing packages.

There have been suggestions of using the special None Python variable, but again this is not possible, because even though one can use None in a setup.py file, this is not possible in declarative files such as setup.cfg or pyproject.toml, and furthermore ultimately extras names have to be converted to strings in the package metadata. Having:

Provides-Extra: None

would be indistinguishable from the string ‘None’ which may already be used as an extra name in a Python package. If we were to modify the core metadata syntax to allow non-string ‘special’ extras names, then we would be back to modifying the core metadata specification, which is no better than introducing Default-Extra.

Relying on tooling to deselect any default extras

Another option to unselect extras would be to implement this at the level of packaging tools. For instance, pip could include an option such as:

$ pip install package --no-default-extras

This option could apply to all or specific packages, similar to the --no-binary option, e.g.,:

$ pip install package --no-default-extras :all:

The advantage of this approach is that tools supporting default extras could also support unselecting them. This approach would be similar to the --no-install-recommends option for the apt tool.

However, this solution is not ideal because it would not allow packages to specify themselves that they do not need some of the default extras of a dependency. It would also carry risks for users who might disable all default extras in a big dependency tree, potentially breaking packages in the tree that rely on default extras at any point. Nevertheless, this PEP does not disallow this approach and it is up to the maintainers of different packaging tools to decide if they want to support this kind of option.

package[] disables default extras

Another way to specify not to install any extras, including default extras, would be to use package[]. However, this would break the current assumption in packaging tools that package[] is equivalent to package, and may also result in developers overusing [] by default even when it is not needed. As highlighted in How to teach this, there may also be cases where package maintainers do not actually want to support an installation without any extras, for example in cases where at least one backend or frontend must be installed.

Open issues

Should package[] disable default extras?

Currently, the PEP as written above does not allow package[] to be equivalent to installing the package with no extras, but there would be some benefits to allowing this:

  • It would avoid different packages using different names for a ‘no default’ extras (e.g. minimal, no-default, no-defaults) and reduce the burden for people who don’t want to have to scan through source code or documentation to figure out whether there is the equivalent of a minimal extra.
  • It would allow people who want to use existing packages as-is and future versions of those packages with no default extras to use package[] because that syntax works right now, so it would provide a consistent way over time to get a minimal install.

On the other hand, it is not clear at this point whether any tools are currently relying on package[] being identical to package in a way that would break compatibility if this was done, so this needs to be investigated.


Source: https://github.com/python/peps/blob/main/peps/pep-0771.rst

Last modified: 2025-02-06 17:02:06 GMT