Following system colour scheme Selected dark colour scheme Selected light colour scheme

Python Enhancement Proposals

PEP 632 – Deprecate distutils module

Author:
Steve Dower <steve.dower at python.org>
Discussions-To:
Discourse thread
Status:
Final
Type:
Standards Track
Created:
03-Sep-2020
Python-Version:
3.10
Post-History:
03-Sep-2020, 22-Jan-2021
Resolution:
Python-Dev thread
Table of Contents

Abstract

The distutils module [1] has for a long time recommended using the setuptools package [2] instead. Setuptools has recently integrated a complete copy of distutils and is no longer dependent on the standard library [3]. Pip has been silently replacing distutils with setuptools when installing packages for a long time already, and the distutils documentation has stated that it is being phased out since 2014 (or earlier). It is time to remove it from the standard library.

Motivation

distutils [1] is a largely undocumented and unmaintained collection of utilities for packaging and distributing Python packages, including compilation of native extension modules. It defines a configuration format that describes a Python distribution and provides the tools to convert a directory of source code into a source distribution, and some forms of binary distribution. Because of its place in the standard library, many updates can only be released with a major release, and users cannot rely on particular fixes being available.

setuptools [2] is a better documented and well maintained enhancement based on distutils. While it provides very similar functionality, it is much better able to support users on earlier Python releases, and can respond to bug reports more quickly. A number of platform-specific enhancements already exist in setuptools that have not been added to distutils, and there is been a long-standing recommendation in the distutils documentation to prefer setuptools.

Historically, setuptools has extended distutils using subclassing and monkeypatching, but has now taken a copy of the underlying code. [3] As a result, the second last major dependency on distutils is gone and there is no need to keep it in the standard library.

The final dependency on distutils is CPython itself, which uses it to build native extension modules in the standard library (except on Windows). Because this is a CPython build-time dependency, it is possible to continue to use distutils for this specific case without it being part of the standard library.

Deprecation and removal will make it obvious that issues should be fixed in the setuptools project, and will reduce a source of bug reports and unnecessary test maintenance. It will also help promote the development of alternative build backends, which can now be supported more easily thanks to PEP 517.

Specification

In Python 3.10 and 3.11, distutils will be formally marked as deprecated. All known issues will be closed at this time. import distutils will raise a deprecation warning. New issues that would be considered release blocking may still be fixed, but support for new tools or platforms will not be added.

During Python 3.10 and 3.11, uses of distutils within the standard library may change to use alternative APIs.

In Python 3.12, distutils will no longer be installed by make install or any of the first-party distribution. Third-party redistributors should no longer include distutils in their bundles or repositories.

This PEP makes no specification on migrating the parts of the CPython build process that currently use distutils. Depending on contributions, this migration may occur at any time.

After Python 3.12 is started and when the CPython build process no longer depends on distutils being in the standard library, the entire Lib/distutils directory and Lib/test/test_distutils.py file will be removed from the repository.

Other references to distutils will be cleaned up. As of Python 3.9’s initial release, the following modules have references in code or comments:

  • Lib/ctypes/util.py
  • Lib/site.py
  • Lib/sysconfig.py
  • Lib/_aix_support.py
  • Lib/_bootsubprocess.py
  • Lib/_osx_support.py
  • Modules/_decimal/tests/formathelper.py

The following Tools in CPython also refer to distutils. Note that none of these are installed with CPython:

  • PC/layout (references will be removed)
  • Tools/msi (references will be removed)
  • Tools/peg_generator (will be adapted to a different build tool)
  • Tools/test2to3 (example project will be removed)

As the distutils code is already included in setuptools, there is no need to republish it in any other form. Those who require access to the functionality should use setuptools or an alternative build backend.

Backwards Compatibility

Code that imports distutils will no longer work from Python 3.12.

The suggested migration path is to use the equivalent (though not identical) imports from setuptools (see [5]), or to migrate to an alternative build backend (see PEP 517).

Code already exists in setuptools to transparently switch setup.py files using distutils onto their equivalents, and so most working build scripts are already known to work with setuptools. Such scripts may need to update their import statements. Consult the setuptools documentation for specific migration advice. [5]

Some projects use alternate sets of patches over distutils, notably, numpy.distutils. [6] Projects that we know are doing this have been informed.

Many build scripts use custom commands or narrowly scoped patches. As these packages are already subject to setuptools overriding distutils, we expect minimal disruption as a result of distutils being removed. Scripts may still need to be updated to avoid importing distutils.

Reference Implementation

setuptools version 48 includes the complete copy of distutils, and as such is no longer dependent on the standard library’s copy. Most implementation issues they have faced are due to the continuing existence of distutils in the standard library, and so removal will improve the stability of their implementation.

There is not yet a reference implementation for the removal of distutils from the standard library, nor is there an implementation for CPython’s native module builds without relying on the standard library copy of distutils.

Migration Advice

Note

This section suggests some alternative replacements for popular functionality that is being formally deprecated with this PEP. It is current at time of writing, but is not kept up to date.

For these modules or types, setuptools is the best substitute:

  • distutils.ccompiler
  • distutils.cmd.Command
  • distutils.command
  • distutils.config
  • distutils.core.Distribution
  • distutils.errors

For these modules or types, use the standards-defined Python Packaging Authority packages specified:

  • distutils.version — use the packaging package

For these modules or functions, use the standard library module shown:

  • distutils.fancy_getopt — use the argparse module
  • distutils.spawn.find_executable — use the shutil.which function
  • distutils.spawn.spawn — use the subprocess.run function
  • distutils.sysconfig — use the sysconfig module
  • distutils.util.get_platform — use the platform module

For these functions, and any others not mentioned here, you will need to reimplement the functionality yourself. The legacy documentation can be found at https://docs.python.org/3.9/distutils/apiref.html

  • distutils.dir_util.create_tree
  • distutils.util.change_root
  • distutils.util.strtobool

Rejected Ideas

Deprecate but do not delete

The primary concern with this approach is that distutils most frequently breaks because of platform differences, which means that without maintenance, it will stop working out-of-sync with any Python release. This makes it impossible for libraries to reliably detect when they will stop working.

In contrast, this PEP proposes a concrete date, known well in advance, when distutils will stop working, and commits to not breaking the API before that time. This gives maintainers a predictable schedule, ensures any breakage occurs at a point where users will already be expecting changed behavior, and provides a reliable detection mechanism (specifically, that import distutils raises).

Finally, as long as distutils remains in the standard library in any form, it will interfere with third-party packages that provide shims or replacements, including setuptools. Completely removing the package at a known version makes it possible for third-parties to safely use a substitute.

Only deprecate the setuptools-like functionality

This suggestion assumes that there exists a volunteer to maintain whatever is left, which is not true. It also implies that anybody knows which functionality should remain, which as seen in the discussions is not at all clear.

Most helper functions in distutils already have supported (and improved) alternatives, often in the standard library, and there is little that can be done to the legacy versions without breaking compatibility. (And any break requiring maintainers to update their code is essentially equivalent to requiring them to import a different function.)

The last point from the previous section also applies here.

References

Source: https://github.com/python/peps/blob/main/peps/pep-0632.rst

Last modified: 2023-10-10 11:05:14 GMT