Abstract

This PEP describes a way to record the provenance of installed Python distributions. The record is created by an installer and is available to users in the form of a JSON file provenance_url.json in the .dist-info directory. The mentioned JSON file captures additional metadata to allow recording a URL to a distribution package together with the installed distribution hash. This proposal is built on top of PEP 610 following its corresponding canonical PyPA spec and complements direct_url.json with provenance_url.json for when packages are identified by a name, and optionally a version.

Motivation

Installing a Python Project involves downloading a Distribution Package from a Package Index and extracting its content to an appropriate place. After the installation process is done, information about the release artifact used as well as its source is generally lost. However, there are use cases for keeping records of distributions used for installing packages and their provenance.

Python wheels can be built with different compiler flags or supporting different wheel tags. In both cases, users might get into a situation in which multiple wheels might be considered by installers (possibly from different package indexes) and immediately finding out which wheel file was actually used during the installation might be helpful. This way, developers can use information about wheels to debug issues making sure the desired wheel was actually installed. Another use case could be tools reporting software installed, such as tools reporting a SBOM (Software Bill of Materials), that might give more accurate reports. Yet another use case could be reconstruction of the Python environment by pinning each installed package to a specific distribution artifact consumed from a Python package index.

Rationale

The motivation described in this PEP is an extension of Recording the Direct URL Origin of installed distributions specification. In addition to recording provenance information for packages installed using a direct URL, installers should also do so for packages installed by name (and optionally version) from Python package indexes.

The idea described in this PEP originated in a tool called micropipenv that is used to install distribution packages in containerized environments (see the reported issue thoth-station/micropipenv#206). Currently, the assembled containerized application does not implicitly carry information about the provenance of installed distribution packages (unless these are installed from full URLs and recorded via direct_url.json). This requires container image suppliers to link container images with the corresponding build process, its configuration and the application source code for checking requirements files in cases when software present in containerized environments needs to be audited.

The subsequent discussion in the Discourse thread also brought up pip’s new --report option that can generate a detailed JSON report about the installation process. This option could help with the provenance problem this PEP approaches. Nevertheless, this option needs to be explicitly passed to pip to obtain the provenance information, and includes additional metadata that might not be necessary for checking the provenance (such as Python version requirements of each distribution package). Also, this option is specific to pip as of the writing of this PEP.

Note the current spec for recording installed packages defines a RECORD file that records installed files, but not the distribution artifact from which these files were obtained. Auditing installed artifacts can be performed based on matching the entries listed in the RECORD file. However, this technique requires a pre-computed database of files each artifact provides or a comparison with the actual artifact content. Both approaches are relatively expensive and time consuming operations which could be eliminated with the proposed provenance_url.json file.

Recording provenance information for installed distribution packages, both those obtained from direct URLs and by name/version from an index, can simplify auditing Python environments in general, beyond just the specific use case for containerized applications mentioned earlier. A community project pip-audit raised their possible interest in pypa/pip-audit#170.

Specification

The keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119.

The provenance_url.json file SHOULD be created in the .dist-info directory by installers when installing a Distribution Package specified by name (and optionally by Version Specifier).

This file MUST NOT be created when installing a distribution package from a requirement specifying a direct URL reference (including a VCS URL).

Only one of the files provenance_url.json and direct_url.json (from Recording the Direct URL Origin of installed distributions specification and the corresponding specification of the Direct URL Data Structure), may be present in a given .dist-info directory; installers MUST NOT add both.

The provenance_url.json JSON file MUST be a dictionary, compliant with RFC 8259 and UTF-8 encoded.

If present, it MUST contain exactly two keys. The first MUST be url, with type string. The second key MUST be archive_info with a value defined below.

The value of the url key MUST be the URL from which the distribution package was downloaded. If a wheel is built from a source distribution, the url value MUST be the URL from which the source distribution was downloaded. If a wheel is downloaded and installed directly, the url field MUST be the URL from which the wheel was downloaded. As in the Direct URL Data Structure specification, the url value MUST be stripped of any sensitive authentication information for security reasons.

The user:password section of the URL MAY however be composed of environment variables, matching the following regular expression:

\$\{[A-Za-z0-9-_]+\}(:\$\{[A-Za-z0-9-_]+\})?

Additionally, the user:password section of the URL MAY be a well-known, non-security sensitive string. A typical example is git in the case of an URL such as ssh://git@gitlab.com.

The value of archive_info MUST be a dictionary with a single key hashes. The value of hashes is a dictionary mapping hash function names to a hex-encoded digest of the file referenced by the url value. At least one hash MUST be recorded. Multiple hashes MAY be included, and it is up to the consumer to decide what to do with multiple hashes (it may validate all of them or a subset of them, or nothing at all).

Each hash MUST be one of the single argument hashes provided by hashlib.algorithms_guaranteed, excluding sha1 and md5 which MUST NOT be used. As of Python 3.11, with shake_128 and shake_256 excluded for being multi-argument, the allowed set of hashes is:

>>> import hashlib
>>> sorted(hashlib.algorithms_guaranteed - {"shake_128", "shake_256", "sha1", "md5"})
['blake2b', 'blake2s', 'sha224', 'sha256', 'sha384', 'sha3_224', 'sha3_256', 'sha3_384', 'sha3_512', 'sha512']

Each hash MUST be referenced by the canonical name of the hash, always lower case.

Hashes sha1 and md5 MUST NOT be present, due to the security limitations of these hash algorithms. Conversely, hash sha256 SHOULD be included.

Installers that cache distribution packages from an index SHOULD keep information related to the cached distribution artifact, so that the provenance_url.json file can be created even when installing distribution packages from the installer’s cache.

Backwards Compatibility

Following the Recording installed projects specification, installers may keep additional installer-specific files in the .dist-info directory. To make sure this PEP does not cause any backwards compatibility issues, a comprehensive survey of installers and libraries found no current tools that are using a similarly-named file, or other major feasibility concerns.

The Wheel specification lists files that can be present in the .dist-info directory. None of these file names collide with the proposed provenance_url.json file from this PEP.

Security Implications

One of the main security features of the provenance_url.json file is the ability to audit installed artifacts in Python environments. Tools can check which Python package indexes were used to install Python distribution packages as well as the hash digests of their release artifacts.

As an example, we can take the recent compromised dependency chain in the PyTorch incident. The PyTorch index provided a package named torchtriton. An attacker published torchtriton on PyPI, which ran a malicious binary. By checking the URL of the installed Python distribution stated in the provenance_url.json file, tools can automatically check the source of the installed Python distribution. In case of the PyTorch incident, the URL of torchtriton should point to the PyTorch index, not PyPI. Tools can help identifying such malicious Python distributions installed by checking the installed Python distribution URL. A more exact check can include also the hash of the installed Python distribution stated in the provenance_url.json file. Such checks on hashes can be helpful for mirrored Python package indexes where Python distributions are not distinguishable by their source URLs, making sure only desired Python package distributions are installed.

A malicious actor can intentionally adjust the content of provenance_url.json to possibly hide provenance information of the installed Python distribution. A security check which would uncover such malicious activity is beyond scope of this PEP as it would require monitoring actions on the filesystem and eventually reviewing user or file permissions.

How to Teach This

The provenance_url.json metadata file is intended for tools and is not directly visible to end users.

Examples

{
  "archive_info": {
    "hashes": {
      "blake2s": "fffeaf3d0bd71dc960ca2113af890a2f2198f2466f8cd58ce4b77c1fc54601ff",
      "sha256": "236bcb61156d76c4b8a05821b988c7b8c35bf0da28a4b614e8d6ab5212c25c6f",
      "sha3_256": "c856930e0f707266d30e5b48c667a843d45e79bb30473c464e92dfa158285eab",
      "sha512": "6bad5536c30a0b2d5905318a1592948929fbac9baf3bcf2e7faeaf90f445f82bc2b656d0a89070d8a6a9395761f4793c83187bd640c64b2656a112b5be41f73d"
    }
  },
  "url": "https://files.pythonhosted.org/packages/07/51/2c0959c5adf988c44d9e1e0d940f5b074516ecc87e96b1af25f59de9ba38/pip-23.0.1-py3-none-any.whl"
}

{
  "archive_info": {
    "hashes": {
      "sha256": "236bcb61156d76c4b8a05821b988c7b8c35bf0da28a4b614e8d6ab5212c25c6f"
    }
  },
  "url": "https://files.pythonhosted.org/packages/07/51/2c0959c5adf988c44d9e1e0d940f5b074516ecc87e96b1af25f59de9ba38/pip-23.0.1-py3-none-any.whl"
}

{
  "archive_info": {
    "hashes": {
      "sha256": "8bfe29f17c10e2f2e619de8033a07a224058d96b3bfe2ed61777596f7ffd7fa9"
    }
  },
  "url": "https://files.pythonhosted.org/packages/1d/43/ad8ae671de795ec2eafd86515ef9842ab68455009d864c058d0c3dcf680d/micropipenv-0.0.1.tar.gz"
}

{
  "archive_info": {
    "hash": "sha256=236bcb61156d76c4b8a05821b988c7b8c35bf0da28a4b614e8d6ab5212c25c6f",
    "hashes": {
      "sha256": "236bcb61156d76c4b8a05821b988c7b8c35bf0da28a4b614e8d6ab5212c25c6f"
    }
  },
  "url": "https://files.pythonhosted.org/packages/07/51/2c0959c5adf988c44d9e1e0d940f5b074516ecc87e96b1af25f59de9ba38/pip-23.0.1-py3-none-any.whl"
}

{
  "archive_info": {
    "hashes": {
      "SHA-256": "236bcb61156d76c4b8a05821b988c7b8c35bf0da28a4b614e8d6ab5212c25c6f"
    }
  },
  "url": "https://files.pythonhosted.org/packages/07/51/2c0959c5adf988c44d9e1e0d940f5b074516ecc87e96b1af25f59de9ba38/pip-23.0.1-py3-none-any.whl"
}

{
  "archive_info": {
    "hashes": {}
   }
  "url": "https://files.pythonhosted.org/packages/07/51/2c0959c5adf988c44d9e1e0d940f5b074516ecc87e96b1af25f59de9ba38/pip-23.0.1-py3-none-any.whl"
}

Reference Implementation

A proof-of-concept for creating the provenance_url.json metadata file when installing a Python Distribution Package is available in the PR to pip pypa/pip#11865. It reuses the already available implementation for the direct URL data structure to provide the provenance_url.json metadata file for cases when direct_url.json is not created.

A reference implementation for supporting the provenance_url.json file in PDM exists is available in pdm-project/pdm#3013.

A prototype called pip-preserve was developed to demonstrate creation of requirements.txt files considering direct_url.json and provenance_url.json metadata files. This tool mimics the pip freeze functionality, but the listing of installed packages also includes the hashes of the Python distribution artifacts.

To further support this proposal, pip-sbom demonstrates creation of SBOM in the SPDX format. The tool uses information stored in the provenance_url.json file.

Rejected Ideas

Open Issues

Appendix: Survey of installers and libraries

Acknowledgements

Thanks to Dustin Ingram, Brett Cannon, and Paul Moore for the initial discussion in which this idea originated.

Thanks to Donald Stufft, Ofek Lev, and Trishank Kuppusamy for early feedback and support to work on this PEP.

Thanks to Gregory P. Smith, Stéphane Bidoul, and C.A.M. Gerlach for reviewing this PEP and providing valuable suggestions.

Thanks to Seth Michael Larson for providing valuable suggestions and for the proposed pip-sbom prototype.

Thanks to Stéphane Bidoul and Chris Jerdonek for PEP 610, and related Recording the Direct URL Origin of installed distributions and Direct URL Data Structure specifications.

Thanks to Frost Ming for raising possible concern around storing index URL in the provenance_url.json file and initial PEP 710 support in PDM.

Last, but not least, thanks to Donald Stufft for sponsoring this PEP.

Copyright

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.