Measurability and Phantom Dependencies

Python packages are particularly affected by the “phantom dependency” problem, where software components that aren’t written in Python are included in Python packages for many reasons, such as ease of installation and compatibility with standards:

• Python serves scientific, data, web, and machine-learning use-cases which use compiled or non-Python languages like Rust, C, C++, Fortran, JavaScript, and others.
• The Python wheel format is preferred by users due to the ease-of-installation. No code is executed during the installation step, only extracting the archive.
• The Python wheel format requires bundling shared compiled libraries without a method to encode metadata about these libraries.
• Packages related to Python packaging sometimes need to solve the “bootstrapping” problem, so include pure Python projects inside their source code.

These software components can’t be described using Python package metadata and thus are likely to be missed by software composition analysis (SCA) software which can mean vulnerable software components aren’t reported accurately.

For example, the Python package Pillow includes 16 shared object libraries in the wheel that were bundled by auditwheel as a part of the build. None of those shared object libraries are detected when using common SCA tools like Syft and Grype. If an SBOM document is included annotating all the included shared libraries then SCA tools can identify the included software reliably.

Regulations

SBOMs are required by recent software security regulations, like the Secure Software Development Framework (SSDF) and the Cyber Resilience Act (CRA). Due to their inclusion in these regulations, the demand for SBOM documents of open source projects is expected to be high. One goal is to minimize the demands on open source project maintainers by enabling open source users that need SBOMs to self-serve using existing tooling.

Another goal is to enable contributions from users who need SBOMs to annotate projects they depend on with SBOM information. Today there is no mechanism to propagate the results of those contributions for a Python package so there is no incentive for users to contribute this type of work.

Using SBOM standards instead of Core Metadata fields

Attempting to add every field offered by SBOM standards into Python package Core Metadata would result in an explosion of new Core Metadata fields, including the need to keep up-to-date as SBOM standards continue to evolve to suit new needs in that space.

Instead, this proposal delegates SBOM-specific metadata to SBOM documents that are included in Python packages and adds a new Core Metadata field for discoverability of included SBOM documents.

This standard also doesn’t aim to replace Core Metadata with SBOMs, instead focusing on the SBOM information being supplemental to Core Metadata. Included SBOMs only contain information about dependencies included in the package archive or information about the top-level software in the package that can’t be encoded into Core Metadata but is relevant for the SBOM use-case (“software identifiers”, “purpose”, “support level”, etc).

Zero-or-more SBOM documents per package

Rather than requiring at most one included SBOM document per Python package, this PEP proposes that one or more SBOM documents may be included in a Python package. This means that code attempting to annotate a Python package with SBOM data may do so without being concerned about corrupting data already contained within other SBOM documents.

This decision also means this PEP is capable of supporting multiple SBOM standards without favoring one, instead deferring support to SBOM consuming tools. This decision also means that the merging of SBOM documents never needs to be handled by Python tools, instead deferring this to SBOM consuming tools who are better placed to handle cross-standard conversions.

Core Metadata

Project source metadata

This PEP specifies changes to the project’s source metadata under a [project] table in the pyproject.toml file.

[project]
sbom-files = ["bom.json"]

[project]
sbom-files = ["sboms/openssl.cdx.json", "licenses/openssl.spdx.json"]

[project]
sbom-files = ["sboms/*"]

[project]
sbom-files = []

[project]
sbom-files = ["..\bom.json"]

[project]
sbom-files = ["bom{.json*"]

SBOM files in project formats

A few additions will be made to the existing specifications.

Project source trees

Per Project source metadata section, the Declaring Project Metadata specification will be updated to reflect that SBOM file paths MUST be relative to the project root directory; i.e. the directory containing the pyproject.toml (or equivalently, other legacy project configuration, e.g. setup.py, setup.cfg, etc).

Source distributions (sdists)

The sdist specification will be updated to reflect that if the Metadata-Version is 2.5 or greater, the sdist MUST contain any SBOM files specified by the Sbom-File field in the PKG-INFO at their respective paths relative to the sdist (containing the pyproject.toml and the PKG-INFO Core Metadata).

Built distributions (wheels)

The wheel specification will be updated to reflect that if the Metadata-Version is 2.5 or greater and one or more Sbom-File fields are specified, the .dist-info directory MUST contain an sboms subdirectory, which MUST contain the files listed in the Sbom-File fields in the METADATA file at their respective paths relative to the sboms directory.

Installed projects

The Recording Installed Projects specification will be updated to reflect that if the Metadata-Version is 2.5 or greater and one or more Sbom-File fields is specified, the .dist-info directory MUST contain an sboms subdirectory which MUST contain the files listed in the Sbom-File fields in the METADATA file at their respective paths relative to the sboms directory, and that any files in this directory MUST be copied from wheels by install tools.

Why not require a single SBOM standard?

Most discussion and development around SBOMs today focuses on two SBOM standards: CycloneDX and SPDX. There is no clear “winner” between these two standards, both standards are frequently used by projects and software ecosystems.

Because both standards are frequently used, tools for consuming and processing SBOM documents commonly need to support both standards. This means that this PEP is not constrained to select a single SBOM standard by its consumers and thus can allow tools creating SBOM documents for inclusion in Python packages to choose which SBOM standard works best for their use-case.

Conditional project source SBOM files

How can a project specify an SBOM file that is conditional? Under what circumstances would an SBOM document be conditional?