File Name

A lock file MUST be named pylock.toml. The use of the .toml file extension is to make syntax highlighting in editors easier and to reinforce the fact that the file format is meant to be human-readable.

The lock file SHOULD be located in the directory as appropriate for the scope of the lock file. Locking against a single pyproject.toml, for instance, would place the pylock.toml in the same directory. If the lock file covered multiple projects in a monorepo, then the expectation is the pylock.toml file would be in the directory that held all the projects being locked.

File Format

The format of the file is TOML.

All keys listed below are required unless otherwise noted. If two keys are mutually exclusive to one another, then one of the keys is required while the other is disallowed.

Keys in tables – including the top-level table – SHOULD be emitted by lockers in the order they are listed in this PEP when applicable unless another sort order is specified to minimize noise in diffs. If the keys are not explicitly specified in this PEP, then the keys SHOULD be sorted by lexicographic order.

As well, lockers SHOULD sort arrays in lexicographic order unless otherwise specified for the same reason.

Examples

version = '1.0'
hash-algorithm = 'sha256'

[locker]
name = 'mousebender'
version = 'pep'
run = { module = 'mousebender', args = ['lock', '--platform', 'cpython3.12-manylinux2014-x64', '--platform', 'cpython3.12-windows-x64', 'cattrs', 'numpy'] }

[[groups]]
name = 'Default'
requirements = [
  { name = 'cattrs' },
  { name = 'numpy' },
]

[[packages]]
name = 'attrs'
version = '24.2.0'
groups = ['Default']
index_url = 'https://pypi.org/simple/attrs'
direct = false
requires_python = '>=3.7'
dependencies = [
  { name = 'importlib-metadata', marker = 'python_version < "3.8"' },
  { name = 'cloudpickle', marker = 'platform_python_implementation == "CPython"', feature = 'benchmark' },
  { name = 'hypothesis', feature = 'benchmark' },
  { name = 'mypy', version = '>=1.11.1', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9"', feature = 'benchmark' },
  { name = 'pympler', feature = 'benchmark' },
  { name = 'pytest-codspeed', feature = 'benchmark' },
  { name = 'pytest-mypy-plugins', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9" and python_version < "3.13"', feature = 'benchmark' },
  { name = 'pytest-xdist', extras = ['psutil'], feature = 'benchmark' },
  { name = 'pytest', version = '>=4.3.0', feature = 'benchmark' },
  { name = 'cloudpickle', marker = 'platform_python_implementation == "CPython"', feature = 'cov' },
  { name = 'coverage', extras = ['toml'], version = '>=5.3', feature = 'cov' },
  { name = 'hypothesis', feature = 'cov' },
  { name = 'mypy', version = '>=1.11.1', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9"', feature = 'cov' },
  { name = 'pympler', feature = 'cov' },
  { name = 'pytest-mypy-plugins', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9" and python_version < "3.13"', feature = 'cov' },
  { name = 'pytest-xdist', extras = ['psutil'], feature = 'cov' },
  { name = 'pytest', version = '>=4.3.0', feature = 'cov' },
  { name = 'cloudpickle', marker = 'platform_python_implementation == "CPython"', feature = 'dev' },
  { name = 'hypothesis', feature = 'dev' },
  { name = 'mypy', version = '>=1.11.1', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9"', feature = 'dev' },
  { name = 'pre-commit', feature = 'dev' },
  { name = 'pympler', feature = 'dev' },
  { name = 'pytest-mypy-plugins', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9" and python_version < "3.13"', feature = 'dev' },
  { name = 'pytest-xdist', extras = ['psutil'], feature = 'dev' },
  { name = 'pytest', version = '>=4.3.0', feature = 'dev' },
  { name = 'cogapp', feature = 'docs' },
  { name = 'furo', feature = 'docs' },
  { name = 'myst-parser', feature = 'docs' },
  { name = 'sphinx', feature = 'docs' },
  { name = 'sphinx-notfound-page', feature = 'docs' },
  { name = 'sphinxcontrib-towncrier', feature = 'docs' },
  { name = 'towncrier', version = '<24.7', feature = 'docs' },
  { name = 'cloudpickle', marker = 'platform_python_implementation == "CPython"', feature = 'tests' },
  { name = 'hypothesis', feature = 'tests' },
  { name = 'mypy', version = '>=1.11.1', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9"', feature = 'tests' },
  { name = 'pympler', feature = 'tests' },
  { name = 'pytest-mypy-plugins', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9" and python_version < "3.13"', feature = 'tests' },
  { name = 'pytest-xdist', extras = ['psutil'], feature = 'tests' },
  { name = 'pytest', version = '>=4.3.0', feature = 'tests' },
  { name = 'mypy', version = '>=1.11.1', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9"', feature = 'tests-mypy' },
  { name = 'pytest-mypy-plugins', marker = 'platform_python_implementation == "CPython" and python_version >= "3.9" and python_version < "3.13"', feature = 'tests-mypy' }
]
editable = false
wheels = [
  { tags = ['py3-none-any'], url = 'https://files.pythonhosted.org/packages/6a/21/5b6702a7f963e95456c0de2d495f67bf5fd62840ac655dc451586d23d39a/attrs-24.2.0-py3-none-any.whl', hash = '81921eb96de3191c8258c199618104dd27ac608d9366f5e35d011eae1867ede2', upload_time = 2024-08-06T14:37:36.958006+00:00, size = 63001 }
]

[[packages]]
name = 'cattrs'
version = '24.1.2'
groups = ['Default']
index_url = 'https://pypi.org/simple/cattrs'
direct = false
requires_python = '>=3.8'
dependencies = [
  { name = 'attrs', version = '>=23.1.0' },
  { name = 'exceptiongroup', version = '>=1.1.1', marker = 'python_version < "3.11"' },
  { name = 'typing-extensions', version = '!=4.6.3,>=4.1.0', marker = 'python_version < "3.11"' },
  { name = 'pymongo', version = '>=4.4.0', feature = 'bson' },
  { name = 'cbor2', version = '>=5.4.6', feature = 'cbor2' },
  { name = 'msgpack', version = '>=1.0.5', feature = 'msgpack' },
  { name = 'msgspec', version = '>=0.18.5', marker = 'implementation_name == "cpython"', feature = 'msgspec' },
  { name = 'orjson', version = '>=3.9.2', marker = 'implementation_name == "cpython"', feature = 'orjson' },
  { name = 'pyyaml', version = '>=6.0', feature = 'pyyaml' },
  { name = 'tomlkit', version = '>=0.11.8', feature = 'tomlkit' },
  { name = 'ujson', version = '>=5.7.0', feature = 'ujson' }
]
editable = false
wheels = [
  { tags = ['py3-none-any'], url = 'https://files.pythonhosted.org/packages/c8/d5/867e75361fc45f6de75fe277dd085627a9db5ebb511a87f27dc1396b5351/cattrs-24.1.2-py3-none-any.whl', hash = '67c7495b760168d931a10233f979b28dc04daf853b30752246f4f8471c6d68d0', upload_time = 2024-09-22T14:58:34.812643+00:00, size = 66446 }
]

[[packages]]
name = 'numpy'
version = '2.1.2'
groups = ['Default']
index_url = 'https://pypi.org/simple/numpy'
direct = false
requires_python = '>=3.10'
dependencies = [

]
editable = false
wheels = [
  { tags = ['cp312-cp312-manylinux2014_x86_64', 'cp312-cp312-manylinux_2_17_x86_64'], url = 'https://files.pythonhosted.org/packages/9b/b4/e3c7e6fab0f77fff6194afa173d1f2342073d91b1d3b4b30b17c3fb4407a/numpy-2.1.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl', hash = '6d95f286b8244b3649b477ac066c6906fbb2905f8ac19b170e2175d3d799f4df', upload_time = 2024-10-05T18:36:20.729642+00:00, size = 16041825 },
  { tags = ['cp312-cp312-win_amd64'], url = 'https://files.pythonhosted.org/packages/4c/79/73735a6a5dad6059c085f240a4e74c9270feccd2bc66e4d31b5ca01d329c/numpy-2.1.2-cp312-cp312-win_amd64.whl', hash = '456e3b11cb79ac9946c822a56346ec80275eaf2950314b249b512896c0d2505e', upload_time = 2024-10-05T18:37:38.159022+00:00, size = 12568254 }
]

Expectations for Lockers

• Lockers MUST make sure that entering the dependency graph via a specific group will not lead to ambiguity for installers as to which value in [[packages]] to install for any environment (this can be controlled for via packages.version and packages.groups).
• Lockers SHOULD try to make all logically related groups resolve together (i.e. no ambiguity if grouped together).
• If a groups.project would have extras that cause ambiguity or installation failure due to conflicts between the extras, the locker MAY create separate groups.requirements entries instead, otherwise the locker MUST raise an error.
• Lockers MAY try to lock for multiple environments in a single lock file.
• Lockers MAY try to update a lock file containing [tool] and [packages.tool] for other tools than themselves.
• Lockers MAY want to provide a way to let users provide the information necessary to lock for other environments, e.g., supporting a JSON file format which specifies wheel tags and marker values.

{
    "marker-values": {"<marker>": "<value>"},
    "wheel-tags": ["<tag>"]
}

Expectations for Installers

• Installers MAY support installation of non-binary files (i.e. source trees and source distributions), but are not required to.
• Installers MUST provide a way to avoid non-binary file installation for reproducibility and security purposes.
• Installers SHOULD make it opt-in to use non-binary file installation to facilitate a secure-by-default approach.
• If a traversal of the graph leads to any ambiguity as to what package version to install (i.e. more than one package version qualifies), an error MUST be raised.
• Installers MUST only consider package versions included in any selected groups (i.e. installers cannot consider packages outside of the groups selected to install from).
• Installers MUST error out if a package version lacks a way to install into the chosen environment.
• Installers MUST support installing into an empty environment.

class UnsatisfiableError(Exception):
    """Raised when a requirement cannot be satisfied."""


class AmbiguityError(Exception):
    """Raised when a requirement has multiple solutions."""


def install_packages(lock_file_contents):
    # Hard-coded out of laziness.
    packages = choose_packages(lock_file_contents, (GROUP_NAME, frozenset()))

    for package in packages:
        tags = list(packaging.tags.sys_tags())
        for tag in tags:  # Prioritize by tag order.
            tag_str = str(tag)
            for wheel in package["wheels"]:
                if tag_str in wheel["tags"]:
                    break
            else:
                continue
            break
        else:
            raise UnsatisfiableError(
                f"No wheel for {package['name']} {package['version']}"
            )
        print(f"Installing {package['name']} {package['version']} ({tag_str})")


def choose_packages(lock_file_data, *selected_groups):
    """Select the package versions that should be installed based on the requested groups.

    'selected_groups' is a sequence of two-item tuples, representing a group name and
    optionally any requested extras if the group is a project.
    """
    group_names = frozenset(operator.itemgetter(0)(group) for group in selected_groups)
    available_packages = {}  # The packages in the selected groups.
    for pkg in lock_file_data["packages"]:
        if frozenset(pkg["groups"]) & group_names:
            available_packages.setdefault(pkg["name"], []).append(pkg)
    selected_packages = {}  # The package versions that have been selected.
    handled_extras = {}  # The extras that have been handled.
    requirements = []  # A stack of requirements to satisfy.

    # First, get our starting list of requirements.
    for group in selected_groups:
        requirements.extend(gather_requirements(lock_file_data, group))

    # Next, go through the requirements and try to find a **single** package version
    # that satisfies each requirement.
    while requirements:
        req = requirements.pop()
        # Ignore requirements whose markers disqualify it.
        if not applies_to_env(req):
            continue
        name = req["name"]
        if pkg := selected_packages.get(name):
            # Safety check that the cross-section of groups doesn't cause issues.
            # It somewhat assumes the locker didn't mess up such that there would be
            # ambiguity by what package version was initially selected.
            if not version_satisfies(req, pkg):
                raise UnsatisfiableError(
                    f"requirement {req!r} not satisfied by "
                    f"{selected_packages[req['name']]!r}"
                )
            if "extras" not in req:
                continue
            needed_extras = req["extras"]
            if not (extras := handled_extras.set_default(name, set())).difference(
                needed_extras
            ):
                continue
            # This isn't optimal as we may tread over the same extras multiple times,
            # but eventually the maximum set of extras for the package will be handled
            # and thus the above guard will short-circuit adding any more requirements.
            extras.update(needed_extras)
        else:
            # Raises UnsatisfiableError or AmbiguityError if no suitable, single package
            # version is found.
            pkg = compatible_package_version(req, available_packages[req["name"]])
            selected_packages[name] = pkg
        requirements.extend(dependencies(pkg, req))

    return selected_packages.values()


def gather_requirements(locked_file_data, group):
    """Return a collection of all requirements for a group."""
    # Hard-coded to support `groups.requirements` out of laziness.
    group_name, _extras = group
    for group in locked_file_data["groups"]:
        if group["name"] == group_name:
            return group["requirements"]
    else:
        raise ValueError(f"Group {group_name!r} not found in lock file")


def applies_to_env(requirement):
    """Check if the requirement applies to the current environment."""
    try:
        markers = requirement["marker"]
    except KeyError:
        return True
    else:
        return packaging.markers.Marker(markers).evaluate()


def version_satisfies(requirement, package):
    """Check if the package version satisfies the requirement."""
    try:
        raw_specifier = requirement["version"]
    except KeyError:
        return True
    else:
        specifier = packaging.specifiers.SpecifierSet(raw_specifier)
        return specifier.contains(package["version"], prereleases=True)


def compatible_package_version(requirement, available_packages):
    """Return the package version that satisfies the requirement.

    If no package version can satisfy the requirement, raise UnsatisfiableError. If
    multiple package versions can satisfy the requirement, raise AmbiguityError.
    """
    possible_packages = [
        pkg for pkg in available_packages if version_satisfies(requirement, pkg)
    ]
    if not possible_packages:
        raise UnsatisfiableError(f"No package version satisfies {requirement!r}")
    elif len(possible_packages) > 1:
        raise AmbiguityError(f"Multiple package versions satisfy {requirement!r}")
    return possible_packages[0]


def dependencies(package, requirement):
    """Return the dependencies of the package.

    The extras from the requirement will extend the base requirements as needed.
    """
    applicable_deps = []
    extras = frozenset(requirement.get("extras", []))
    for dep in package["dependencies"]:
        if "feature" not in dep or dep["feature"] in extras:
            applicable_deps.append(dep)
    return applicable_deps

A flat set of packages to install

An earlier version of this PEP proposed to use a flat set of package versions instead of a graph. The idea was that each package version could be evaluated in isolation as to whether it applied to an environment for installation. The hope was that would lend itself to easier auditing as one wouldn’t have to worry about how a package version fit into the graph when looking at e.g., a diff for a lock file.

Unfortunately this was deemed not as flexible as using a graph. For instance, recording the graph assists in dependency analysis for tools like GitHub. A graph also makes following how you ended up with dependencies within your lock file from any point in the graph. It also balances out the implementation costs a bit more between lockers and installers by alleviating the complexity off of lockers a bit for only a minor increase in complexity for installers by involving standard graph-traversing algorithms instead of a linear walk.

And if the dependency graph is already being recorded for the above benefits, then recording that same data in a flattened manner is redundant that makes lock files larger and potentially more unruly.

Specifying a new core metadata version that requires consistent metadata across files

At one point, to handle the issue of metadata varying between files and thus require examining every released file for a package and version for accurate locking results, the idea was floated to introduce a new core metadata version which would require all metadata for all wheel files be the same for a single version of a packages. Ultimately, though, it was deemed unnecessary as this PEP will put pressure on people to make files consistent for performance reasons or to make indexes provide all the metadata separate from the wheel files themselves. As well, there’s no easy enforcement mechanism, and so community expectation would work as well as a new metadata version.

Have the installer do dependency resolution

In order to support a format more akin to how Poetry worked when this PEP was drafted, it was suggested that lockers effectively record the packages and their versions which may be necessary to make an install work in any possible scenario, and then the installer resolves what to install. But that complicates auditing a lock file by requiring much more mental effort to know what packages may be installed in any given scenario. Also, one of the Poetry developers suggested that markers as represented in the package locking approach of this PEP may be sufficient to cover the needs of Poetry. Not having the installer do a resolution also simplifies their implementation, centralizing complexity in lockers.

Requiring specific hash algorithm support

It was proposed to require a baseline hash algorithm for the files. This was rejected as no other Python packaging specification requires specific hash algorithm support. As well, the minimum hash algorithm suggested may eventually become an outdated/unsafe suggestion, requiring further updates. In order to promote using the best algorithm at all times, no baseline is provided to avoid simply defaulting to the baseline in tools without considering the security ramifications of that hash algorithm.

Require a URL or file path for files

Originally references to files were required, e.g., packages.sdist.url or packages.sdist.path. But at least one use-case surfaced during discussions about this PEP where statically specifying the location of files would be problematic. And in earlier discussions the idea of the location being a hint wasn’t preferred. Hence the PEP now makes the data optional, but considers the locations accurate if specified.

File naming

File format

Other keys

Specify requires-python at the file level?

The lock file formats from PDM, Poetry, and uv all specify requires-python at the top level for the absolute minimum Python version needed for the lock file. This can be inferred, though, by examining all packages.requires-python values. The global value might also not be accurate for all platforms depending on how environment markers influence what package versions are installed and what their Python version requirements are.

Don’t pre-parse data?

This PEP currently takes the viewpoint that if a piece of data is going to be parsed by installers everytime they run, then trying to pre-parse as much as possible so the TOML parser can help is a good thing. The thinking is TOML parsers have a higher chance of being optimized, and so letting them do more parsing leads to a faster outcome. It should also increase readability by breaking apart data upfront more.

But in the case of doing this to wheel file names, some might consider it too much. The question becomes whether separating out all the parts of a wheel file name hinders readability because people are used to reading the file names already, or by clearly separating its parts it actually helps make installers faster, easier to write, and doesn’t hinder readability.

This all equally applies to requirement specifiers.

Per-file locking

An earlier version of this PEP supported two approaches to locking: per-file and per-package. The idea for the former approach to locking was that if you were locking for an a-priori set of environments you could lock to just the files necessary to install into those environments. The thinking was that by only listing a subset of files that auditing would be easier.

Unfortunately there was disagreement on how best to express upfront what the supported environment requirements would be. Since what this PEP currently proposes still prevents accidental success of installation into unsupported environments, this idea has been deferred until such time someone can come up with a representation that makes sense.

Allowing for multiple lock files

Before the introduction of [[groups]], this PEP proposed supporting multiple lock files that would match the regular expression r"pylock\.(.+)\.toml" if a name for the lock file is desired or if multiple lock files exist. But since [[groups]] subsumes a lot of the need to support multiple lock files, this specific feature can be postponed until such time that a need is shown to support multiple lock files.