Per-file Locking

Per-file locking operates under the premise that one wants to install exactly the same files in any matching environment. As such, the lock file specifies what files to install. There can be multiple environments specified in a single file, each with their own set of files to install. By specifying the exact files to install, installers avoid performing any resolution to decide what to install.

The motivation for this approach to locking is for those who have controlled environments that they work with. For instance, if you have specific, controlled development and production environments then you can use per-file locking to make sure the same files are installed in both environments for everyone. This is similar to what pip freeze and pip-tools support, but with more strictness of the exact files as well as incorporating support to specify the locked files for multiple environments in the same file.

Per-file locking should be used when the installation attempt should fail outright if there is no explicitly pre-approved set of installation artifacts for the target platform. For example: locking the deployment dependencies for a managed web service.

Package Locking

Package locking lists the packages and their versions that may apply to any environment being installed for. The list of packages and their versions are evaluated individually and independently from any other packages and versions listed in the file. This allows installation to be linear – read each package and version and make an isolated decision as to whether it should be installed. This avoids requiring the installer to perform a resolution (i.e. determine what to install based on what else is to be installed).

The motivation of this approach comes from PDM lock files. By listing the potential packages and versions that may be installed, what’s installed is controlled in a way that’s easy to reason about. This also allows for not specifying the exact environments that would be supported by the lock file so there’s more flexibility for what environments are compatible with the lock file. This approach supports scenarios like open-source projects that want to lock what people should use to build the documentation without knowing upfront what environments their contributors are working from.

As already mentioned, this approach is supported by PDM. Poetry has shown some interest.

Per-package locking should be used when the exact set of potential target platforms is not known when generating the lock file, as it allows installation tools to choose the most appropriate artifacts for each platform from the pre-approved set. For example: locking the development dependencies for an open source project.

version

• String
• The version of the lock file format.
• This PEP specifies the initial version – and only valid value until future updates to the standard change it – as "1.0".
• If an installer supports the major version but not the minor version, a tool SHOULD warn when an unknown key is seen.
• If an installer doesn’t support a major version, it MUST raise an error.

hash-algorithm

• String
• The name of the hash algorithm used for calculating all hash values.
• Only a single hash algorithm is used for the entire file to allow the [[packages.files]] table to be written inline for readability and compactness purposes by only listing a single hash value instead of multiple values based on multiple hash algorithms.
• Specifying a single hash algorithm guarantees that an algorithm that the user prefers is used consistently throughout the file without having to audit each file hash value separately.
• Allows for updating the entire file to a new hash algorithm without running the risk of accidentally leaving an old hash value in the file.
• JSON-based Simple API for Python Package Indexes and the hashes dictionary of of the files dictionary of the Project Details dictionary specifies what values are valid and guidelines on what hash algorithms to use.
• Failure to validate any hash values for any file that is to be installed MUST raise an error.

dependencies

• Array of strings
• A listing of the dependency specifiers that act as the input to the lock file, representing the direct, top-level dependencies to be installed.

[[file-locks]]

• Array of tables
• Mutually exclusive with [package-lock].
• The array’s existence implies the use of the per-file locking approach.
• An environment that meets all of the specified criteria in the table will be considered compatible with the environment that was locked for.
• Lockers MUST NOT generate multiple [file-locks] tables which would be considered compatible for the same environment.
• In instances where there would be a conflict but the lock is still desired, either separate lock files can be written or per-package locking can be used.
• Entries in array SHOULD be sorted by file-locks.name lexicographically.

[package-lock]

• Table
• Mutually exclusive with [[file-locks]].
• Signifies the use of the package locking approach.

[[packages]]

• Array of tables
• The array contains all data on the locked package versions.
• Lockers SHOULD record packages in order by packages.name lexicographically , packages.version by the sort order for version specifiers, and packages.markers lexicographically.
• Lockers SHOULD record keys in the same order as written in this PEP to minimize changes when updating.
• Entries are designed so that relevant details as to why a package is included are in one place to make diff reading easier.

[tool]

• Optional
• Table
• Same usage as that of the equivalent [tool] table from the pyproject.toml specification.

Per-file locking

version = '1.0'
hash-algorithm = 'sha256'
dependencies = ['cattrs', 'numpy']

[[file-locks]]
name = 'CPython 3.12 on manylinux 2.17 x86-64'
marker-values = {}
wheel-tags = ['cp312-cp312-manylinux_2_17_x86_64', 'py3-none-any']

[[file-locks]]
name = 'CPython 3.12 on Windows x64'
marker-values = {}
wheel-tags = ['cp312-cp312-win_amd64', 'py3-none-any']

[[packages]]
name = 'attrs'
version = '23.2.0'
multiple-entries = false
description = 'Classes Without Boilerplate'
requires-python = '>=3.7'
dependents = ['cattrs']
dependencies = []
direct = false
files = [
    {name = 'attrs-23.2.0-py3-none-any.whl', lock = ['CPython 3.12 on manylinux 2.17 x86-64', 'CPython 3.12 on Windows x64'], origin = 'https://files.pythonhosted.org/packages/e0/44/827b2a91a5816512fcaf3cc4ebc465ccd5d598c45cefa6703fcf4a79018f/attrs-23.2.0-py3-none-any.whl', hash = '99b87a485a5820b23b879f04c2305b44b951b502fd64be915879d77a7e8fc6f1'}
]

[[packages]]
name = 'cattrs'
version = '23.2.3'
multiple-entries = false
description = 'Composable complex class support for attrs and dataclasses.'
requires-python = '>=3.8'
dependents = []
dependencies = ['attrs']
direct = false
files = [
    {name = 'cattrs-23.2.3-py3-none-any.whl', lock = ['CPython 3.12 on manylinux 2.17 x86-64', 'CPython 3.12 on Windows x64'], origin = 'https://files.pythonhosted.org/packages/b3/0d/cd4a4071c7f38385dc5ba91286723b4d1090b87815db48216212c6c6c30e/cattrs-23.2.3-py3-none-any.whl', hash = '0341994d94971052e9ee70662542699a3162ea1e0c62f7ce1b4a57f563685108'}
]

[[packages]]
name = 'numpy'
version = '2.0.1'
multiple-entries = false
description = 'Fundamental package for array computing in Python'
requires-python = '>=3.9'
dependents = []
dependencies = []
direct = false
files = [
    {name = 'numpy-2.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl', lock = ['cp312-manylinux_2_17_x86_64'], origin = 'https://files.pythonhosted.org/packages/2c/f3/61eeef119beb37decb58e7cb29940f19a1464b8608f2cab8a8616aba75fd/numpy-2.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl', hash = '6790654cb13eab303d8402354fabd47472b24635700f631f041bd0b65e37298a'},
    {name = 'numpy-2.0.1-cp312-cp312-win_amd64.whl', lock = ['cp312-win_amd64'], origin = 'https://files.pythonhosted.org/packages/b5/59/f6ad30785a6578ad85ed9c2785f271b39c3e5b6412c66e810d2c60934c9f/numpy-2.0.1-cp312-cp312-win_amd64.whl', hash = 'bb2124fdc6e62baae159ebcfa368708867eb56806804d005860b6007388df171'}
]

Per-package locking

Some values for packages.files.origin left out to make creating this example more easily as it was done by hand.

version = '1.0'
hash-algorithm = 'sha256'
dependencies = ['cattrs', 'numpy']

[package-lock]
requires-python = ">=3.9"


[[packages]]
name = 'attrs'
version = '23.2.0'
multiple-entries = false
description = 'Classes Without Boilerplate'
requires-python = '>=3.7'
dependents = ['cattrs']
dependencies = []
direct = false
files = [
    {name = 'attrs-23.2.0-py3-none-any.whl', lock = ['cp312-manylinux_2_17_x86_64', 'cp312-win_amd64'], origin = 'https://files.pythonhosted.org/packages/e0/44/827b2a91a5816512fcaf3cc4ebc465ccd5d598c45cefa6703fcf4a79018f/attrs-23.2.0-py3-none-any.whl', hash = '99b87a485a5820b23b879f04c2305b44b951b502fd64be915879d77a7e8fc6f1'}
]

[[packages]]
name = 'cattrs'
version = '23.2.3'
multiple-entries = false
description = 'Composable complex class support for attrs and dataclasses.'
requires-python = '>=3.8'
dependents = []
dependencies = ['attrs']
direct = false
files = [
    {name = 'cattrs-23.2.3-py3-none-any.whl', lock = ['cp312-manylinux_2_17_x86_64', 'cp312-win_amd64'], origin = 'https://files.pythonhosted.org/packages/b3/0d/cd4a4071c7f38385dc5ba91286723b4d1090b87815db48216212c6c6c30e/cattrs-23.2.3-py3-none-any.whl', hash = '0341994d94971052e9ee70662542699a3162ea1e0c62f7ce1b4a57f563685108'}
]

[[packages]]
name = 'numpy'
version = '2.0.1'
multiple-entries = false
description = 'Fundamental package for array computing in Python'
requires-python = '>=3.9'
dependents = []
dependencies = []
direct = false
files = [
    {name = "numpy-2.0.1-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:6bf4e6f4a2a2e26655717a1983ef6324f2664d7011f6ef7482e8c0b3d51e82ac"},
    {name = "numpy-2.0.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:7d6fddc5fe258d3328cd8e3d7d3e02234c5d70e01ebe377a6ab92adb14039cb4"},
    {name = "numpy-2.0.1-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:5daab361be6ddeb299a918a7c0864fa8618af66019138263247af405018b04e1"},
    {name = "numpy-2.0.1-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:ea2326a4dca88e4a274ba3a4405eb6c6467d3ffbd8c7d38632502eaae3820587"},
    {name = "numpy-2.0.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:529af13c5f4b7a932fb0e1911d3a75da204eff023ee5e0e79c1751564221a5c8"},
    {name = "numpy-2.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6790654cb13eab303d8402354fabd47472b24635700f631f041bd0b65e37298a"},
    {name = "numpy-2.0.1-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:cbab9fc9c391700e3e1287666dfd82d8666d10e69a6c4a09ab97574c0b7ee0a7"},
    {name = "numpy-2.0.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:99d0d92a5e3613c33a5f01db206a33f8fdf3d71f2912b0de1739894668b7a93b"},
    {name = "numpy-2.0.1-cp312-cp312-win32.whl", hash = "sha256:173a00b9995f73b79eb0191129f2455f1e34c203f559dd118636858cc452a1bf"},
    {name = "numpy-2.0.1-cp312-cp312-win_amd64.whl", hash = "sha256:bb2124fdc6e62baae159ebcfa368708867eb56806804d005860b6007388df171"},
]

Installing for per-file locking

• If no compatible environment is found an error MUST be raised.
• If multiple environments are found to be compatible then an error MUST be raised.
• If a [[packages.files]] contains multiple matching entries an error MUST be raised due to ambiguity for what is to be installed.
• If multiple [[packages]] entries for the same package have matching files an error MUST be raised due to ambiguity for what is to be installed.

Installing for package locking

• Verify that the environment is compatible with package-lock.requires-python; if it isn’t an error MUST be raised.
• If no way to install a required package is found, an error MUST be raised.

Using *.pylock.toml as the file name

It was proposed to put the pylock constant part of the file name after the identifier for the purpose of the lock file. It was decided not to do this so that lock files would sort together when looking at directory contents instead of purely based on their purpose which could spread them out in a directory.

Using *.pylock as the file name

Not using .toml as the file extension and instead making it .pylock itself was proposed. This was decided against so that code editors would know how to provide syntax highlighting to a lock file without having special knowledge about the file extension.

Not having a naming convention for the file

Having no requirements or guidance for a lock file’s name was considered, but ultimately rejected. By having a standardized naming convention it makes it easy to identify a lock file for both a human and a code editor. This helps facilitate discovery when e.g. a tool wants to know all of the lock files that are available.

Use JSON over TOML

Since having a format that is machine-writable was a goal of this PEP, it was suggested to use JSON. But it was deemed less human-readable than TOML while not improving on the machine-writable aspect enough to warrant the change.

Use YAML over TOML

Some argued that YAML met the machine-writable/human-readable requirement in a better way than TOML. But as that’s subjective and pyproject.toml already existed as the human-writable file used by Python packaging standards it was deemed more important to keep using TOML.

Multiple hashes per file

An initial version of this PEP proposed supporting multiple hashes per file. The idea was to allow one to choose which hashing algorithm they wanted to go with when installing. But upon reflection it seemed like an unnecessary complication as there was no guarantee the hashes provided would satisfy the user’s needs. As well, if the single hash algorithm used in the lock file wasn’t sufficient, rehashing the files involved as a way to migrate to a different algorithm didn’t seem insurmountable.

Hashing the contents of the lock file itself

Hashing the contents of the bytes of the file and storing hash value within the file itself was proposed at some point. This was removed to make it easier when merging changes to the lock file as each merge would have to recalculate the hash value to avoid a merge conflict.

Hashing the semantic contents of the file was also proposed, but it would lead to the same merge conflict issue.

Regardless of which contents were hashed, either approach could have the hash value stored outside of the file if such a hash was desired.

Recording the creation date of the lock file

To know how potentially stale the lock file was, an earlier proposal suggested recording the creation date of the lock file. But for some same merge conflict reasons as storing the hash of the file contents, this idea was dropped.

Recording the package indexes used

Recording what package indexes were used by the locker to decide what to lock for was considered. In the end, though, it was rejected as it was deemed unnecessary bookkeeping.