Following system colour scheme Selected dark colour scheme Selected light colour scheme

Python Enhancement Proposals

PEP 778 – Supporting Symlinks in Wheels

Author:
Emma Harper Smith <emma at python.org>
Sponsor:
Barry Warsaw <barry at python.org>
PEP-Delegate:
Paul Moore <p.f.moore at gmail.com>
Discussions-To:
Discourse thread
Status:
Deferred
Type:
Standards Track
Topic:
Packaging
Requires:
777
Created:
18-May-2024
Post-History:
10-Oct-2024

Table of Contents

Abstract

Wheels currently do not handle symlinks well, copying content instead of making symlinks when installed. To properly handle distributing libraries in wheels, we propose a new LINKS metadata file to handle symlinks in a platform portable manner. This specification requires a new wheel major version, discussed in PEP 777.

PEP Deferral

This PEP has been deferred until a better compatibility story for major changes to the wheel format is established. Once a compatibility story is established for wheels which allows backwards incompatible behavior in an unobtrusive way, the following points should be addressed in this PEP:

  • Re-focus this topic to just symlinks for shared libraries on POSIX platforms, perhaps tied to platform tags?
  • Should the symlinks be materialized as file attributes in the archive or a LINKS file? Could it be encoded in RECORD?
  • Clarify that this PEP is insufficient to be useful for PEP 660 editable installs since it will no longer be cross platform.
  • Describe fallback behavior in instances where symlinks are unavailable on POSIX platforms.

Motivation

Today, symlinks in wheels get created as copies of files, as the zipfile module in CPython does not support handling symlinks in-place for security reasons.

This presents problems to projects that would like to ship large compiled libraries in wheels, as they must choose to either greatly increase the install size of the project on disk, or omit the symlink and potentially break some downstream use cases.

To ship a library that can properly be loaded for runtime use or build time linking on POSIX, a library should follow the conventions of POSIX-style loader and linker search. The two main file names for the loader to use is the “soname” and the “real name”. The “soname” is a file like libfoo.so.3 where 3 is a number that is incremented when the interface of the library changes. The “real name” is a file named like libfoo.so.3.1.4, where the extra version information lets the loader find a specific version of a library. Finally, when compiling code to link against a library, the linker searches for a “linker name”, named like libfoo.so. A more detailed description is available in this Linux documentation on shared libraries. To fully support all runtime and build time use cases, a project requires shipping all 3 files. Normally, this is handled on POSIX platforms by using symlinks, so that the library is not duplicated on disk 3 times.

Returning to Python packaging, there are many popular projects which ship binary libraries, such as numpy, scipy, and pyarrow. Other site-packages dlopen libraries in other wheels, such as pytorch and jax. These projects currently rely on a single library in the wheel, but this can cause the linker to find the wrong library if there are system libraries that have a “real name” library version available.

There is also the potential benefit that symlinks in wheels would allow for simpler editable installs by simply placing a symlink in the user’s site-packages directory, but this PEP leaves that as an open question to be explored in a future PEP.

Rationale

To support the 3 main namings of a library used in loading and library linking on POSIX, we propose adding support for symlinks in Python wheels. To allow for tracking symlinks made, and to potentially support other platforms that may not support POSIX symlinks directly, we propose the use of a new wheel metadata file LINKS, which will exist in the .dist-info directory alongside METADATA, RECORD, and other metadata files.

Using a LINKS file will allow for more cross-platform uses of symlink-like usage. On Windows, symlinks require either a group policy allowing the user to make symlinks (e.g. by enabling Dev Mode) or Administrative permissions. This means that it may be the case that symlinks are unsupported on some user systems. By using a LINKS file, installers will be able to potentially use other methods for handling symlinks, such as junctions on Windows, where otherwise the installer would have to fail.

This PEP also describes checks that installers must make when installing an updated wheel. These checks exist to handle security risks from allowing wheels to install symlinks. For more information on why these checks are important, see Security Implications.

Specification

Wheel Major Version Bump

This PEP requires a wheel major version bump, so the Wheel-Version for wheels generated with LINKS MUST be at least version 2.0, so that older installers do not silently fail to install symlinks and break user environments. For more see PEP 777.

Installer Behavior Specification

Installers MUST resolve the paths of any link contained in the LINKS file before deciding if any source_path or target_path are valid. Installers MUST verify that source_path and target_path are located inside any namespace or package coming from the wheel. Installers MUST reject cyclic symlinks in wheels. Installers MAY error if a long chain of symlinks (symlinks pointing to symlinks many times repeated) exceeds a limit set by the installer.

Installers MUST follow the following steps when handling a wheel with symlinks:

  1. Check for the existence of a LINKS file in the .dist-info. If it does not exist, no further steps are required.
  2. Extract all files in the wheel packages and data directory as in wheel 1.x.
  3. Verify that for each source_path and target_path pairs, the target_path exists in one of the package namespaces just extracted.
  4. Next, check that the installer can make some kind of link for each pair in the site directory. If the installer cannot make a link for the file/folder target_path for the current platform, an error MUST be raised. An example of a failure mode would be a POSIX symlink to a file target, where the installer is running on Windows and the installer cannot make symlinks but can make junctions. In this case the installer MUST error because it cannot handle the link.
  5. Finally, the installer MUST add a platform-relevant link between source_path and target_path.

Installers MUST NOT by default copy files instead of generating a symlink when handling symlinks. Installers MAY have such behavior available under an alternate configuration or command line flag.

Build Backend Specification

When creating a wheel, build backends MUST treat symlinks in the same way as its target when deciding whether to include the symlink in a wheel. Build backends MUST verify that there are no dangling symlinks in the LINKS file. Build backends SHOULD recognize platform-relevant symlinks that would be included in builds. On POSIX systems this is typically symlinks, on Windows this includes symlinks and junctions.

Backwards Compatibility

Introducing symlinks would require an increment to the wheel format major version. This would mean new wheels that use the new wheel format would raise an error on older installer tools, per the wheel specification.

Please see PEP 777 on “Wheel 2.0”.

Security Implications

Symlinks can be quite dangerous if not handled carefully. A simple example would be if a user were to run sudo pip install malicious, and there were no protections, then the malicious package could overwrite /etc/shadow and replace the password hash on the system, allowing malicious logins.

This PEP lists several requirements on checks to run by installers on symlinks in wheels to ensure attacks like the one described above cannot happen. This means it is critical that installers carefully implement these security safeguards and prevent malicious use on package installation.

In particular, the following checks MUST be made by installers:

  1. That the symlinks do not point outside of any packages or namespaces coming from the wheel
  2. That the symlinks are not dangling (the target exists at install time)
  3. That the symlinks are not cyclical, stopping after a certain depth of checking to avoid denial of service requests

Do not follow symlinks on removal.

How to Teach This

End users should, once the changes have propagated through the ecosystem, transparently experience the benefits of symlinks in wheels. It is important for installers to give clear error messages if symlinks are unsupported on the platform, and explain why installation has failed.

For people building libraries, documentation on packaging.python.org should describe the use cases and caveats (especially platform support) of symlinks in wheels. Otherwise it should be handled transparently by build backends in the same way any normal file would be handled.

Reference Implementation

TODO

Rejected Ideas

Library Maintainers Should Use Python to Locate Libraries

Using Python to locate libraries would be much easier. However, some libraries like libtorch are used by extension modules and themselves require loading dependencies. Some compiled libraries cannot use Python to find their loader dependencies.

Open Issues

PEP 660 and Deferring Editable Installation Support

This PEP leaves the specification and implementation of a PEP 660 editable installation mechanism as unresolved for a later PEP; should that be specified in this PEP?

Security

This PEP needs to be reviewed to make sure it would not allow for new security vulnerabilities. Are there other restrictions we should place on the source or target of symlinks to protect users?

Previous Discussion

https://discuss.python.org/t/symbolic-links-in-wheels/1945/25


Source: https://github.com/python/peps/blob/main/peps/pep-0778.rst

Last modified: 2025-07-09 03:07:54 GMT