PEP 778 – Supporting Symlinks in Wheels
- Author:
- Emma Harper Smith <emma at python.org>
- Sponsor:
- Barry Warsaw <barry at python.org>
- PEP-Delegate:
- Paul Moore <p.f.moore at gmail.com>
- Discussions-To:
- Discourse thread
- Status:
- Deferred
- Type:
- Standards Track
- Topic:
- Packaging
- Requires:
- 777
- Created:
- 18-May-2024
- Post-History:
- 10-Oct-2024
Abstract
Wheels currently do not handle symlinks well, copying content instead of making symlinks when
installed. To properly handle distributing libraries in wheels, we propose a new LINKS
metadata file to handle symlinks in a platform portable manner. This specification requires
a new wheel major version, discussed in PEP 777.
PEP Deferral
This PEP has been deferred until a better compatibility story for major changes to the wheel format is established. Once a compatibility story is established for wheels which allows backwards incompatible behavior in an unobtrusive way, the following points should be addressed in this PEP:
- Re-focus this topic to just symlinks for shared libraries on POSIX platforms, perhaps tied to platform tags?
- Should the symlinks be materialized as file attributes in the archive or a
LINKS
file? Could it be encoded inRECORD
? - Clarify that this PEP is insufficient to be useful for PEP 660 editable installs since it will no longer be cross platform.
- Describe fallback behavior in instances where symlinks are unavailable on POSIX platforms.
Motivation
Today, symlinks in wheels get created as copies of files, as the zipfile module in CPython does not support handling symlinks in-place for security reasons.
This presents problems to projects that would like to ship large compiled libraries in wheels, as they must choose to either greatly increase the install size of the project on disk, or omit the symlink and potentially break some downstream use cases.
To ship a library that can properly be loaded for runtime use or build time linking on POSIX, a
library should follow the conventions of POSIX-style loader and linker search. The two main file names for
the loader to use is the “soname” and the “real name”. The “soname” is a file like
libfoo.so.3
where 3
is a number that is incremented when the interface of the library
changes. The “real name” is a file named like libfoo.so.3.1.4
, where the extra version
information lets the loader find a specific version of a library. Finally, when compiling code to
link against a library, the linker searches for a “linker name”, named like libfoo.so
. A more
detailed description is available in this Linux documentation on shared libraries. To fully support all
runtime and build time use cases, a project requires shipping all 3 files. Normally, this is
handled on POSIX platforms by using symlinks, so that the library is not duplicated on disk 3 times.
Returning to Python packaging, there are many popular projects which ship binary libraries, such as
numpy
, scipy
, and pyarrow
. Other site-packages dlopen
libraries in other wheels, such as
pytorch
and jax
. These projects currently rely on a single library in the wheel, but
this can cause the linker to find the wrong library if there are system libraries that have a
“real name” library version available.
There is also the potential benefit that symlinks in wheels would allow for simpler editable
installs by simply placing a symlink in the user’s site-packages
directory, but this PEP
leaves that as an open question to be explored in a future PEP.
Rationale
To support the 3 main namings of a library used in loading and library linking on POSIX, we
propose adding support for symlinks in Python wheels. To allow for tracking symlinks made, and to
potentially support other platforms that may not support POSIX symlinks directly, we propose the
use of a new wheel metadata file LINKS
, which will exist in the .dist-info
directory alongside
METADATA
, RECORD
, and other metadata files.
Using a LINKS
file will allow for more cross-platform uses of symlink-like usage. On Windows,
symlinks require either a group policy allowing the user to make symlinks
(e.g. by enabling Dev Mode)
or Administrative permissions. This means that it may be the case that symlinks are unsupported on
some user systems. By using a LINKS
file, installers will be able to potentially use other
methods for handling symlinks, such as junctions on Windows, where otherwise the installer would
have to fail.
This PEP also describes checks that installers must make when installing an updated wheel. These checks exist to handle security risks from allowing wheels to install symlinks. For more information on why these checks are important, see Security Implications.
Specification
Wheel Major Version Bump
This PEP requires a wheel major version bump, so the Wheel-Version
for wheels generated with
LINKS
MUST be at least version 2.0
, so that older installers do not silently fail to
install symlinks and break user environments. For more see PEP 777.
New LINKS
Metadata File
To enable cross-platform symlinks, this PEP introduces a new wheel metadata file, LINKS
. An
example of a LINKS
file is below:
my_package/libfoo.so.3.1.4,my_package/libfoo.so.3
my_package/libfoo.so.3,my_package/libfoo.so
The format of LINKS
, as can seen above, is source_path,target_path
where source_path
is a path relative to the root of any namespace or package root in the wheel. target_path
is a
non-dangling path (i.e. a path that exists on the filesystem after the wheel’s contents are
extracted) in a package or namespace of any package in the wheel. This means that if a wheel
contains multiple packages, all paths in packages in the wheel are acceptable.
Installer Behavior Specification
Installers MUST resolve the paths of any link contained in the LINKS
file before
deciding if any source_path
or target_path
are valid. Installers MUST verify that
source_path
and target_path
are located inside any namespace or package coming from the
wheel. Installers MUST reject cyclic symlinks in wheels. Installers MAY error if a long
chain of symlinks (symlinks pointing to symlinks many times repeated) exceeds a limit set by the
installer.
Installers MUST follow the following steps when handling a wheel with symlinks:
- Check for the existence of a
LINKS
file in the.dist-info
. If it does not exist, no further steps are required. - Extract all files in the wheel packages and data directory as in wheel 1.x.
- Verify that for each
source_path
andtarget_path
pairs, thetarget_path
exists in one of the package namespaces just extracted. - Next, check that the installer can make some kind of link for each pair in the site directory.
If the installer cannot make a link for the file/folder
target_path
for the current platform, an error MUST be raised. An example of a failure mode would be a POSIX symlink to a file target, where the installer is running on Windows and the installer cannot make symlinks but can make junctions. In this case the installer MUST error because it cannot handle the link. - Finally, the installer MUST add a platform-relevant link between
source_path
andtarget_path
.
Installers MUST NOT by default copy files instead of generating a symlink when handling symlinks. Installers MAY have such behavior available under an alternate configuration or command line flag.
Build Backend Specification
When creating a wheel, build backends MUST treat symlinks in the same way as its target when
deciding whether to include the symlink in a wheel. Build backends MUST verify that there are
no dangling symlinks in the LINKS
file. Build backends SHOULD recognize platform-relevant
symlinks that would be included in builds. On POSIX systems this is typically symlinks, on Windows this
includes symlinks and junctions.
Backwards Compatibility
Introducing symlinks would require an increment to the wheel format major version. This would mean new wheels that use the new wheel format would raise an error on older installer tools, per the wheel specification.
Please see PEP 777 on “Wheel 2.0”.
Security Implications
Symlinks can be quite dangerous if not handled carefully. A simple example would be if a user were
to run sudo pip install malicious
, and there were no protections, then the malicious package
could overwrite /etc/shadow
and replace the password hash on the system, allowing malicious
logins.
This PEP lists several requirements on checks to run by installers on symlinks in wheels to ensure attacks like the one described above cannot happen. This means it is critical that installers carefully implement these security safeguards and prevent malicious use on package installation.
In particular, the following checks MUST be made by installers:
- That the symlinks do not point outside of any packages or namespaces coming from the wheel
- That the symlinks are not dangling (the target exists at install time)
- That the symlinks are not cyclical, stopping after a certain depth of checking to avoid denial of service requests
Do not follow symlinks on removal.
How to Teach This
End users should, once the changes have propagated through the ecosystem, transparently experience the benefits of symlinks in wheels. It is important for installers to give clear error messages if symlinks are unsupported on the platform, and explain why installation has failed.
For people building libraries, documentation on packaging.python.org
should describe the use
cases and caveats (especially platform support) of symlinks in wheels. Otherwise it should be
handled transparently by build backends in the same way any normal file would be handled.
Reference Implementation
TODO
Rejected Ideas
Just Use POSIX Symlinks Everywhere
This PEP wants to allow for LINKS
to be used for a potential future PEP 660 editable
installation. This future PEP should support Windows, so it may need to use junctions.
Don’t Use Junctions in LINKS
Junctions are a limited way to support symlinks between folders on Windows. They do not support files. This PEP allows for junctions as users may wish to only link folders to a different location, and future PEP 660 implementations may need to rely on this feature.
Put symlinks in the RECORD
Metadata File
While this could be done, it would clutter the RECORD
file. Furthermore the most
straightforward implementation would place the target at the end of the record. This would
make it harder to scan across the line and visually see what symlinks exist in the wheel.
Library Maintainers Should Use Python to Locate Libraries
Using Python to locate libraries would be much easier. However, some libraries like libtorch
are used by extension modules and themselves require loading dependencies. Some compiled libraries
cannot use Python to find their loader dependencies.
Include Support for Hardlinks
This PEP does not specify any behavior around hardlinks. This is intentional. This is left as an extension to a future PEP.
Open Issues
PEP 660 and Deferring Editable Installation Support
This PEP leaves the specification and implementation of a PEP 660 editable installation mechanism as unresolved for a later PEP; should that be specified in this PEP?
Security
This PEP needs to be reviewed to make sure it would not allow for new security vulnerabilities. Are there other restrictions we should place on the source or target of symlinks to protect users?
Allow inter-package symlinks
This could be useful for projects that want to shard dependencies such as large libraries between wheels but make them available in the main parent wheel.
The Format of LINKS
Currently the format is derived from RECORD
, but perhaps a better format exists.
Previous Discussion
https://discuss.python.org/t/symbolic-links-in-wheels/1945/25
Copyright
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
Source: https://github.com/python/peps/blob/main/peps/pep-0778.rst
Last modified: 2025-07-09 03:07:54 GMT