Abstract

Wheels currently do not handle symlinks well, copying content instead of making symlinks when installed. To properly handle distributing libraries in wheels, we propose a new LINKS metadata file to handle symlinks in a platform portable manner. This specification requires a new wheel major version, discussed in PEP 777.

PEP Deferral

This PEP has been deferred until a better compatibility story for major changes to the wheel format is established. Once a compatibility story is established for wheels which allows backwards incompatible behavior in an unobtrusive way, the following points should be addressed in this PEP:

• Re-focus this topic to just symlinks for shared libraries on POSIX platforms, perhaps tied to platform tags?
• Should the symlinks be materialized as file attributes in the archive or a LINKS file? Could it be encoded in RECORD?
• Clarify that this PEP is insufficient to be useful for PEP 660 editable installs since it will no longer be cross platform.
• Describe fallback behavior in instances where symlinks are unavailable on POSIX platforms.

Motivation

Today, symlinks in wheels get created as copies of files, as the zipfile module in CPython does not support handling symlinks in-place for security reasons.

This presents problems to projects that would like to ship large compiled libraries in wheels, as they must choose to either greatly increase the install size of the project on disk, or omit the symlink and potentially break some downstream use cases.

To ship a library that can properly be loaded for runtime use or build time linking on POSIX, a library should follow the conventions of POSIX-style loader and linker search. The two main file names for the loader to use is the “soname” and the “real name”. The “soname” is a file like libfoo.so.3 where 3 is a number that is incremented when the interface of the library changes. The “real name” is a file named like libfoo.so.3.1.4, where the extra version information lets the loader find a specific version of a library. Finally, when compiling code to link against a library, the linker searches for a “linker name”, named like libfoo.so. A more detailed description is available in this Linux documentation on shared libraries. To fully support all runtime and build time use cases, a project requires shipping all 3 files. Normally, this is handled on POSIX platforms by using symlinks, so that the library is not duplicated on disk 3 times.

Returning to Python packaging, there are many popular projects which ship binary libraries, such as numpy, scipy, and pyarrow. Other site-packages dlopen libraries in other wheels, such as pytorch and jax. These projects currently rely on a single library in the wheel, but this can cause the linker to find the wrong library if there are system libraries that have a “real name” library version available.

There is also the potential benefit that symlinks in wheels would allow for simpler editable installs by simply placing a symlink in the user’s site-packages directory, but this PEP leaves that as an open question to be explored in a future PEP.

Rationale

To support the 3 main namings of a library used in loading and library linking on POSIX, we propose adding support for symlinks in Python wheels. To allow for tracking symlinks made, and to potentially support other platforms that may not support POSIX symlinks directly, we propose the use of a new wheel metadata file LINKS, which will exist in the .dist-info directory alongside METADATA, RECORD, and other metadata files.

Using a LINKS file will allow for more cross-platform uses of symlink-like usage. On Windows, symlinks require either a group policy allowing the user to make symlinks (e.g. by enabling Dev Mode) or Administrative permissions. This means that it may be the case that symlinks are unsupported on some user systems. By using a LINKS file, installers will be able to potentially use other methods for handling symlinks, such as junctions on Windows, where otherwise the installer would have to fail.

This PEP also describes checks that installers must make when installing an updated wheel. These checks exist to handle security risks from allowing wheels to install symlinks. For more information on why these checks are important, see Security Implications.

Specification

my_package/libfoo.so.3.1.4,my_package/libfoo.so.3
my_package/libfoo.so.3,my_package/libfoo.so

Backwards Compatibility

Introducing symlinks would require an increment to the wheel format major version. This would mean new wheels that use the new wheel format would raise an error on older installer tools, per the wheel specification.

Please see PEP 777 on “Wheel 2.0”.

Security Implications

Symlinks can be quite dangerous if not handled carefully. A simple example would be if a user were to run sudo pip install malicious, and there were no protections, then the malicious package could overwrite /etc/shadow and replace the password hash on the system, allowing malicious logins.

This PEP lists several requirements on checks to run by installers on symlinks in wheels to ensure attacks like the one described above cannot happen. This means it is critical that installers carefully implement these security safeguards and prevent malicious use on package installation.

In particular, the following checks MUST be made by installers:

1. That the symlinks do not point outside of any packages or namespaces coming from the wheel
2. That the symlinks are not dangling (the target exists at install time)
3. That the symlinks are not cyclical, stopping after a certain depth of checking to avoid denial of service requests

Do not follow symlinks on removal.

How to Teach This

End users should, once the changes have propagated through the ecosystem, transparently experience the benefits of symlinks in wheels. It is important for installers to give clear error messages if symlinks are unsupported on the platform, and explain why installation has failed.

For people building libraries, documentation on packaging.python.org should describe the use cases and caveats (especially platform support) of symlinks in wheels. Otherwise it should be handled transparently by build backends in the same way any normal file would be handled.

Reference Implementation

TODO

Rejected Ideas

Open Issues

Previous Discussion

https://discuss.python.org/t/symbolic-links-in-wheels/1945/25

Copyright

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.