name

In this proposal, a module’s “name” refers to its fully-qualified name, meaning the fully-qualified name of the module’s parent (if any) joined to the simple name of the module by a period.

finder

A “finder” is an object that identifies the loader that the import system should use to load a module. Currently this is accomplished by calling the finder’s find_module() method, which returns the loader.

Finders are strictly responsible for providing the loader, which they do through their find_module() method. The import system then uses that loader to load the module.

loader

A “loader” is an object that is used to load a module during import. Currently this is done by calling the loader’s load_module() method. A loader may also provide APIs for getting information about the modules it can load, as well as about data from sources associated with such a module.

Right now loaders (via load_module()) are responsible for certain boilerplate, import-related operations. These are:

1. Perform some (module-related) validation
2. Create the module object
3. Set import-related attributes on the module
4. “Register” the module to sys.modules
5. Exec the module
6. Clean up in the event of failure while loading the module

This all takes place during the import system’s call to Loader.load_module().

origin

This is a new term and concept. The idea of it exists subtly in the import system already, but this proposal makes the concept explicit.

“origin” in an import context means the system (or resource within a system) from which a module originates. For the purposes of this proposal, “origin” is also a string which identifies such a resource or system. “origin” is applicable to all modules.

For example, the origin for built-in and frozen modules is the interpreter itself. The import system already identifies this origin as “built-in” and “frozen”, respectively. This is demonstrated in the following module repr: “<module ‘sys’ (built-in)>”.

In fact, the module repr is already a relatively reliable, though implicit, indicator of a module’s origin. Other modules also indicate their origin through other means, as described in the entry for “location”.

It is up to the loader to decide on how to interpret and use a module’s origin, if at all.

location

This is a new term. However the concept already exists clearly in the import system, as associated with the __file__ and __path__ attributes of modules, as well as the name/term “path” elsewhere.

A “location” is a resource or “place”, rather than a system at large, from which a module is loaded. It qualifies as an “origin”. Examples of locations include filesystem paths and URLs. A location is identified by the name of the resource, but may not necessarily identify the system to which the resource pertains. In such cases the loader would have to identify the system itself.

In contrast to other kinds of module origin, a location cannot be inferred by the loader just by the module name. Instead, the loader must be provided with a string to identify the location, usually by the finder that generates the loader. The loader then uses this information to locate the resource from which it will load the module. In theory you could load the module at a given location under various names.

The most common example of locations in the import system are the files from which source and extension modules are loaded. For these modules the location is identified by the string in the __file__ attribute. Although __file__ isn’t particularly accurate for some modules (e.g. zipped), it is currently the only way that the import system indicates that a module has a location.

A module that has a location may be called “locatable”.

cache

The import system stores compiled modules in the __pycache__ directory as an optimization. This module cache that we use today was provided by PEP 3147. For this proposal, the relevant API for module caching is the __cache__ attribute of modules and the cache_from_source() function in importlib.util. Loaders are responsible for putting modules into the cache (and loading out of the cache). Currently the cache is only used for compiled source modules. However, loaders may take advantage of the module cache for other kinds of modules.

package

The concept does not change, nor does the term. However, the distinction between modules and packages is mostly superficial. Packages are modules. They simply have a __path__ attribute and import may add attributes bound to submodules. The typically perceived difference is a source of confusion. This proposal explicitly de-emphasizes the distinction between packages and modules where it makes sense to do so.

importlib.machinery.ModuleSpec (new)

An encapsulation of a module’s import-system-related state during import. See the ModuleSpec section below for a more detailed description.

• ModuleSpec(name, loader, *, origin=None, loader_state=None, is_package=None)

Attributes:

• name - a string for the fully-qualified name of the module.
• loader - the loader to use for loading.
• origin - the name of the place from which the module is loaded, e.g. “builtin” for built-in modules and the filename for modules loaded from source.
• submodule_search_locations - list of strings for where to find submodules, if a package (None otherwise).
• loader_state - a container of extra module-specific data for use during loading.
• cached (property) - a string for where the compiled module should be stored.
• parent (RO-property) - the fully-qualified name of the package to which the module belongs as a submodule (or None).
• has_location (RO-property) - a flag indicating whether or not the module’s “origin” attribute refers to a location.

importlib.util Additions

These are ModuleSpec factory functions, meant as a convenience for finders. See the Factory Functions section below for more detail.

• spec_from_file_location(name, location, *, loader=None, submodule_search_locations=None) - build a spec from file-oriented information and loader APIs.
• spec_from_loader(name, loader, *, origin=None, is_package=None) - build a spec with missing information filled in by using loader APIs.

Other API Additions

• importlib.find_spec(name, path=None, target=None) will work exactly the same as importlib.find_loader() (which it replaces), but return a spec instead of a loader.

For finders:

• importlib.abc.MetaPathFinder.find_spec(name, path, target) and importlib.abc.PathEntryFinder.find_spec(name, target) will return a module spec to use during import.

For loaders:

• importlib.abc.Loader.exec_module(module) will execute a module in its own namespace. It replaces importlib.abc.Loader.load_module(), taking over its module execution functionality.
• importlib.abc.Loader.create_module(spec) (optional) will return the module to use for loading.

For modules:

• Module objects will have a new attribute: __spec__.

API Changes

• InspectLoader.is_package() will become optional.

Deprecations

• importlib.abc.MetaPathFinder.find_module()
• importlib.abc.PathEntryFinder.find_module()
• importlib.abc.PathEntryFinder.find_loader()
• importlib.abc.Loader.load_module()
• importlib.abc.Loader.module_repr()
• importlib.util.set_package()
• importlib.util.set_loader()
• importlib.find_loader()

Removals

These were introduced prior to Python 3.4’s release, so they can simply be removed.

• importlib.abc.Loader.init_module_attrs()
• importlib.util.module_to_load()

Other Changes

• The import system implementation in importlib will be changed to make use of ModuleSpec.
• importlib.reload() will make use of ModuleSpec.
• A module’s import-related attributes (other than __spec__) will no longer be used directly by the import system during that module’s import. However, this does not impact use of those attributes (e.g. __path__) when loading other modules (e.g. submodules).
• Import-related attributes should no longer be added to modules directly, except by the import system.
• The module type’s __repr__() will be a thin wrapper around a pure Python implementation which will leverage ModuleSpec.
• The spec for the __main__ module will reflect the appropriate name and origin.

Backward-Compatibility

• If a finder does not define find_spec(), a spec is derived from the loader returned by find_module().
• PathEntryFinder.find_loader() still takes priority over find_module().
• Loader.load_module() is used if exec_module() is not defined.

What Will not Change?

• The syntax and semantics of the import statement.
• Existing finders and loaders will continue to work normally.
• The import-related module attributes will still be initialized with the same information.
• Finders will still create loaders (now storing them in specs).
• Loader.load_module(), if a module defines it, will have all the same requirements and may still be called directly.
• Loaders will still be responsible for module data APIs.
• importlib.reload() will still overwrite the import-related attributes.

Responsibilities

Here’s a quick breakdown of where responsibilities lie after this PEP.

finders:

• create/identify a loader that can load the module.
• create the spec for the module.

loaders:

• create the module (optional).
• execute the module.

ModuleSpec:

• orchestrate module loading
• boilerplate for module loading, including managing sys.modules and setting import-related attributes
• create module if loader doesn’t
• call loader.exec_module(), passing in the module in which to exec
• contain all the information the loader needs to exec the module
• provide the repr for modules

Attributes

Each of the following names is an attribute on ModuleSpec objects. A value of None indicates “not set”. This contrasts with module objects where the attribute simply doesn’t exist. Most of the attributes correspond to the import-related attributes of modules. Here is the mapping. The reverse of this mapping describes how the import machinery sets the module attributes right before calling exec_module().

While parent and has_location are read-only properties, the remaining attributes can be replaced after the module spec is created and even after import is complete. This allows for unusual cases where directly modifying the spec is the best option. However, typical use should not involve changing the state of a module’s spec.

origin

“origin” is a string for the name of the place from which the module originates. See origin above. Aside from the informational value, it is also used in the module’s repr. In the case of a spec where “has_location” is true, __file__ is set to the value of “origin”. For built-in modules “origin” would be set to “built-in”.

has_location

As explained in the location section above, many modules are “locatable”, meaning there is a corresponding resource from which the module will be loaded and that resource can be described by a string. In contrast, non-locatable modules can’t be loaded in this fashion, e.g. builtin modules and modules dynamically created in code. For these, the name is the only way to access them, so they have an “origin” but not a “location”.

“has_location” is true if the module is locatable. In that case the spec’s origin is used as the location and __file__ is set to spec.origin. If additional location information is required (e.g. zipimport), that information may be stored in spec.loader_state.

“has_location” may be implied from the existence of a load_data() method on the loader.

Incidentally, not all locatable modules will be cache-able, but most will.

submodule_search_locations

The list of location strings, typically directory paths, in which to search for submodules. If the module is a package this will be set to a list (even an empty one). Otherwise it is None.

The name of the corresponding module attribute, __path__, is relatively ambiguous. Instead of mirroring it, we use a more explicit attribute name that makes the purpose clear.

loader_state

A finder may set loader_state to any value to provide additional data for the loader to use during loading. A value of None is the default and indicates that there is no additional data. Otherwise it can be set to any object, such as a dict, list, or types.SimpleNamespace, containing the relevant extra information.

For example, zipimporter could use it to pass the zip archive name to the loader directly, rather than needing to derive it from origin or create a custom loader for each find operation.

loader_state is meant for use by the finder and corresponding loader. It is not guaranteed to be a stable resource for any other use.

Factory Functions

spec_from_file_location(name, location, *, loader=None, submodule_search_locations=None)

Build a spec from file-oriented information and loader APIs.

• “origin” will be set to the location.
• “has_location” will be set to True.
• “cached” will be set to the result of calling cache_from_source().
• “origin” can be deduced from loader.get_filename() (if “location” is not passed in.
• “loader” can be deduced from suffix if the location is a filename.
• “submodule_search_locations” can be deduced from loader.is_package() and from os.path.dirname(location) if location is a filename.

spec_from_loader(name, loader, *, origin=None, is_package=None)

Build a spec with missing information filled in by using loader APIs.

• “has_location” can be deduced from loader.get_data.
• “origin” can be deduced from loader.get_filename().
• “submodule_search_locations” can be deduced from loader.is_package() and from os.path.dirname(location) if location is a filename.

Backward Compatibility

ModuleSpec doesn’t have any. This would be a different story if Finder.find_module() were to return a module spec instead of loader. In that case, specs would have to act like the loader that would have been returned instead. Doing so would be relatively simple, but is an unnecessary complication. It was part of earlier versions of this PEP.

Subclassing

Subclasses of ModuleSpec are allowed, but should not be necessary. Simply setting loader_state or adding functionality to a custom finder or loader will likely be a better fit and should be tried first. However, as long as a subclass still fulfills the requirements of the import system, objects of that type are completely fine as the return value of Finder.find_spec(). The same points apply to duck-typing.

Module Objects

Other than adding __spec__, none of the import-related module attributes will be changed or deprecated, though some of them could be; any such deprecation can wait until Python 4.

A module’s spec will not be kept in sync with the corresponding import-related attributes. Though they may differ, in practice they will typically be the same.

One notable exception is that case where a module is run as a script by using the -m flag. In that case module.__spec__.name will reflect the actual module name while module.__name__ will be __main__.

A module’s spec is not guaranteed to be identical between two modules with the same name. Likewise there is no guarantee that successive calls to importlib.find_spec() will return the same object or even an equivalent object, though at least the latter is likely.

Finders

Finders are still responsible for identifying, and typically creating, the loader that should be used to load a module. That loader will now be stored in the module spec returned by find_spec() rather than returned directly. As is currently the case without the PEP, if a loader would be costly to create, that loader can be designed to defer the cost until later.

MetaPathFinder.find_spec(name, path=None, target=None)

PathEntryFinder.find_spec(name, target=None)

Finders must return ModuleSpec objects when find_spec() is called. This new method replaces find_module() and find_loader() (in the PathEntryFinder case). If a loader does not have find_spec(), find_module() and find_loader() are used instead, for backward-compatibility.

Adding yet another similar method to loaders is a case of practicality. find_module() could be changed to return specs instead of loaders. This is tempting because the import APIs have suffered enough, especially considering PathEntryFinder.find_loader() was just added in Python 3.3. However, the extra complexity and a less-than-explicit method name aren’t worth it.

The “target” parameter of find_spec()

A call to find_spec() may optionally include a “target” argument. This is the module object that will be used subsequently as the target of loading. During normal import (and by default) “target” is None, meaning the target module has yet to be created. During reloading the module passed in to reload() is passed through to find_spec() as the target. This argument allows the finder to build the module spec with more information than is otherwise available. Doing so is particularly relevant in identifying the loader to use.

Through find_spec() the finder will always identify the loader it will return in the spec (or return None). At the point the loader is identified, the finder should also decide whether or not the loader supports loading into the target module, in the case that “target” is passed in. This decision may entail consulting with the loader.

If the finder determines that the loader does not support loading into the target module, it should either find another loader or raise ImportError (completely stopping import of the module). This determination is especially important during reload since, as noted in How Reloading Will Work, loaders will no longer be able to trivially identify a reload situation on their own.

Two alternatives were presented to the “target” parameter: Loader.supports_reload() and adding “target” to Loader.exec_module() instead of find_spec(). supports_reload() was the initial approach to the reload situation. [6] However, there was some opposition to the loader-specific, reload-centric approach. [7]

As to “target” on exec_module(), the loader may need other information from the target module (or spec) during reload, more than just “does this loader support reloading this module”, that is no longer available with the move away from load_module(). A proposal on the table was to add something like “target” to exec_module(). [8] However, putting “target” on find_spec() instead is more in line with the goals of this PEP. Furthermore, it obviates the need for supports_reload().

Namespace Packages

Currently a path entry finder may return (None, portions) from find_loader() to indicate it found part of a possible namespace package. To achieve the same effect, find_spec() must return a spec with “loader” set to None (a.k.a. not set) and with submodule_search_locations set to the same portions as would have been provided by find_loader(). It’s up to PathFinder how to handle such specs.

Loaders

Loader.exec_module(module)

Loaders will have a new method, exec_module(). Its only job is to “exec” the module and consequently populate the module’s namespace. It is not responsible for creating or preparing the module object, nor for any cleanup afterward. It has no return value. exec_module() will be used during both loading and reloading.

exec_module() should properly handle the case where it is called more than once. For some kinds of modules this may mean raising ImportError every time after the first time the method is called. This is particularly relevant for reloading, where some kinds of modules do not support in-place reloading.

Loader.create_module(spec)

Loaders may also implement create_module() that will return a new module to exec. It may return None to indicate that the default module creation code should be used. One use case, though atypical, for create_module() is to provide a module that is a subclass of the builtin module type. Most loaders will not need to implement create_module(),

create_module() should properly handle the case where it is called more than once for the same spec/module. This may include returning None or raising ImportError.

Other changes:

PEP 420 introduced the optional module_repr() loader method to limit the amount of special-casing in the module type’s __repr__(). Since this method is part of ModuleSpec, it will be deprecated on loaders. However, if it exists on a loader it will be used exclusively.

Loader.init_module_attr() method, added prior to Python 3.4’s release, will be removed in favor of the same method on ModuleSpec.

However, InspectLoader.is_package() will not be deprecated even though the same information is found on ModuleSpec. ModuleSpec can use it to populate its own is_package if that information is not otherwise available. Still, it will be made optional.

In addition to executing a module during loading, loaders will still be directly responsible for providing APIs concerning module-related data.

Implementation Notes

* The implementation of this PEP needs to be cognizant of its impact on pkgutil (and setuptools). pkgutil has some generic function-based extensions to PEP 302 which may break if importlib starts wrapping loaders without the tools’ knowledge.

* Other modules to look at: runpy (and pythonrun.c), pickle, pydoc, inspect.

For instance, pickle should be updated in the __main__ case to look at module.__spec__.name.