Lazy module imports
Lazy module imports#
Python modules may become large enough for import-time code execution to have a perceptible overhead. This applies to Eradiate, whose codebase and dependency stack are big enough for imports on a high-performance desktop CPU at the time of writing (AMD Ryzen 9 5900X) to take more than a second.
Code optimisation may not even completely solve the issue: unavoidable dependencies may have significant overhead. This applies, in particular, to xarray.
The solution the community seems to settle with is the lazy import of modules:
module import-time code is executed upon module access, rather than upon import.
The Scientific Python community came up with
an Ecosystem Coordination proposal
aiming to address this very issue. An implementation is
available publicly and
used in Eradiate, available as the
Why vendor rather than depend?
As of the time of writing, the
lazy_loader package is unstable; however,
lazy_loader ambitions to be a standard for lazy imports (at least from
the understanding of the Eradiate authors) and should therefore become a
dependency of many packages.
Lack of stability would therefore advocate for pinning this dependency; but (potentially) wide adoption advises against it to avoid clashes. Also, this package is not distributed on conda-forge, which is problematic for Eradiate.
Therefore, it was decided that the simplest way to implement this feature
without risking dependency clashes was to vendor this code. We will
transition to a dependency-based pattern when
lazy_loader will be stable
and packaged on conda-forge.
Usage is documented in the SPEC. Docstrings also provide useful usage information. Be sure to read the TYPE CHECKERS section carefully as it describes how we use stub files.
In addition, checking that all lazy imports can be resolved is recommended. The
way to do so is to use the
EAGER_IMPORT environment variable, e.g.
EAGER_IMPORT=1 python -c "import eradiate"
This will force eager import of lazy module definitions. Missing modules will
result in a
ModuleNotFoundError being raised.