Lazy module imports

Python modules may become large enough for import-time code execution to have a perceptible overhead. This applies to Eradiate, whose codebase and dependency stack are big enough for imports on a high-performance desktop CPU at the time of writing (AMD Ryzen 9 5900X) to take more than a second.

Code optimisation may not even completely solve the issue: unavoidable dependencies may have significant overhead. This applies, in particular, to xarray.

The solution the community seems to settle with is the lazy import of modules: module import-time code is executed upon module access, rather than upon import. The Scientific Python community came up with an Ecosystem Coordination proposal aiming to address this very issue. An implementation is available publicly and used in Eradiate, available as the eradiate.util.lazy_loader module.

Why vendor rather than depend?

As of the time of writing, the lazy_loader package is unstable; however, lazy_loader ambitions to be a standard for lazy imports (at least from the understanding of the Eradiate authors) and should therefore become a dependency of many packages.

Lack of stability would therefore advocate for pinning this dependency; but (potentially) wide adoption advises against it to avoid clashes. Also, this package is not distributed on conda-forge, which is problematic for Eradiate.

Therefore, it was decided that the simplest way to implement this feature without risking dependency clashes was to vendor this code. We will transition to a dependency-based pattern when lazy_loader will be stable and packaged on conda-forge.

Using lazy_loader

Usage is documented in the SPEC. Docstrings also provide useful usage information. Be sure to read the TYPE CHECKERS section carefully as it describes how we use stub files.

In addition, checking that all lazy imports can be resolved is recommended. The way to do so is to use the EAGER_IMPORT environment variable, e.g.

EAGER_IMPORT=1 python -c "import eradiate"

This will force eager import of lazy module definitions. Missing modules will result in a ModuleNotFoundError being raised.