Lazy module imports

Lazy module imports#

Python modules may become large enough for import-time code execution to have a perceptible overhead. This applies to Eradiate, whose codebase and dependency stack are big enough for imports on a high-performance desktop CPU at the time of writing (AMD Ryzen 9 5900X) to take more than a second.

Code optimization may not even completely solve the issue: unavoidable dependencies may have significant overhead. This used to apply, for instance, to xarray, ubiquitous in our codebase.

The solution the community seems to settle with is the lazy import of modules: module import-time code is executed upon module access, rather than upon import. The Scientific Python community came up with an Ecosystem Coordination proposal aiming to address this very issue. We use a publicly available implementation to both implement component lazy loading and populate our public API.

Using lazy_loader#

Usage is documented on the GitHub repository. Be sure to read the Type checkers section carefully as it describes how we use stub files.

In addition, checking that all lazy imports can be resolved is recommended. The way to do so is to use the EAGER_IMPORT environment variable, e.g.

EAGER_IMPORT=1 python -c "import eradiate"

This will force eager import of lazy module definitions. Missing modules will result in a ModuleNotFoundError being raised.