eradiate.pipelines¶
This module contains the infrastructure, definitions, and logic for all post-processing pipeline operations. The pipeline is modelled as a directed acyclic graph (DAG) built with a lightweight custom engine.
For clarity, the implementation is split as follows:
eradiate.pipelines.engineprovides the fundamental logic powering eacheradiate.pipelines.logicprovides the fundamental logic powering each step of post-processing;eradiate.pipelines.definitionscontains the imperative pipeline builder (build_pipeline());eradiate.pipelines._config(private) provides theconfig()utility for generating pipeline configuration dictionaries.
Pipeline configuration¶
Post-processing pipelines are configured using a dictionary specifying
parameters listed in the documentation of the
build_pipeline() function, which
implements post-processing pipeline assembly.
Configuration generation is automated by the config()
helper function.
Many nodes of the DAG defining the pipeline are tagged for filtering upon query. Tags explicitly used in the pipeline setup and execution are listed in the table below.
Tag |
Value |
Description |
|---|---|---|
|
|
The node’s output will be selected to be part of the default pipeline output. |
|
Defines the type of variable the node’s output corresponds to.
Data with the |
|
|
The output is a data variable and can be assembled into a dataset. |
|
|
The output is a coordinate variable. |
- eradiate.pipelines.config(measure, mode=None, integrator=None)[source]¶
Generate a pipeline configuration for a specific scene setup.
- Parameters:
measure (
Measure)mode (
Modeorstr, optional) – Mode or mode ID for which the pipeline is configured. By default, the current active mode is used.integrator (
IntegratororNone, optional) – Integrator used for the experiment; indicates whether the moment was calculated during the integration.
- Returns:
See also
Pipeline assembly (eradiate.pipelines.definitions)¶
Post-processing pipeline builder definitions.
This module contains components to imperatively assemble a
Pipeline from a configuration dictionary.
Functions:
|
Build a post-processing pipeline from a configuration dictionary. |
- eradiate.pipelines.definitions.build_pipeline(config)[source]¶
Build a post-processing pipeline from a configuration dictionary.
- Parameters:
config (
dict) – Pipeline configuration dictionary. Expected keys:mode_idstrEradiate mode identifier.
measure_distantboolWhether the measure is a distant measure.
add_viewing_anglesboolWhether to compute viewing angles.
var_namestrName of the processed physical variable.
var_metadatadictMetadata to attach to the variable data array.
apply_spectral_responseboolWhether to apply SRF weighting.
calculate_varianceboolWhether to compute variance.
calculate_stokesboolWhether to compute the full Stokes vector.
- Returns:
Pipeline– A configured pipeline ready for execution.
Pipeline engine (eradiate.pipelines.engine)¶
Core pipeline engine implementation.
Classes:
|
Represents a computation node in the pipeline. |
|
A lightweight DAG-based pipeline engine. |
- class eradiate.pipelines.engine.Node(name, func, dependencies=NOTHING, description='', pre_funcs=NOTHING, post_funcs=NOTHING, validate=True, metadata=NOTHING)[source]¶
Represents a computation node in the pipeline.
- Parameters:
name (
str) – Unique identifier for the node.func (
callable()) – Function to execute for this node. Parameters must match dependency names.dependencies (
listofstr, optional) – Names of nodes whose outputs are inputs to this node.description (
str, optional) – Human-readable description of what this node does.pre_funcs (
listofcallable(), optional) – Functions to run before executing the node. Each function receives the inputs dictionary. Can be used for validation or inspection.post_funcs (
listofcallable(), optional) – Functions to run after executing the node. Each function receives the node output. Can be used for validation or inspection.validate (
bool, default:True) – Whether pre/post functions are enabled for this node.metadata (
dict, optional) – Additional metadata/tags for the node.
- class eradiate.pipelines.engine.Pipeline(validate=True)[source]¶
A lightweight DAG-based pipeline engine.
This class provides an imperative API for building and executing computational pipelines. It uses networkx for graph operations and supports features like input injection and validation.
Examples
Basic usage:
>>> pipeline = Pipeline() >>> pipeline.add_node("a", lambda: 1) >>> pipeline.add_node("b", lambda: 2) >>> pipeline.add_node("c", lambda a, b: a + b, dependencies=["a", "b"]) >>> results = pipeline.execute(outputs=["c"]) >>> print(results["c"]) 3
Virtual inputs: dependencies that don’t exist as nodes are automatically treated as virtual inputs. These must be provided via the
inputsparameter during execution:>>> pipeline = Pipeline() >>> pipeline.add_node("b", lambda a: a + 1, dependencies=["a"]) >>> pipeline.get_virtual_inputs() ['a'] >>> results = pipeline.execute(outputs=["b"], inputs={"a": 10}) >>> results["b"] 11
Methods:
add_node(name, func[, dependencies, ...])Add a computation node to the pipeline.
Clear the execution cache.
execute([outputs, inputs])Execute the pipeline and return results.
extract_subgraph(outputs)Extract a subgraph containing only nodes needed for specified outputs.
get_node(name)Get a node by name.
get_nodes_by_metadata(**kwargs)Return names of nodes whose metadata matches all given key-value pairs.
get_required_inputs([outputs, inputs])Get virtual inputs required for computing specific outputs.
Get all virtual input names in the pipeline.
is_virtual_input(name)Check if a name corresponds to a virtual input.
List all node names in topological order.
Print a text summary of the pipeline structure.
remove_node(name)Remove a node from the pipeline.
visualize([highlight_nodes, legend, ...])Generate and display pipeline visualization as SVG in Jupyter notebooks.
write_dot(filename[, highlight_nodes, ...])Export pipeline to Graphviz DOT format.
write_png(filename[, highlight_nodes, ...])Export pipeline visualization to a PNG file.
write_svg(filename[, highlight_nodes, ...])Export pipeline visualization to an SVG file.
- add_node(name, func, dependencies=None, description='', pre_funcs=None, post_funcs=None, validate=True, metadata=None, outputs=None)[source]¶
Add a computation node to the pipeline.
Dependencies that don’t exist as nodes are automatically treated as virtual inputs that must be provided via
inputsduring execution.When
outputsis provided,funcis expected to return a dict. Each entry inoutputsbecomes an independent child node, letting downstream nodes depend on individual fields rather than the whole dict.- Parameters:
name (
str) – Unique identifier for the node. Whenoutputsis given, this node holds the intermediate dict; by convention, prefix it with an underscore (e.g."_stats").func (
callable()) – Function to execute. Its parameters must match dependency names. Whenoutputsis given, must return a mapping (typically, a dict).dependencies (
listofstr, optional) – Names of nodes or virtual inputs whose outputs feed into this node.description (
str, optional) – Human-readable description.pre_funcs (
listofcallable(), optional) – Functions to run before node execution (validation, inspection).post_funcs (
listofcallable(), optional) – Functions to run after node execution (validation, inspection).validate (
bool, default:True) – Enable pre/post functions for this node.metadata (
dict, optional) – Additional metadata to attach to the node.outputs (
listofstrordict, optional) – Specifies child nodes to extract from the output offunc. This requiresfuncto return a mapping (typically, a dict). Three forms are accepted:list[str]: each string becomes both the node ID and the dict key to extract.["x", "y"]is equivalent to{"x": "x", "y": "y"}.dict[str, str]: maps node ID to dict key.{"x_node": "x_key"}extractsd["x_key"]into node"x_node".dict[str, Callable]: maps node ID to an extractor callable that receives the full dict and returns the node value.{"x": lambda d: d["x"]}for full control.
- Returns:
Pipeline– Self for method chaining.- Raises:
ValueError – If node name already exists or if adding creates a cycle.
Examples
Simple node:
>>> pipeline = Pipeline() >>> pipeline.add_node("a", lambda: 1) >>> pipeline.add_node("b", lambda a: a + 1, dependencies=["a"]) >>> pipeline.execute(outputs=["b"]) {'b': 2}
Field extraction — list form (node ID == dict key):
>>> pipeline = Pipeline() >>> pipeline.add_node("_raw", lambda: {"x": 1, "y": 2}, outputs=["x", "y"]) >>> pipeline.execute(outputs=["x", "y"]) {'x': 1, 'y': 2}
Field extraction — dict[str, str] form (node ID → dict key):
>>> pipeline = Pipeline() >>> pipeline.add_node( ... "_raw", ... lambda: {"x_internal": 1}, ... outputs={"x": "x_internal"}, ... ) >>> pipeline.execute(outputs=["x"]) {'x': 1}
Field extraction — dict[str, Callable] form (full control):
>>> pipeline = Pipeline() >>> pipeline.add_node( ... "_raw", ... lambda: {"x": 1, "y": 2}, ... outputs={"sum": lambda d: d["x"] + d["y"]}, ... ) >>> pipeline.execute(outputs=["sum"]) {'sum': 3}
- execute(outputs=None, inputs=None)[source]¶
Execute the pipeline and return results.
- Parameters:
outputs (
listofstr, optional) – Names of nodes whose values should be returned. These can be any nodes in the pipeline, including intermediate ones. All required ancestor nodes will be computed automatically. If None, computes all leaf nodes.inputs (
dict, optional) – Dictionary mapping node names or virtual inputs to data values.For virtual inputs: provides the required input values.
For regular nodes: the node will not be executed; the provided value is used instead, effectively bypassing its computation.
- Returns:
dict– Dictionary mapping requested node names to their computed values.- Raises:
ValueError – If output nodes don’t exist, if required virtual inputs are missing, or if outputs are not reachable from provided inputs.
- extract_subgraph(outputs)[source]¶
Extract a subgraph containing only nodes needed for specified outputs.
Virtual inputs required for the outputs are preserved in the subgraph.
- Parameters:
- Returns:
Pipeline– A new pipeline containing only necessary nodes and virtual inputs.- Raises:
ValueError – If any output node doesn’t exist.
- get_node(name)[source]¶
Get a node by name.
- Parameters:
name (
str) – Name of the node.- Returns:
Node– The node object.- Raises:
ValueError – If node doesn’t exist.
- get_nodes_by_metadata(**kwargs)[source]¶
Return names of nodes whose metadata matches all given key-value pairs.
Virtual inputs are excluded. Results are returned in topological order.
- Parameters:
**kwargs – Key-value pairs that must all be present in a node’s metadata.
- Returns:
listofstr– Node names (in topological order) whose metadata contains all specified key-value pairs.
Examples
>>> pipeline = Pipeline() >>> pipeline.add_node("a", lambda: 1, metadata={"final": True, "kind": "data"}) >>> pipeline.add_node("b", lambda: 2, metadata={"final": True, "kind": "debug"}) >>> pipeline.get_nodes_by_metadata(final=True, kind="data") ['a'] >>> pipeline.get_nodes_by_metadata(final=True) ['a', 'b']
- get_required_inputs(outputs=None, inputs=None)[source]¶
Get virtual inputs required for computing specific outputs.
This helps users determine what data they need to provide in
inputsto execute specific outputs.- Parameters:
- Returns:
Examples
>>> pipeline = Pipeline() >>> pipeline.add_node("b", lambda a: a + 1, dependencies=["a"]) >>> pipeline.add_node("c", lambda b: b * 2, dependencies=["b"]) >>> pipeline.get_required_inputs(outputs=["c"]) ['a'] >>> pipeline.get_required_inputs(outputs=["c"], inputs={"b": 5}) []
- get_virtual_inputs()[source]¶
Get all virtual input names in the pipeline.
Virtual inputs are dependencies that don’t exist as nodes and must be provided via
inputsduring execution.Examples
>>> pipeline = Pipeline() >>> pipeline.add_node("b", lambda a: a + 1, dependencies=["a"]) >>> pipeline.get_virtual_inputs() ['a']
- remove_node(name)[source]¶
Remove a node from the pipeline.
- Parameters:
name (
str) – Name of the node to remove.- Returns:
Pipeline– Self for method chaining.- Raises:
ValueError – If node doesn’t exist or has downstream dependencies.
- visualize(highlight_nodes=None, legend=False, outputs=None, inputs=None, show_inactive=False)[source]¶
Generate and display pipeline visualization as SVG in Jupyter notebooks.
This function creates an SVG visualization using the Graphviz dot backend and displays it inline in Jupyter notebooks using IPython.display.
When
outputsand/orinputsare provided, an execution-context view is shown: the active execution path is highlighted, bypassed nodes are shown in a different color, and nodes outside the execution path are omitted by default.- Parameters:
highlight_nodes (
listofstr, optional) – Node names to highlight in the visualization.legend (
bool) – If True, add a legend explaining node styles.outputs (
listofstr, optional) – Restrict the view to the subgraph needed for these outputs.inputs (
dict, optional) – Inputs as passed toexecute(). Bypassed nodes and satisfied virtual inputs are styled distinctly.show_inactive (
bool) – If True, render inactive nodes with a dimmed style instead of omitting them. Default is False.
- Returns:
IPython.display.SVG– SVG object that will display inline in Jupyter notebooks.- Raises:
ImportError – If pydot or IPython is not installed.
Examples
>>> pipeline = Pipeline() >>> pipeline.add_node("a", lambda: 1) >>> pipeline.add_node("b", lambda a: a + 1, dependencies=["a"]) >>> pipeline.visualize() >>> pipeline.visualize(outputs=["b"], inputs={"a": 10})
- write_dot(filename, highlight_nodes=None, legend=False, outputs=None, inputs=None, show_inactive=False)[source]¶
Export pipeline to Graphviz DOT format.
- Parameters:
filename (
str) – Output filename.highlight_nodes (
listofstr, optional) – Node names to highlight in the visualization.legend (
bool) – If True, add a legend explaining node styles.outputs (
listofstr, optional) – Restrict the view to the subgraph needed for these outputs.inputs (
dict, optional) – Inputs as passed toexecute(). Bypassed nodes and satisfied virtual inputs are styled distinctly.show_inactive (
bool) – If True, render inactive nodes with a dimmed style instead of omitting them. Default is False.
- Raises:
ImportError – If pydot is not installed.
- write_png(filename, highlight_nodes=None, legend=False, outputs=None, inputs=None, show_inactive=False)[source]¶
Export pipeline visualization to a PNG file.
- Parameters:
filename (
str) – Output PNG filename.highlight_nodes (
listofstr, optional) – Node names to highlight in the visualization.legend (
bool) – If True, add a legend explaining node styles.outputs (
listofstr, optional) – Restrict the view to the subgraph needed for these outputs.inputs (
dict, optional) – Inputs as passed toexecute(). Bypassed nodes and satisfied virtual inputs are styled distinctly.show_inactive (
bool) – If True, render inactive nodes with a dimmed style instead of omitting them. Default is False.
- Raises:
ImportError – If pydot is not installed.
- write_svg(filename, highlight_nodes=None, legend=False, outputs=None, inputs=None, show_inactive=False)[source]¶
Export pipeline visualization to an SVG file.
- Parameters:
filename (
str) – Output SVG filename.highlight_nodes (
listofstr, optional) – Node names to highlight in the visualization.legend (
bool) – If True, add a legend explaining node styles.outputs (
listofstr, optional) – Restrict the view to the subgraph needed for these outputs.inputs (
dict, optional) – Inputs as passed toexecute(). Bypassed nodes and satisfied virtual inputs are styled distinctly.show_inactive (
bool) – If True, render inactive nodes with a dimmed style instead of omitting them. Default is False.
- Raises:
ImportError – If pydot is not installed.
Pipeline logic (eradiate.pipelines.logic)¶
Basic components of post-processing pipeline assembly.
Functions:
|
Compute CKD quadrature. |
|
Apply SRF weighting (a.k.a. |
|
Compute the bi-hemispherical reflectance (a.k.a surface albedo) from radiosity and irradiance records. |
Compute the spectral bidirectional reflectance distribution function (BRDF) and bidirectional reflectance factor (BRF) from radiance and irradiance records. |
|
|
Calculate the degree of linear polarization from a Stokes vector. |
|
Derive an irradiance dataset from the irradiance spectrum of an illuminant, as well as solar angles. |
|
Gather a collection of Mitsuba bitmaps into xarray data arrays. |
|
Calculate the variance (central 2nd moment) from the raw 2nd moment and expected value. |
|
Aggregate sector radiosity into a full-hemisphere radiosity dataset. |
|
Evaluate a spectral response function as a data array. |
|
Collect viewing angles associated with each film pixel from the measure, if relevant. |
- eradiate.pipelines.logic.aggregate_ckd_quad(mode_id, raw_data, spectral_grid, ckd_quads, is_variance)[source]¶
Compute CKD quadrature.
In CKD modes, this pipeline step aggregates spectral data and evaluates the selected quadrature rule. The following updates to the input data are expected:
the
gdimension is dropped;a
bin_wmin(resp.bin_wmax) coordinate is created and contains the lower (resp. upper) spectral bound of each bin;the dataset is reordered by ascending
wvalues.
- Parameters:
mode_id (
str) – Eradiate mode for which this pipeline step is configured. If set to “ckd”, CKD quadrature computation is performed; otherwise, it is a no-op.raw_data (
DataArray) – A data array holding raw bitmap data indexed against pixel indices.spectral_grid (
CKDSpectralGrid) – Spectral set for which the CKD quadrature is computed.ckd_quads (
listofQuad) – List of quadrature rules matching the spectral bins held byspectral_grid.is_variance (
bool) – Flag that specifies whether the raw_data is a variance value.
- Returns:
DataArray– A data array indexed against spectral coordinates associated with each spectral bin.
Notes
In non-CKD modes, this step is a no-op.
During pipeline assembly, this node expands as a single node named
<var>.The
sppvariable is averaged on theindexdimension.
- eradiate.pipelines.logic.apply_spectral_response(spectral_data, srf)[source]¶
Apply SRF weighting (a.k.a. convolution) to spectral data and turn it into a band aggregate.
- eradiate.pipelines.logic.compute_albedo(radiosity_data, irradiance_data)[source]¶
Compute the bi-hemispherical reflectance (a.k.a surface albedo) from radiosity and irradiance records.
- Parameters:
- Returns:
dict– A dictionary with the following content:albedoxarray.DataArrayAn albedo data array.
- eradiate.pipelines.logic.compute_bidirectional_reflectance(radiance_data, irradiance_data, calculate_stokes)[source]¶
Compute the spectral bidirectional reflectance distribution function (BRDF) and bidirectional reflectance factor (BRF) from radiance and irradiance records.
- Parameters:
- Returns:
dict– A dictionary with the following content:brdfxarray.DataArrayA BRDF data array.
brfxarray.DataArrayA BRF data array.
Notes
This function is expanded to
brdfandbrfnodes during pipeline assembly.
- eradiate.pipelines.logic.degree_of_linear_polarization(stokes_vec)[source]¶
Calculate the degree of linear polarization from a Stokes vector.
- eradiate.pipelines.logic.extract_irradiance(mode_id, illumination, spectral_grid)[source]¶
Derive an irradiance dataset from the irradiance spectrum of an illuminant, as well as solar angles.
- Parameters:
mode_id (
str) – Eradiate mode for which this pipeline step is to be configured.illumination (
Illumination) – The illuminant whose irradiance is to be evaluated.spectral_grid (
SpectralGrid) – Spectral grid driving the simulation.
- Returns:
dict– A dictionary with the following content:irradiancexarray.DataArrayA spectral irradiance data array compatible with processed spectral radiance records and suitable for deriving BRDF data.
solar_anglesxarray.DatasetorNoneIf relevant, a dataset holding solar angles.
Notes
This function is expanded into
irradianceandsolar_anglesnodes during pipeline assembly.
- eradiate.pipelines.logic.gather_bitmaps(mode_id, var_name, var_metadata, gather_variance, calculate_stokes, bitmaps, viewing_angles, solar_angles)[source]¶
Gather a collection of Mitsuba bitmaps into xarray data arrays.
- Parameters:
mode_id (
str) – Eradiate mode from which this pipeline step is configured.var_name (
str) – Name of the processed physical variable.var_metadata (
dict) – A metadata dictionary to be attached to the data array holding the processed physical variable.calculate_stokes (
bool) – Flag that specifies whether the variable is calculated as a Stokes vector or not.gather_variance (
bool) – Flag that specifies whether second moment bitmaps (for variance computation) should be gathered as well.bitmaps (
dict) – A dictionary mapping spectral loop indexes to the corresponding bitmap. Dictionary structure is as follows:{ <loop_index_0>: { "bitmap": <bitmap_0>, "spp": <sample_count_0>, }, <loop_index_1>: { "bitmap": <bitmap_1>, "spp": <sample_count_1>, }, ... }
viewing_angles (
Dataset, optional) – A dataset holding the viewing angles associated with each pixel in the processed bitmaps.solar_angles (
Dataset, optional) – A dataset holding the solar angles associated with the processed observation data.
- Returns:
data_vars (
dict[str,DataArray]) – A dictionary mapping data variable names to a corresponding data array. These can easily be aggregated into an xarray dataset, or scattered around other nodes of the post-processing pipeline.
- eradiate.pipelines.logic.moment2_to_variance(expectation, m2, spp, calculate_stokes)[source]¶
Calculate the variance (central 2nd moment) from the raw 2nd moment and expected value.
- Parameters:
expectation (
DataArray) – The data expected value. For Monte Carlo integration, this is the output of the estimator.m2 (
DataArray) – The data’s raw 2nd moment.spp (
DataArray) – The number of samples per pixels used during the Monte Carlo integration.calculate_stokes (
bool) – Specifies whether the expectation is a Stokes vector.
- Returns:
DataArray– The variance (central 2nd moment) of the data. The DataArray has the same shape as the expectation or m2.- Raises:
RuntimeError – If the
expectationandm2arrays have different dimensions (we don’t want unexpected broadcasting to happen).
Notes
This function calculates the Monte-Carlo variance, which aggregates the variance of each sample. It thus needs to be divided by the spp.
In polarized mode (
calculate_stokes=True), the m2 array only contains the ‘I’ (intensity) component, while the expectation may contain the full Stokes vector. The function automatically selects the ‘I’ component from expectation when needed.
- eradiate.pipelines.logic.radiosity(sector_radiosity)[source]¶
Aggregate sector radiosity into a full-hemisphere radiosity dataset.
- Parameters:
sector_radiosity (
DataArray) – Radiosity values for the hemisphere sector corresponding to each film pixel. This quantity is expressed in spectral flux units (typically W/m²/nm) and, when summed over the entire film, aggregates into a spectral radiosity (W/m²/nm).- Returns:
DataArray– A global radiosity record, with no film pixel / viewing angle indexing left.
- eradiate.pipelines.logic.spectral_response(srf)[source]¶
Evaluate a spectral response function as a data array.
- Parameters:
srf (
Spectrum) – The spectral function to be evaluated.- Returns:
DataArray– A data array mapping SRF values against the wavelength. To avoid confusion with the spectral coordinate, the wavelength dimension is here namedsrf_w.
- eradiate.pipelines.logic.viewing_angles(angles)[source]¶
Collect viewing angles associated with each film pixel from the measure, if relevant.
- Parameters:
angles (
ndarray) – An array with shape (m, n, 2) containing the processed measure’s viewing angles in degree. Typically obtained by callingDistantMeasure.viewing_angles().- Returns:
Dataset– An xarray dataset holding viewing angle values indexed by film coordinates.