Expert interface

Introduction

The expert interface provides advanced users with fine-grained control over Mitsuba scene dictionaries and parameter maps. By granting total access to all Mitsuba primitives and Eradiate’s scene parameter update system, it allows to integrate custom materials and surfaces into Eradiate experiments.

The expert interface is the tool to use to create complex 3D scenes, ranging from a single-material textured digital elevation model to complex canopies or urban scenes.

This guide serves as an introduction and basic tutorial for the expert interface.

Warning

This is an advanced feature, be sure to check the prerequisites before diving into it.

Prerequisites:

• Familiarity with Eradiate’s radiometric kernel interface.

• Basic understanding of Mitsuba 3’s scene representation (see the Editing a scene tutorial).

Fundamentals

The core concepts are introduced as part of the radiometric kernel interface guide. The expert interface grants the user to possibility to do manually what the basic interface automates: create a kernel dictionary, and define a scene parameter update map template that will be rendered automatically during the parametric loops. In practice, the defined kernel dictionary and scene parameter maps are merged with those created automatically by an Experiment.

The steps are as follows:

1. Create a kernel dictionary template.[1]

2. Create a kernel scene parameter map template.

3. Initialize an Experiment with the additional scene definitions.

What makes using this interface slighlty more challenging than the basic interface is that it requires some knowledge of some issues commonly encountered when building Mitsuba scenes. We will see how to address them.

Documentation references

• Kernel dictionary setup:

  • Mitsuba scene language

  • Mitsuba plugin reference

  • Eradiate plugin reference

• Parameter definitions:

  • Mitsuba’s Editing a scene tutorial

  • Mitsuba plugin reference

  • Eradiate plugin reference

  • API documentation: SceneParameter, KernelSceneParameterFlags, SearchSceneParameter

• Experiment entry points:

  • kdict and kpmap parameters of the EarthObservationExperiment constructor.

First usage example

Let us start with a simple example in which we will create an AtmosphereExperiment and add, at the ground level, a surface patch with Lambertian reflectance. We will then texture it with a coloured checkerboard pattern.

We import Numpy, Mitsuba, Matplotlib, Eradiate and a few Eradiate components to use them later:

import eradiate
import matplotlib.pyplot as plt
import mitsuba as mi
import numpy as np

from eradiate import KernelContext
from eradiate.experiments import AtmosphereExperiment
from eradiate.kernel import SceneParameter, SearchSceneParameter
from eradiate.xarray.interp import dataarray_to_rgb

We start by activating an Eradiate mode:

eradiate.set_mode("mono")

We will base our example on the AtmosphereExperiment class, which will produce the background atmosphere. We set the following parameters for it:

surface = None  # Suppress the default surface: we will add our own
atmosphere = {"type": "molecular"}
illumination = {"type": "directional"}
camera = {  # Nadir-looking camera at 3 m altitude
    "type": "perspective",
    "origin": [0, 0, 3],
    "target": [0, 0, 0],
    "up": [0, 1, 0],
    "film_resolution": (128, 128),
    "srf": {"type": "delta", "wavelengths": 550.0},
}

To check that our background atmosphere setup works, we can render the camera’s view and see if we get non-zero values (due to atmospheric scattering):

exp = AtmosphereExperiment(
    surface=surface,
    atmosphere=atmosphere,
    illumination=illumination,
    measures=camera,
)
eradiate.run(exp)["radiance"].squeeze().plot.imshow();

Adding a shape to the scene

Now, let’s add a square surface in the field of view of our camera. We assign to it a diffuse material (i.e. Lambertian reflection):

kdict={
    # We use numerical prefixes in object IDs to emphasize sectioning and
    # ordering.

    # First, materials (i.e. surface scattering models).
    # The 'id' field of this material must be defined because we reference it
    # later in the geometric shape definition. It is best to keep it consistent
    # with the material's dictionary key.
    "01_mat_diffuse": {"type": "diffuse", "id": "01_mat_diffuse"},

    # Then, the geometric shape (the [-1, 1]² square).
    # The previously defined material is referenced by its ID.
    "02_shape_square": {
        "type": "rectangle",
        "bsdf": {"type": "ref", "id": "01_mat_diffuse"},
    },
}

We can then inject this definition in the experiment using the kdict parameter.

exp = AtmosphereExperiment(
    surface=surface,
    atmosphere=atmosphere,
    illumination=illumination,
    measures=camera,
    kdict=kdict,
)
eradiate.run(exp)["radiance"].squeeze().plot.imshow();

Changing the reflectance

At this point, the surface is visible, and it has the default reflectance value (0.5). To change this, we have to declare a scene parameter update protocol to the experiment using a scene parameter update map template. The first step towards this is to declare how the reflectance of the diffuse material behaves as a function of the wavelength: to do so, we create a callable with signature f(ctx: KernelContext) -> float.

def eval_reflectance(ctx: KernelContext):
    return 1.0

We can now use this callable to declare a scene parameter.

kpmap = {
    # This key is arbitrary; we however choose to keep it consistent with
    # the expected scene parameter path.
    "01_mat_diffuse.reflectance.value": SceneParameter(
        eval_reflectance,
        # To accommodate the relative unpredictibility of the mechanism used by
        # Mitsuba to expose its scene parameters, we have to hint Eradiate at
        # where we expect to find the parameters in the Mitsuba scene graph.
        # Here, we tell it that the parameter is expected to be held by a
        # BSDF object with ID '01_mat_checkerboard', and, relative to that
        # node, it should have the path 'reflectance.value'.
        search=SearchSceneParameter(
            node_type=mi.BSDF,
            node_id="01_mat_diffuse",
            parameter_relpath="reflectance.value",
        ),
    )
}

Now, we can render the scene again and see that the reflected radiance is different from the previous time:

exp = AtmosphereExperiment(
    surface=surface,
    atmosphere=atmosphere,
    illumination=illumination,
    measures=camera,
    kdict=kdict,
    kpmap=kpmap,
)
eradiate.run(exp)["radiance"].squeeze().plot.imshow();

Texturing the reflectance

We can now complexify the example by texturing the reflectance against space coordinates. The solution we will implement now is just one of several ways to achieve this—not the most optimal, but straightforward and interesting to demonstrate a more advanced expert interface usage pattern.

For this, we need to redefine the BSDF assigned to the surface so that its reflectance is textured, i.e. varies with space coordinates:

kdict={
    "01_mat_checkerboard": {
        "type": "diffuse",
        "id": "01_mat_checkerboard",
        "reflectance": {
            # For additional information about the `bitmap` plugin,
            # see the Mitsuba documentation
            "type": "bitmap",
            "filter_type": "nearest",  # Use nearest neighbour interpolation between pixels
            "wrap_mode": "clamp",
            "raw": True,  # Do not apply colour processing
            "data": np.full((1, 1, 1), 0.5),  # Arbitrary init buffer with the right number of dimensions (3)
        },
    },
    "02_shape_checkerboard": {
        "type": "rectangle",
        "bsdf": {"type": "ref", "id": "01_mat_checkerboard"},
    },
}

Now, we can write the new parameter update protocol and declare it:

def eval_checkerboard(ctx: KernelContext):
    value_0 = 0.2
    value_1 = 0.8
    checkerboard = np.indices((16, 16)).sum(axis=0) % 2
    return np.atleast_3d(np.where(checkerboard == 1, value_0, value_1))

kpmap = {
    "01_mat_checkerboard.reflectance.data": SceneParameter(
        eval_checkerboard,
       search=SearchSceneParameter(
            node_type=mi.BSDF,
            node_id="01_mat_checkerboard",
            parameter_relpath="reflectance.data",
        ),
    )
}

We can now render the scene once more to see the result:

exp = AtmosphereExperiment(
    surface=surface,
    atmosphere=atmosphere,
    illumination=illumination,
    measures=camera,
    kdict=kdict,
    kpmap=kpmap,
)
eradiate.run(exp)["radiance"].squeeze().plot.imshow();

Applying spectral parameters

Finally, we can apply a spectral dependency to the checkerboard cell reflectance. For that purpose, we will write a new version of the eval_checkerboard() callable that will take the spectral dependency of the reflectance of the checkerboard cells into account. We will also need to update our camera definition so that we simulate all three RGB channels.

# Define colours
BLUE = {440.0: 0.9, 550.0: 0.1, 660.0: 0.1}
RED = {440.0: 0.1, 550.0: 0.1, 660.0: 0.9}

# Declare convenience functions that will evaluate colours
# based on contextual wavelength
def blue(ctx: KernelContext):
    return BLUE[ctx.si.w.m_as("nm")]


def red(ctx: KernelContext):
    return RED[ctx.si.w.m_as("nm")]

# Define a new parameter update protocol for the checkerboard
# texture that accounts for spectral dependencies
def eval_checkerboard_spectral(ctx: KernelContext):
    value_0 = blue(ctx)
    value_1 = red(ctx)
    checkerboard = np.indices((16, 16)).sum(axis=0) % 2
    return np.atleast_3d(np.where(checkerboard == 1, value_0, value_1))

# Use the new spectral checkerboard update protocol to update the
# checkerboard texture parameters
kpmap = {
    "01_mat_checkerboard.reflectance.data": SceneParameter(
        eval_checkerboard_spectral,
       search=SearchSceneParameter(
            node_type=mi.BSDF,
            node_id="01_mat_checkerboard",
            parameter_relpath="reflectance.data",
        ),
    )
}

# Update the sensor to simulate 3 wavelengths instead of 1
camera_spectral = {
    "type": "perspective",
    "origin": [0, 0, 3],
    "target": [0, 0, 0],
    "up": [0, 1, 0],
    "film_resolution": (128, 128),
    "srf": {"type": "delta", "wavelengths": [440.0, 550.0, 660.0]},
}

We can now render the scene again and convert the created xarray data array to a Numpy array suitable for image display:

exp = AtmosphereExperiment(
    surface=surface,
    atmosphere=atmosphere,
    illumination=illumination,
    measures=camera_spectral,
    kdict=kdict,
    kpmap=kpmap,
)
img = dataarray_to_rgb(
    eradiate.run(exp)["radiance"],
    channels=[("w", w) for w in [660., 550., 440.]]
)
plt.imshow(img);

Best practices

• Scene parameter search. Careful definition of scene parameter search strategies is important as it will make scene parameter update definitions more robust to hard-to-predict Mitsuba scene parameter naming changes.

• Scene dictionary organization and plugin naming. Section the scene dictionary by object type, ordering them according to their dependencies (e.g. BSDFs → shapes). Use numeric prefixes to enforce ordering. Make sure scene dictionary keys are consistent with object IDs.

• Split experiment run. Invoke the Experiment.init() and Experiment.process() methods separately to get a better understanding of performance bottlenecks.

• Object positioning. Objects added with the expert interface are positioned in the kernel’s reference frame, using kernel default units. The reference point depends on the geometry: the kernel’s (0, 0, 0) point will match the surface level only with a plane-parallel geometry and 0 m surface elevation. It is recommended to define a reference positioning point (e.g. (0, 0, 0) in plane-parallel geometry, or (0, 0, EARTH_RADIUS.m_as('m')) in spherical-shell geometry) and position objects relative to it.

Troubleshooting

• Missing or unresolved scene parameters. This is a common issue that occurs when a scene parameter search fails. This is usually due to incorrect parameters when defining a SearchSceneParameter. This will, for instance, manifest as spectral properties not changing with wavelength. To solve this:

  1. Instantiate the Experiment and call Experiment.init() with drop_parameters=False. Find the parameter you want to update in the Experiment.mi_scene.parameters table.

  2. Call Experiment.mi_scene.drop_parameters() . Check if your target parameter is still there. If not, it means the parameter search is not correctly configured.

• Mitsuba plugin configuration. Being primarily a rendering system, Mitsuba makes some scene pre-processing and default configuration choices that might not be suitable for Earth observation applications. For instance:

  • Pay attention to texture interpolation methods. You will likely want nearest neighbour, but defaults are usually bi/trilinear.

  • Pay attention to automatic colour processing. In doubt, it is generally safer to pass an explicit spectrum type rather than a more concised, but interpreted scalar value. For instance, 1.0 will be interpreted as {"type": "rgb", "value": (1.0, 1.0, 1.0)}, not as {"type": "uniform", "value": 1.0} (which is what we usually want).

• Performance issues. Large parameter updates (e.g. big textures) can become a performance bottleneck, especially if repeated unnecessarily. For instance, in CKD mode, only one update per spectral bin is needed for surface parameters; but, by default, updates will occur at every iteration of the spectral loop. To reduce the amount of unnecessary computation, it can be beneficial to implement simple caching in the update protocols.

---

[1]

In practice, kernel dictionary templates will very rarely have rendered parameters and will mostly be regular Mitsuba scene dictionary fragments.