Custom Metrics

You can write your own custom metrics to be calculated and stored for later use within the Metrics Dashboard.

Tensorleap stores metrics and metadata for every sample. This allows the user to investigate certain populations within the dataset. This also means that the custom metric is calculated per prediction on a single sample.

This function can get multiple np.ndarray arrays as inputs which will be exposed in the Metric UI connections. These are of shape (batch, dim-1,...,dim-n). The function returns a np.ndarray that contains a batched metric result.

Example of usage:

@tensorleap_custom_metric(name='metrics',
                          direction=MetricDirection.Downward,
                          compute_insights=None)
def custom_metric_distances(
    y_true: npt.NDArray[np.float32],
    y_pred: npt.NDArray[np.float32]
) -> dict[str, npt.NDArray[np.float32]]:
    diff = y_true - y_pred
    axis = tuple(range(1, diff.ndim))

    return {
        "mean_difference": np.mean(diff, axis=axis),
        "mean_absolute_difference": np.mean(np.abs(diff), axis=axis)
    }

The @tensorleap_custom_metric decorator registers each metric collection into the Tensorleap integration.

Once this metric is parsed within the dataset script, it should be added to the model as a block.

This is done by:

1. right clicking on the Network view,

2. choosing Metric

3. Selecting the required Metric from the Node Details menu

4. Connecting the metric to the relevant blocks.

Adding a custom metric to the model