Custom Metrics

You can write your own custom metrics to be calculated and stored for later use within the Metrics Dashboard.

Tensorleap stores metrics and metadata for every sample. This allows the user to investigate certain populations within the dataset. This also means that the custom metric is calculated per prediction on a single sample.

This function can get multiple np.ndarray arrays as inputs which will be exposed in the Metric UI connections. These are of shape (batch, dim-1,...,dim-n). The function returns a np.ndarray that contains a batched metric result.

Example of usage:

@tensorleap_custom_metric(name='metrics',
                          direction=MetricDirection.Downward,
                          compute_insights=None)
def custom_metric_distances(
    y_true: npt.NDArray[np.float32],
    y_pred: npt.NDArray[np.float32]
) -> dict[str, npt.NDArray[np.float32]]:
    diff = y_true - y_pred
    axis = tuple(range(1, diff.ndim))

    return {
        "mean_difference": np.mean(diff, axis=axis),
        "mean_absolute_difference": np.mean(np.abs(diff), axis=axis)
    }

The @tensorleap_custom_metric decorator registers each metric collection into the Tensorleap integration, and can be added to the model analysis using the integration test.