Evaluate Process
This describes the evaluate process
Overview
Once the Evaluate button is clicked , a new evaluate process will initiate.
This process will infer your model on all of the provided data, and run Tensorleap analysis on top of it. It:
Initializes the Preprocess from the integration script.
Iteratively loads all of your samples, feeds them to the model and calculate all of your metrics, metadata and losses
Extracts a latent space representation from each sample
Uses the Tensorleap engine to estimate values on unlabeled data and bulk labelling suggestion
Resuming an evaluate process
The evaluate process is stateful, so if it failed mid-run due to server instability or any other, non reproducible error, it is possible to continue the evaluation.
To continue a terminated or Failed evaluation, within the runs and processes click the relevant Evaluate job, and click continue.
Common Run Issues:
Evaluate mostly fail due to these reasons:
An evaluation was initiated before Assets validation and assets are not valid. It is recommended to validate the assets first (and see common validation issues) - and make sure that the connected code integration is the intended script to use.
Only a small amount of the samples are able to be parsed using the integration script. Since the Asset Validation only verify a successful ingest of the first sample in the dataset, it might be that other samples return invalid inputs (i.e. different shapes, types) that would cause the evaluate to fail. It is recommended to run the integration test on a larger amount of samples to verify validity and ensure all Tensorleap decoders are static - i.e. return the same shape and type for every sample. It is highly advised to ensure metadata return flat dictionaries for every sample with the same type per value.
OOM in evaluate. please adjust the limits in the settings page.
Server shutdown in the middle of evaluate or any other instability. It is possible to resume the evaluate.
Last updated
Was this helpful?