Preprocess Function

The preprocessing_func (custom name) is a preprocess function that is called just once before the training/evaluating process. It prepares the data for later use in input encoders, output encoders, and metadata functions.

from code_loader.contract.datasetclasses import PreprocessResponse
from code_loader.inner_leap_binder.leapbinder_decorators import tensorleap_preprocess

@tensorleap_preprocess()
def preprocessing_func() -> List[PreprocessResponse]:
    ...
    train = PreprocessResponse(sample_ids=list(train_df.index), data=train_df, state=DataStateType.training)
    val = PreprocessResponse(sample_ids=list(val_df.index), data=val_df, state=DataStateType.validation)
    test = PreprocessResponse(sample_ids=list(test_df.index), data=test_df, state=DataStateType.test)
    unlabeled = PreprocessResponse(sample_ids=list(unlabeled_df.index), data=unlabeled_df, state=DataStateType.unlabeled)
​
    return [train, val, test, unlabeled]

The @tensorleap_preprocess decorator registers the preprocess function into the Tensorleap integration.

This function returns a List of PreprocessResponse objects. The elements on that list correspond to the train , validation, test and unlabeled data slices.

For a successful Tensorleap integration, supplying a train and validation set is mandatory, the rest is optional. Refer to the Code Integration section for complete usage instructions.

It is mandatory to return at least a training and a validation set.