LogoLogo
  • Tensorleap
  • Examples
    • Semantic Segmentation
    • Image Analysis
    • Sentiment Analysis
    • MNIST Project Walkthrough
    • IMDB Project Walkthrough
  • Quickstart using CLI
  • Guides
    • Full Guides
      • MNIST Guide
        • Dataset Integration
        • Model Integration
        • Model Perception Analysis
        • Advanced Metrics
      • IMDB Guide
        • Dataset Integration
        • Model Integration
        • Model Perception Analysis
        • Advanced Metrics
    • Integration Script
      • Preprocess Function
      • Input Encoder
      • Ground Truth Encoder
      • Metadata Function
      • Visualizer Function
      • Prediction
      • Custom Metrics
      • Custom Loss Function
      • Custom Layers
      • Unlabeled Data
      • Examples
        • CelebA Object Detection (YoloV7)
        • Wikipedia Toxicity (using Tensorflow Datasets)
        • Confusion Matrix
        • CelebA Classification (using GCS)
  • Platform
    • Resources Management
    • Project
    • Dataset
    • Secret Manager
    • Network
      • Dataset Node
      • Layers
      • Loss and Optimizer
      • Visualizers
      • Import Model
      • Metrics
    • Evaluate / Train Model
    • Metrics Dashboard
    • Versions
    • Issues
    • Tests
    • Analysis
      • helpers
        • detection
          • YOLO
    • Team management
    • Insights
  • API
    • code_loader
      • leap_binder
        • add_custom_metric
        • set_preprocess
        • set_unlabeled_data_preprocess
        • set_input
        • set_ground_truth
        • set_metadata
        • add_prediction
        • add_custom_loss
        • set_visualizer
      • enums
        • DatasetMetadataType
        • LeapDataType
      • datasetclasses
        • PreprocessResponse
      • visualizer_classes
        • LeapImage
        • LeapImageWithBBox
        • LeapGraph
        • LeapText
        • LeapHorizontalBar
        • LeapImageMask
        • LeapTextMask
  • Tips & Tricks
    • Import External Code
  • Legal
    • Terms of Use
    • Privacy Policy
Powered by GitBook
On this page
  • Training
  • Metrics and Analytics
  • Population Exploration
  • Sample Root Cause Analysis
  • Summary

Was this helpful?

  1. Examples

MNIST Project Walkthrough

PreviousSentiment AnalysisNextIMDB Project Walkthrough

Last updated 2 years ago

Was this helpful?

The MNIST project, included in the Free Trial, uses the (handwritten digits) with a simple classification model.

In the following steps, you will train the model, view the analytics, and perform basic analyses. For a more in-depth guide to the MNIST use-case, see the .

To open the project, in the Welcome screen, go to Projects and click MNIST:

You can zoom in and out using the scroll wheel, and pan by dragging the background. Click on a node to view its details.

Training

The pre-saved projects provided are not yet trained. They must be trained before viewing the analytics and analyses.

Once training has initiated, a PENDING notification will appear indicating that the training process is initializing. This could take a minute or so. Once the training begins you will see a STARTED notification.

Metrics and Analytics

  • Loss (error) vs Batch - decreases as training progresses

  • Accuracy vs Batch - increases as training progresses

  • List of samples ordered by loss, from high to low

  • Image Mean (a.k.a brightness) vs Loss - validation samples with high loss values tend to fail

  • Distance from the Class Centroid vs Loss (the class centroid is the averaged image of all the images within the class) - the more unique the samples are, the more they tend to fail

Population Exploration

Tensorleap creates a latent space that is relatively close to the entire model's latent space. This latent space is composed of feature activations from all of the model's layers in order to distribute the data in the most informative way.

Using this information, Tensorleap creates a similarity map between samples as they are interpreted by the model. A more intuitive explanation would be that similar samples would activate similar learned features within the model.

To find patterns in the similarity map that correlate with selected metrics and metadata, perform the following steps (also shown in the video below):

Perform the following steps only after training has finished

  1. Resize the Population Exploration analysis panel, by dragging the bottom right corner.

Notice that clusters are formed when projecting the samples onto the latent space. The clusters represent how the model perceives the data. When coloring the dots according to the ground truth label, we can see that each cluster correlates to a class. Large dots represent high loss values for these samples. Hover over the dots to see a preview of the sample.

Note how the model misinterpreted some of the samples - shown by the large dots and multi-colored samples within clusters.

Tensorleap analyzes the sample and displays the results in the Analyzer panel. This process should take a few moments. The next section describes the Sample Analysis.

Sample Root Cause Analysis

In the example shown below, a sample from the validation subset is analyzed. The horizontal bar graph reveals that the ground truth for the sample is 8, while the model interprets it as 6.

Next, the Loss Analysis shows a heat-map of the features that were correlated with the wrong prediction. In this case, a corner on the top left.

Moreover, you can access the heat-maps corresponding to each of the output classes by clicking the heat-map on the right.

Summary

Congratulations on completing this short MNIST walkthrough for the Free Trial. This is a rather simple use-case and model, which achieves more than 98% accuracy on the Validation subset.

For more information about the Standalone Trial, see Quickstart Standalone Trial.

The tab displays the model's nodes and connections in a simple convolutional neural network (CNN) model:

The orange node on the left relates to the MNIST dataset (the script can be viewed in the ).

The light blue nodes seen in the center of the model represent the model's layers, and the colored nodes at the end of the model represent the .

The model includes Tensorleap nodes (dark blue) to extract visualizations from different outputs.

To train the model for one epoch, on the top bar, click and in the training dialog:

Training will take somewhere between 10 to 45 minutes, depending on your system resources. You can track the status by clicking .

To display the model's analytics on the dashboard, on the top left of the dashboard, click to open the Versions view. Expand the version and make sure that the current model is selected, as shown in the video below:

This includes the following :

This similarity map is called a analysis, and it is performed automatically after each epoch.

On the top left click .

Color the dots by their ground-truth label, clicking , where the loss is currently selected, and change it to (ground truth label).

To learn more about the analysis, see the .

Click each sample to show its preview, metadata, and metrics. To analyze a sample, select it and click on the right panel, as shown below:

The Sample Analysis tool runs explainability algorithms on selected samples and displays the visualizations correlated with the blocks.

Next, you can follow the full guide at , which takes you through dataset integration, model building, and importing, as well as reviewing and analyzing additional metrics.

You can also check our for more advanced use cases.

To learn about integrating your custom data into the platform, see the .

Network
Resources Management
Loss and Optimizer
Visualizer
Visualizer
MNIST Guide
Full Guides
Integration Script
MNIST database
MNIST Guide
Dashboard
Full Guides
Population Exploration
Population Exploration
Dashlets
Training (click-to-zoom)
Metrics and Analytics
Population Exploration (click-to-zoom)
Sample Analysis
Ground Truth
Prediction
Sample Loss Analysis