# Semantic Segmentation

In this example, we demonstrate the use of Tensorleap on a Computer Vision task - Semantic Segmentation with [**COCO**](https://cocodataset.org/) data. For training and validation **coco 14** data files were used, combined with a **MobileNetV2** `backbone` and a **pix2pix** based decoder.

In this example, a model is trained to segment images to three categories: `background`, `person` and `car`. After training, the Mean IoU for **person** and **car** was`0.309` and `0.262` respectively.

### Population Exploration

The Tensorleap platform tracks how each learned feature, within each layer, responds to each sample. From that information, it constructs a vector that captures how the model ***perceives*** each sample. This allows the platform to create a **similarity map** between samples as they are interpreted by the model. A more intuitive explanation would be that similar samples would activate similar learned features within the model.

The **Population Exploration** analysis generates a plot that represents the samples' similarity map based on the model's latent space, built using the extracted features from the trained model.\
An unsupervised clustering algorithm was used to define the colors of the dots. Moving the cursor along different areas reveals samples that have common features, for example:

{% tabs %}
{% tab title="Black & White Cluster" %}
![](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FZJcKjNI0Msz4XNDmcqrz%2Fblack-and-white-cluster.gif?alt=media\&token=64ec2679-978e-4a50-8eab-f0f42b8af002)
{% endtab %}

{% tab title="Ski Cluster" %}
![](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2Fp1B9aTGPqaQxh1TdtiFB%2Fsnow-cluster.gif?alt=media\&token=7405fefa-0496-4456-a921-8ad0067cdcde)
{% endtab %}

{% tab title="Surf Cluster" %}
![](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FyS13UOkmjU9djK7nV0a7%2Fsurfing-cluster.gif?alt=media\&token=0964ca3c-5bbe-4cb0-aa56-354e2aa162d4)
{% endtab %}

{% tab title="Tennis Cluster" %}
![](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2Fehe0IywJWuJiUugB3oNH%2Ftennis-cluster.gif?alt=media\&token=36a75d96-2230-436c-a85a-79c369f0aad2)
{% endtab %}
{% endtabs %}

### Cluster Analysis

Tensorleap's **Fetch Similars** tool is used to return a cluster of samples that are similar to a chosen sample. Next, we will analyze a few clusters from different areas of the model's latent space (presented in the **Population Exploration** plot).

#### **Tennis Cluster**

One of the clusters was a cluster of people playing tennis:

![Tennis Cluster](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FarSLdCC7Ojq4e7hvR5lW%2Fcoco-tennis.gif?alt=media\&token=06861b8a-579d-42ec-8d7c-b7efb4e2e29e)

The tool also provieds heat-maps of the samples that highlight common features within the cluster. In this cluster, for example, one of the highlighted features is the tennis rackets:

![](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FTqpB0lI0Co1LSZMCTkjh%2Fimage.png?alt=media\&token=43d5a84f-8b22-4339-8e3b-d6cbc440a4ab)![](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FkmdWvqKbJLdt80Na7V7L%2Fimage.png?alt=media\&token=c9919eec-5c98-47d8-a5ee-96d20003942f)![](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2F1hffmibgdesZ2CaPISe0%2Fimage.png?alt=media\&token=bd7abb81-670c-4e5d-b7ac-52bb6443eee2)

#### **B\&W cluster**

As seen in the **population exploration** plot above, the platform detected a cluster of gray images. Running the **Fetch Similars** tool resulted in these images:

![Fetch Similar Grayscale Cluster](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FgR2sUij1pWG5M7BDWLmk%2Fimage.png?alt=media\&token=c4420272-cf97-46eb-8073-f049d87c4a42)

From the fetched images we can see that this cluster indeed contains **grayscale** images but also **RGB** images with a small variation in [**hue**](https://en.wikipedia.org/wiki/Hue). Tensorleap's cluster analysis shows that the vast majority of samples are not **grayscale** images but in fact **RGB** images (plotted as red dots below). Moreover, comparing the model's performance on **grayscale** vs **RGB** images yields that on average, RGB images have lower error loss.

![RGB vs Grayscale Loss Comparison](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FKfxPkBg6VmH6ehcZv5Qp%2Fimage.png?alt=media\&token=1b138ff4-3fc4-429e-8f3f-897aaf083bb9) ![Grayed Cluster - RGB (red) and Grayscale (Blue)](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2Fhg5L8diWvIC2aevKvWV7%2Fimage.png?alt=media\&token=a01249e2-3922-4e9e-984f-a6dc96f64f80)

#### **'Vehicle-like' Clusters**

Our model's latent space includes multiple semantically meaningful vehicle clusters, for example, the **Motorcycles/Bicycles** and **Bus** clusters:

![Bus Cluster](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2Fa3uZT6tGqmDHC4YVcsh9%2Fimage.png?alt=media\&token=5232c120-7cf7-4a55-9f20-2545664fe52a)

We can see that the **Bus** cluster also includes quite a few buildings. Further examining the heat-maps of samples in this cluster indeed revealed model's attention not only to **bus** features, but also features from **building** and **towers**, as they share similar features. For example:

![](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2F1oW8JdmAYlCdJDhxijl7%2Fbus-building-3.gif?alt=media\&token=57c29cf5-ba1b-484f-8fbd-6caa84e6c227) ![Cluster-Defining Features Heat-Map](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2Ft7n2ralAxYvsBCYhZt7a%2Fbus-building-4.gif?alt=media\&token=92957cbc-ccd6-4afb-ad65-14379dd47c8b) ![](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FinvnBRDN6hc7nPb03vsb%2Fbus-building-2.gif?alt=media\&token=e1ec4878-bf68-4132-b17b-849121a72c3e)

**Motorcycles/Bicycles** cluster:

![Motorcycles/Bicycles Cluster](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2F4MsPmhMhQ1YvKKxNS8Nh%2Fimage.png?alt=media\&token=d20ebafa-396f-4fca-acf6-1116c8990ae4)

The features defining this cluster are mostly of the *wheels* and *rider*, as seen in these heat-maps:

![](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FR9emautSlrx7djzXWZzf%2Fimage.png?alt=media\&token=4534ee20-f245-44f8-bcbe-d0afca1b38f3) ![](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2F4WdXd1MPwUlVJTdSjsM1%2Fimage.png?alt=media\&token=be8b79be-6160-4d08-8957-78bd4142a442) ![](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2Fyg7Rua2gNawSUGaAi8bq%2Fimage.png?alt=media\&token=667a2c55-cf6c-41fc-bc46-ffaee36c4771) ![](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2Fin5jXYcNoXaXvhmsYkfZ%2Fimage.png?alt=media\&token=8b1fb861-c061-4c16-bbf6-17e528bdef4c)

### Vehicle SuperCategory Model

Analyzing the model on Tensorleap revealed that the model had a difficulty segmenting **cars** as a separate class from **trucks** and **busses** (which are labeled as **background**). One possible solution is to segment the entire **Vehicle** SuperCategory together, which will be reviewed in this section.

After training the model with the **Vehicle** SuperCategory the Mean IoU for **person** and **vehicle** is now **`0.319`**  and **`0.312`** respectively (compared to **`0.309`** for **person** and `0.262` for **cars** previously).

#### **Cluster Analysis**

When **Fetching Similars** to one of the **vehicle** samples, the result is a more homogeneous cluster composed of **cars** and **buses:**

![Vehicles Cluster (click-to-zoom)](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FUc2Zjqqn1B79xfCeogvI%2Fimage.png?alt=media\&token=80414586-8851-4ecb-9597-295da5ff3126)

The attention maps below show that the model is able to find strong, discriminative features in the analyzed cluster, such as **wheels**. In addition, it reveals a possible confusion as some round objects could be categorized as vehicles, due to their similarities to wheels.

![Vehicles Cluster Heat-Map (click-to-zoom)](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FSi7oopmkKjkBQGvyiHhA%2Fimage.png?alt=media\&token=1af650b5-8286-4408-86dd-b5cec330a119)

## Performance and Metadata Analysis

The Tensorleap Dashboard enables you to analyze how your data is distributed across various features. This enables us to identify trends and factors that might be correlated to the model's performance.

![Tensorleap Dashboard (click-to-zoom)](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FbAhvZWA57IlLs9yOzWnd%2Fimage.png?alt=media\&token=b8d731fd-8d84-4500-8ae3-b37c68f246b7)

The **Car Percent** and **Person Percent** represent the ratio between the number of pixels that are labeled as **car** or **person** and the total number of pixels in a sample. The histograms reveal a strong correlation between the two variables and the loss. Zooming in on the **Car Percent vs Loss** histogram, shows a significant increase in loss above \~50% **Car Percent**. One of the bars indicates a relatively high average loss for this bucket (and has \~80% **Car Percent):**

![High Loss Bar](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FKrjdJqvm5XqaucVKiIvT%2Fimage.png?alt=media\&token=65b710d2-2a01-41d3-8c06-f97e87ceccc3)

Examining this bucket we can see that it contains samples with close-up photos of cars that present very few features, or car interiors.

#### **Metrics Filtering**

All metrics dashboards in the Tensorleap platform are interactive, and can be filtered.

For example, we can further explore this bucket by filtering the view to show results only for samples belonging to it, updating the dashboard accordingly:

![Interactive Filtering of High Loss Bar (click-to-zoom)](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FKzIvAmMXF9S9yAbqDfqa%2Fcoco-filter-dashboard.gif?alt=media\&token=d93692fa-501d-4cab-a856-d4c35eff9cb3)

Exploring the validation samples table, we can see that only 6 samples are in that bucket (marked by the red box below):

![Filtered Dashboard (click-to-zoom)](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2F8C4FBrhTwaUMQ4jEHfbl%2Fimage.png?alt=media\&token=53686f53-b112-4f19-b9a6-1434abebe092)

​Examining the sample with the highest loss, we can see it is a close-up on the car's front (as expected due to the high **Car Percent**), and that there are two **cats** that were *mislabeled* as **car:**

![High Loss Sample](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FLZEpLxOnGOsFFdl0lYwm%2Fimage.png?alt=media\&token=c3c6a0f9-7e73-46a3-a0e3-fb4aa31722d5)

Next we'll present a few additional samples that were detected in a similar way.

## **Sample Analysis**

### **False and Ambiguous Labels**

The **Sample Analysis** tool allows us to detect ambiguous labels and mislabeled images. These can later be considered for exclusion in order to improve performance.

#### Examples of Mislabeling

On the right is the model's **prediction,** which correctly segmented all three people, while the Ground Truth includes just one person:

![Mislabeled Ground Truth](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FJz1xVXQiHcpqVETTT0ak%2Fimage.png?alt=media\&token=3c3def08-9037-4fc5-8533-69580fcfbd5b) ![Model's Prediction](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FWQIp2g5XAzKXIKfpPYJY%2Fimage.png?alt=media\&token=05873bf7-4e33-4527-91cb-1f4fa3bb7fbe)

Another example is a **person** labeled as **car**:

![Mislabeled Ground Truth](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FXPlrJsKdekdcl6cT32RV%2Fimage.png?alt=media\&token=6e8a1a64-185d-41a2-ae88-2ff6d3e9e308) ![Model's Prediction](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FFxUt0gteV83LcUntNxQb%2Fimage.png?alt=media\&token=445768e8-f918-44a3-b6d2-24292d97edf9)

#### Inaccurate Labeling

An example of a poor and inaccurate labeling of **persons** in the background and suitcases as **car**:

![Inaccurate Labeling Example (click-to-zoom)](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FDWGbIFmL5kIy6wIxv65G%2Fimage.png?alt=media\&token=40312faa-835c-4764-a801-bd1371f1a3d9) ![Suitcases Labeled as Car (click-to-zoom)](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FqqB5yAOpQ8Y9RWsRkQ5M%2Fimage.png?alt=media\&token=7b5efdc5-cec5-4c9a-8806-0a67609758e4)

#### Ambiguous Labeling

In the examples below, persons in a magazine were labeled as **persons** and a toy was labeled as **car**.

![Magazine as Person (click-to-zoom)](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FIAqfMFIh2ezLmJGvUeID%2Fimage.png?alt=media\&token=88a8756d-1846-47a4-81a4-f53789b51a15) ![Toy as Car (click-to-zoom)](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FdsqpePSPOjyJ0ZueaMkp%2Fimage.png?alt=media\&token=8983d7c3-deea-4d2f-be3c-0d2fb18710bb)

#### Challenging Images

Some samples in the dataset could be very challenging for the model. For example, on the left - a crowded image of people and on the right - a person in very low lighting conditions:

![Crowd (click-to-zoom)](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2FhoRMLCStcs2Rh3lg1Vjv%2Fimage.png?alt=media\&token=867ce902-7601-4766-88a4-23cb0ba6d03a) ![Low Light (click-to-zoom)](https://3509361326-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F9UXeOlFqlw8pl79U2HGU%2Fuploads%2F9qw7fsJClYs9y39fsy9l%2Fimage.png?alt=media\&token=61886fff-a9ee-4067-b849-ed3afce979a4)

## **Summary**

The **Tensorleap** platform provides powerful tools for analyzing and understanding deep learning models. In this example, we presented only a few examples of the types of insights that can be gained using the platform. 

For more information, see our additional [**Examples**](https://docs.tensorleap.ai/tensorleap-integration/writing-integration-code/examples) and [**Guides**](https://docs.tensorleap.ai/guides).