Settings

This describes the settings page of the Tensorleap platform

The Tensorleap settings page

The Tensorleap setting page:

• Controls the limits of Tensorleap use of available compute (1,2,3 in the figure above)

• Controls several global properties (4)

Adjusting Tensorleap compute limits

The tensorleap compute limits is usually set by modifying the required and limit threshold for the kubernetes pods that are running the different Tensorleap processes.

The main settings

This control the main engine settings:

This settings effects all processes except the Population Exploration and Fetch Similar processes

• The required and limit of the CPU per process run

• The required and limit memory per process run

• The number of GPUS per run (default - 1)

• The number of visualizer processes workers. This would increase and decrease according to need dynamically after run starts

The worker settings

• CPU limits

• Worker memory limits

• Number of worker processes. This would increase and decrease according to need dynamically after run starts

This number serves as an initial estimate - once the platform is running a specific process - it would start with this nukber of worker processes and then scale it down or up as needed

Pop Explorer settings

• CPU limits

• Memory limits

• time-to-live

In case OOM is encountered when running a process, this is usually caused by a memory limit that is too low, causing the pods that runs the process to be evacuated.

If you're encountering OOMs, it is recommended to increase either the main memory limit or the pop-explorer memory limit to a larger limit and re-run the process.

To verify that you're error is a result of an OOM - the pod describe within the relevant run logs should show an "OOM" status if the error reason is an OOM.

Another issue (for very large datasets > 3M samples) might be time-to-live in the population exploration. If you see an error starting roughly after time-to-live seconds, increase this limit.

Global Tensorleap properties

This view allow you to control several global Tensorleap settings:

• Build dynamic dependencies: When this flag is turned on, if a dataset parse includes a new requirement file, a virtual environment is build to be used within the dataset loading. When it is turned off, a default environment is being used.

The dynamic dependencies flag is usually needed for when integrating a complete repository with specific dependencies into Tensorleap.

• internal pip server: In case your orginazation has an internal pip server that you are using, you can fill its details within the pip index URL to cause the system to use it for dependency installation

• Keep visualization resolution: by default, some of the visualizer change image resolution to preserve space working memory when displaying these elements. If you want to prevent this, turn this flag on

• Enable warmup: Turning this flag would ensure a short pending time for new processes in the platform.