Segmentation Tools

• Add node ▼, add a new node to the active tree. This will create a new point in the 3D space and connect it to the active point (shown in red) of the tree. The new node will become part of the active tree
• Assign active node ▼ allows you to assign the closest node to the position of the mouse click as the new active node. This can be useful when you want to change the active node to perform specific operations or modifications
• Connect to node ▼, connects active node to another existing node. This will create an edge between the active node and the selected node, establishing a connection between them
• Delete node ▼, this will remove the closest node to the position of the mouse click. The edges of the tree will be rearranged to prevent splitting of the tree
• Insert node after active ▼, allows you to insert a new node after the active node. This can be useful when you want to add a new node in a specific position within the tree
• Modify active node ▼, this will allow you to move the active node to a new position in the 3D space using a mouse click
• New tree ▼, add a new node and assign it to a new tree. This new tree will not be connected to any other trees, providing a separate structure
• Split tree ▼, if you want to split a tree at a specific point, you can use the Split tree option. This will delete the closest node to the position of the mouse click and split the tree at that point.

Use the [✓] Show lines checkbox to toggle visibility of the lines (edges) of the 3D line object displayed in the Image View panel.
Press the Table view button to open a window that displays tables describing the 3D lines. This can provide additional information and details about the structure of the lines (see below).

The table shows list of nodes and offers multiple actions via a popup menu:

The table shows list of edges; certain actions available via a popup menu:

• MATLAB format, *.lines3d, it is recommended to save the 3D lines in the matlab format!
• Amira Spatial graph, *.am, Amira-compatible format in binary or ascii form
• Excel format, *.xls, export Nodes and Edges tables to an Excel file

• The Annotation list button starts a window with the list of existing annotations. It is possible to load and save annotations to the main MATLAB workspace or to a file (matlab and excel formats). See more below.
• The Delete All button removes all annotations
• Precision edit box - specify number of digits after decimal point
• [✓] Show prompt checkbox, when ticked a dialog asking for label and value appear after addition of each annotation, otherwise the dialog is not shown and the label and field values are taken from the previously added annotation
• [✓] Focus on Value checkbox, when ticked, the first parameter during visualization of annotations is value and the second parameter is label.
• The Display as combobox - enables way how the annotations are displayed in the Image View panel.
  The following modes are available:
  
  • Marker, show only location of the annotation using a cross-marker
  • Label, show marker and label for each annotation
  • Value, show marker and value for each annotation
  • Label + Value, show marker, label and value for each annotation

• ^ Ctrl + Mouse wheel, change brush size
• None / ⇧ Shift + left mouse click, paint with brush
• ^ Ctrl + left mouse click, start eraser. The brush radius in the eraser mode could be amplifier using the Eraser, x edit box.
• The [✓] auto fill checkbox in the Selection panel allows to perform automatic filling of areas after use of brush

Radius, define radius of the brush tool in pixels
Eraser, x, define radius size multiplier when brush is in the eraser mode
Interpolation settings, press to start a dialog that allows to modify the interpolation settings. The settings can also be modified from the Preference dialog and the type can be switched using a dedicated button in the toolbar.
The [✓] Watershed checkbox, when ticked, pixels of the image are clustered using the watershed algorithm and can be selected as clusters (see the following section)
The [✓] SLIC checkbox, , when ticked, pixels of the image are clustered using the SLIC algorithm and can be selected as clusters (see the following section)

The Superpixels mode can be initiated by selecting the Watershed or SLIC checkbox. In the Superpixels mode the brush tool selects not individual pixels but rather groups of pixels (superpixels). While drawing the selection of the last superpixel can be undone by pressiong the Ctrl+Z shortcut.

The superpixels are calculated using the

The two additional edit boxed (N, Compact/Invert) offer possibility to modify the size of generated superpixels (see below). The values in the N edit box can be changed using the Ctrl+Alt + mouse wheel or Ctrl+Alt+⇧ Shift + mouse wheel shortcuts.

References:

The set of superpixels is individual for each magnification (especially in the SLIC mode). It is possible to define size of superpixels (the N edit box, default value 220 for SLIC, for Watershed use lower numbers) and compactness (the Compact edit box; higher compactness gives more rectangularly shaped superpixels). In the Watershed mode the Compact edit box is called Invert; when objects have dark boundaries over bright backround number in the Invert edit box should be 1 (or higher), if the objects have bright boundaries over dark background this number should be 0.

During drawing the boundaries of the superpixels for the SLIC mode are shown using the drawregionboundaries.m function written by Peter Kovesi Centre for Exploration Targeting, School of Earth and Environment, The University of Western Australia.

Note 1! The Superpixels mode is sensitive to the [✓] Adapt. checkbox in the Selection panel. When the [✓] Adapt. checkbox is selected the superpixels are selected based on the mean value of the initial selection plus/minus standard deviation within the same area multiplied by the factor specified in the Adapt. editbox of the Selection panel.

Note 2! While drawing the value in the Adapt. editbox may be changed using the mouse wheel.

Note 3! The function relies on slicmex.c that should be compiled for your operating system. Please refer to the Microscopy Image Browser System Requirements section for details.

Use the Low and High sliders and edit boxes to provide threshold values that will be used to select pixels with intensities between these values. The thresholding process is automatically initiated upon interaction with these widgets.
If the Masked area check box is selected the thresholding is performed only for the masked areas of the image, which is very convenient for local black and white thresholding.
Right mouse click above the threshold sliders opens a popup menu that allows to set precision for the slider movement.
The [✓] Adaptive checkbox starts adaptive threshoding with the Sensitivity and Width parameters modified by the scroll bars.

Usage notes
When doing thesholding for large 3D/4D datasets it is recommended to:

Start by	Modifier	[✓] 3D checkbox	Action
Left mouse click over an object	Ctrl	Unchecked	Move the selected object in 2D
Left mouse click over an object	Ctrl	Checked	Move the selected 3D object. Note! Only for MATLAB version 2017b and newer
Left mouse click	⇧ Shift	Unchecked	Move all objects on the shown slice, i.e. 2D movement
Left mouse click	⇧ Shift	Checked	Move all objects on all slices, i.e. 3D movement
Left/Right/Up/Down buttons	-	Unchecked	Move all objects on the shown slice by the specified number of pixels, i.e. 2D movement
Left/Right/Up/Down buttons	-	Checked	Move all objects on all slices by the specified number of pixels, i.e. 3D movement

Modes:

The Lasso tool works also in 3D.

The Magic Wand works also in the 3D mode (the 3D switch in the Selection panel).

Note! When the 3D switch in the Selection panel is enabled the Membrane Click Tracker tool connects points (linearly) in the 3D space (this may be very usefull for tracing microtubules). Alternatively for microtunules, consider 3D lines tool

The membrane is traced with help of Accurate Fast Marching function http://www.mathworks.se/matlabcentral/fileexchange/24531-accurate-fast-marching by Dirk-Jan Kroon, 2011, University of Twente.

Note! It is highly recommended to compile the corresponding function (see details in the Membrane Click Tracker section of the System Requirements page ), otherwise the function is very slow!

• 1. Left mouse click, selects object from the Mask/Model layers with a single mouse click.
  Note 1! Separation of objects is sensitive to the connected neighbourhood connectivity parameter from the Magic Wand tool; Note 2! In the 3D mode the function uses object statistics that is generated by pressing the Recalc. Stats button. If the Mask/Model layers have been changed press the Recalc. Stats button again.
• 2. Lasso: selects object with lasso tool (Press first None/⇧ Shift/Ctrl + Right mouse button, then hold the left mouse button while moving mouse around). Can also make selection for the whole dataset if the [✓] 3D checkbox in the Selection panel is checked.
• 3. Rectangle or Ellipse: these tools work in similar to the lasso tool manner but give rectangle or ellipsoid selection. Works also in 3D.
• 4. With the Polyline option the selection is done by drawing a polygon shape point by point. Start drawing with the right mouse button and finish with a double click or the right mouse click. Works also in 3D.
• 5. Use Brush to select some of the masked/model areas. Size of the Brush tool is defined in the Brush editbox.
• 6. Mask within selection with the left mouse click on the image make a new selection that is an intersection of the existing selection and the mask/model layer. With Ctrl+ mouse click the new selection is Selection -minus- Mask. This action is sensitive to the [✓] 3D checkbox state in the Selection panel.

Segment-anything model (SAM) is developed by Meta AI Research, FAIR research team. It can be used to segment individual objects or the whole image using one or few mouse click. Details of the research and interactive demo is available here: https://segment-anything.com
Implementation of SAM in MIB is done via utilization of an external Python interpreter, please check the Requirements and installation section below for detailed instructions.

Important! even though SAM can work on CPU, GPU is highly recommended as it is x30-60 faster.

List of original SAM available in MIB:

• vit_b (0.4Gb), fastest (x1) but gives less precise results
• vit_l (1.2Gb), moderate speed (~x1.4 slower), better predictions
• vit_h (2.5Gb), slowest (x2.0 slower), best predictions

Reference

• Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, Ross Girshick
  Segment Anything arXiv:2304.02643, https://doi.org/10.48550/arXiv.2304.02643

To extend application of SAM for microscopy images four additional pretrained networks for light and electron microscopy for tests are included:

• vit_b Electron Microscopy (0.4Gb), the lightest implementation for fast response of SAM trained on electron microscopy datasets
• vit_b Light Microscopy (0.4Gb), the lightest implementation for fast response of SAM trained on light microscopy datasets
• vit_h Electron Microscopy (2.5Gb), the extensive implementation for best results of SAM trained on electron microscopy datasets
• vit_h Light Microscopy (2.5Gb), the extensive implementation for best results of SAM trained on light microscopy datasets
• vit_b for EM Generalist (0.4Gb), pretrained for detection of mitochondria from EM datasets
• vit_b for EM Boundaries (0.4Gb), pretrained for detection of membrane encapsulated organelles from EM datasets

Reference

• Anwai Archit, Sushmita Nair, Nabeel Khalid, Paul Hilt, Vikas Rajashekar, Marei Freitag, Sagnik Gupta, Andreas Dengel, Sheraz Ahmed, Constantin Pape
  Segment Anything for Microscopy, bioRxiv, doi: https://doi.org/10.1101/2023.08.21.554208; link

Download links and configuration file
SAM networks are automatically connected to MIB; it is possible to configure any custom SAM network for use in MIB. To do so, please modify sam_links.json located by default under Resources subfolder within MIB. It is also possible to link this file from anywhere, in this case use segment-anything settings in MIB to provide a new location.

Segment-anything is not available by default and it needs to be installed!

Before the installation keep in mind the following requirements:

• MATLAB R2022a or newer (tested on R2022a, R2022b, R2023a)
• Python 3.8, 3.9, 3.10; tested on 3.9
• Here is a list of Python versions compatible with various MATLAB releases
• CUDA-compatible GPU is highly recommended, CPU can also be used but it is significantly slower

Installation details available from https://mib.helsinki.fi/downloads_systemreq_sam.html

SAM segmentation in MIB has 3 different options for use.

Interactive

It is a default operation mode of SAM in MIB. In this mode the segmentation is performed on the part of the image that is visible to the user. To start segmentation do following:

• Use the Destination ▼ dropdown to define the destination layer: selection, mask, model. Depending on this, the results of the segmentation procedure are automatically assigned to the selected layer
• Use the Mode ▼ dropdown to define the operation that needs to be applied to the segmentation results:
  • Replace, replace the objects of the destination layer with the new segmentation results
  • Add, add segmentation results to the objects of the selected destination layer
  • Subtract, subtract segmentation results from the objects of the selected destination layer
  • add, +next material, (only for models with 65535+ materials and destination: model). Add segmentation results to the selected material and add a new material to the model.
• Use left mouse click to specify a point on the image that belongs to an object to be segmented.
  If the object was not segmented with a single click:
  • ⇧ Shift + Left mouse click, expands the object so that the new area under the click is added to the object
  • Ctrl + Left mouse click to deselect the selected area from the object

Landmarks

This mode can be used to process full images. The idea that the key points are provided as annotations:

• Use the Dataset ▼ dropdown to define portion of the image stack to process
• Use the Destination ▼ and Mode ▼ dropdowns to define destination layer and the mode (see the Interactive mode above for details)
• Use Left mouse click to add annotations. The annotation value defines whether the annotation's location defines object (Value=1) or background (Value=0)
  
  Hints:
  • Hint 1: use Annotations ▼-> [✓] Focus on Value to focus the value instead of the annotation label
  • Hint 2: uncheck Annotations ▼-> [✓] show prompt to switch off the dialog asking for annotation label and value
  • Hint 3: use Ctrl + Left mouse click to remove the closest to the mouse click annotation
  • Hint 4: it is possible to access the list of annotation by pressing
• PressSegment to do segmentation

Automatic everything

In this mode, MIB is using SAM to segment all objects automatically and assigns them to a new model:

• Use the the Dataset ▼ dropdown to define portion of the image stack to process
• PressSegment to do segmentation

Press to open the settings dialog.
There are various settings that can be affected to finetune the segmetation process. Here the list of the most important parameters:

There are various settings that can be affected to finetune the segmetation process. Here the list of the most important parameters
• Backbone ▼, define a pretrained backbone to be used. There are 7 backbones of different sizes, the larger the size - the longer it takes to get predictions, but the quality of the prediction is better. See the General information and reference section above for details
• Execution environment ▼, define whether GPU (the cuda mode) or CPU should be used. The CPU mode is significantly slower than the GPU mode
• [✓] Show the progress bar in the interactive mode, uncheck to switch off showing of the progress bar during segmentation
• PATH to segment-anything installation, location where segment-anything package was unzipped. You can click [✓] Check to select path to segment-anything to open a directory selection dialog to specify this folder

Use the Radius, px edit box to specify the radius of the spot.
*Eraser, x*, specifies magnifier for the spot eraser tool

Works also in 3D.

Selection modifiers

• None/⇧ Shift + left mouse click, will add new spot to the existing ones
• Ctrl + left mouse click, will remove selection from the current