sam — Segment-anything helpers

The +utils/+sam package holds the controller-independent helpers of the interactive segment-anything tools (segmentationSAM, segmentationSAM2).

Those tools read and write the layers in the block mode, i.e. only the part of the dataset that is currently shown, and they segment an object with a series of clicks. The state of the layers taken on the first click therefore has to survive a zoom, a pan or a new seeded slice; this package describes the shown block, re-maps a cached image between two blocks and serves the cached states to the segmenters.

It also holds the post-processing applied to what the model returns, currently the removal of the small low-confidence islands that appear beside the segmented object.

utils.sam.alignInitialImage(cachedImage, cachedBox, currentImage, currentBox)

ALIGNINITIALIMAGE - Re-map an image cached over one shown block onto another block.

Syntax:

alignedImage = utils.sam.alignInitialImage(cachedImage, cachedBox, currentImage, currentBox)

The interactive SAM tools cache the state of the destination layer as it was before the segmentation of the current object started, so that every refinement click can be merged with that initial state instead of with the result of the previous click. The cache is taken in the block mode and becomes invalid as soon as the user zooms, pans or seeds a new slice.

This function rebuilds the initial state for the new block: it starts from currentImage (the layer as it is right now, which outside of cachedBox was never touched by SAM and therefore still holds the initial state) and pastes cachedImage over the overlapping part. When the two blocks cannot be related to each other (different orientation or different pyramid level) currentImage is returned unchanged.

Input Arguments:

  • cachedImage - [numeric] image cached over cachedBox, [height, width, depth]

  • cachedBox - struct returned by utils.sam.blockBox() at the moment cachedImage was taken; can be empty

  • currentImage - [numeric] the destination layer fetched over currentBox

  • currentBox - struct returned by utils.sam.blockBox() for the current view

Output Arguments:

  • alignedImage - [numeric] currentImage with cachedImage pasted into the overlapping area; same size and class as currentImage

Usage:

Example 1 - restore the pre-SAM state after the view was zoomed out

currentBox = utils.sam.blockBox(dataset, [z1 z2]);
initialImage = utils.sam.alignInitialImage(cachedImage, cachedBox, currentImage, currentBox);
utils.sam.blockBox(dataset, zRange)

BLOCKBOX - Describe the currently shown block used by the SAM segmenters.

Syntax:

box = utils.sam.blockBox(dataset)
box = utils.sam.blockBox(dataset, zRange)

The interactive SAM tools read and write the destination layer in the block mode, i.e. only the part of the dataset that is currently visible in the image view. The extent of that block changes as soon as the user zooms or pans, so any image cached between two clicks has to be tagged with the block it was taken from. This function returns that tag; it mirrors the cropping performed by core.MibDataset.getData3D() when options.blockModeSwitch = true.

Input Arguments:

  • dataset - [core.MibDataset] dataset that is being segmented

  • zRange (optional) - [numeric] [zMin, zMax] range of slices of the block; when missing or empty, the currently shown slice is used

Output Arguments:

  • box - struct describing the shown block:

    • .x - [xMin, xMax] horizontal extent in the dataset coordinates

    • .y - [yMin, yMax] vertical extent in the dataset coordinates

    • .z - [zMin, zMax] range of slices

    • .orientation - orientation of the dataset the block was taken in

    • .magFactor - scaling between the dataset coordinates and the pixels of the fetched block: 1 for the memory-resident datasets and dataset.magFactor for the pyramidal (BigData, Virtual) datasets, where the block is fetched at the displayed pyramid level

Usage:

Example 1 - tag a cached image with the block it was taken from

box = utils.sam.blockBox(dataset, [z1 z2]);
utils.sam.initialImage(mibModel, dataset, cacheName, layerName, timePoint, materialIndex, getDataOptions, zRange, targetSize)

INITIALIMAGE - Return a layer of the dataset as it was before the current SAM interaction started.

Syntax:

initialImage = utils.sam.initialImage(mibModel, dataset, cacheName, layerName, timePoint, materialIndex, getDataOptions, zRange)
initialImage = utils.sam.initialImage(mibModel, dataset, cacheName, layerName, timePoint, materialIndex, getDataOptions, zRange, targetSize)

The interactive SAM tools segment an object with a series of clicks: the first click creates the object and each Shift/Ctrl-click re-runs the model with an extra positive/negative seed. Both the layer the results are added to and the material the results are restricted to have to be taken as they were before the first click, because every run overwrites what the previous one produced:

  • 'initialImageAddTo' - the destination layer; merging with its current state instead would keep the discarded parts of the previous run and make the negative seeds useless

  • 'initialImageSelected' - the “fix selection to material” mask; using its current state instead would exclude the area already taken by the object itself (the first click moved those pixels out of the restricting material), so every second click would erase the object and the next one would bring it back

Those states are cached in mibModel.sessionSettings.SAMsegmenter by controllers.MibImageDocument.gui_WindowButtonDownFcn on the first click, together with the shown block they were taken from (field <cacheName>Box, see utils.sam.blockBox()). A cache is used as-is while the block is unchanged; after a zoom, a pan or a new seeded slice it is re-mapped onto the current block by utils.sam.alignInitialImage() and, when the block has grown, stored back so that the following clicks keep the pre-interaction state over the wider area.

Input Arguments:

  • mibModel - [models.MibModel] the model, keeps the caches in sessionSettings

  • dataset - [core.MibDataset] dataset that is being segmented

  • cacheName - [char] name of the cached field: 'initialImageAddTo' or 'initialImageSelected'

  • layerName - [char] layer to fetch: 'selection', 'mask' or 'labels'

  • timePoint - [numeric] index of the time point to fetch

  • materialIndex - [numeric] index of the material to fetch

  • getDataOptions - struct with options for models.MibModel.getData3D(); .blockModeSwitch and .z are set by this function

  • zRange - [numeric] [zMin, zMax] range of slices of the current run

  • targetSize (optional) - [numeric] size of the segmentation result the returned image has to be combined with; the image is cropped or zero-padded to it. Guards against the one-pixel differences between the image and the model blocks of the pyramidal datasets

Output Arguments:

  • initialImage - [uint8] the requested layer as it was before the current SAM interaction started, cropped to the shown block over zRange, [height, width, depth]

Usage:

Example 1 - merge the results of the current run with the initial state

initialImage = utils.sam.initialImage(obj.mibModel, dataset, 'initialImageAddTo', 'labels', t, selMaterialIndex, getDataOpt, [z1 z2], size(imgDataset));
obj.mibModel.setData3D({bitor(initialImage, imgDataset)}, 'labels', t, dataset.orientation, selMaterialIndex, getDataOpt);

Example 2 - restrict the results to the selected material

materialMask = utils.sam.initialImage(obj.mibModel, dataset, 'initialImageSelected', 'labels', t, selectedFixToMaterial, getDataOpt, [z1 z2], size(imgDataset));
imgDataset = bitand(imgDataset, materialMask);
utils.sam.removeSmallRegions(mask, minArea)

REMOVESMALLREGIONS - Drop the small islands SAM returns beside the real object.

Syntax:

mask = utils.sam.removeSmallRegions(mask, minArea)

Both segment-anything predictors binarize a continuous field of mask logits at a hard zero, so any structure elsewhere in the view that scores just above it is promoted to full membership with the same weight as the object under the seed. In practice the object comes back at a logit of several units while these islands sit at 0.3-0.8, which makes them small: a handful of pixels of the low-resolution mask the model predicts before it is upsampled to the size of the shown block. An area threshold separates the two cleanly, whereas raising the logit threshold enough to lose the islands also eats into the object and eventually breaks it apart.

Components are measured slice by slice, matching the per-2D-mask semantics of min_mask_region_area in the automatic mask generator, which receives the same preference. Note that for pyramidal (BigData, Virtual) datasets SAM works on the block at the displayed resolution, so the threshold counts pixels of the shown image and not of the full-resolution dataset.

Input Arguments:

  • mask - [numeric | logical] segmentation mask as [height, width] or [height, width, depth]; non-zero pixels are treated as foreground

  • minArea - [numeric] smallest connected component to keep, in pixels. 0, negative and empty all disable the filter and return mask unchanged

Output Arguments:

  • mask - the input mask with every 8-connected component smaller than minArea set to zero; class and pixel values of the surviving components are preserved

Usage:

Example 1 - filter the output of an interactive SAM prediction

minRegionArea = obj.mibModel.preferences.SegmTools.SAM2.min_mask_region_area;
imgOut = utils.sam.removeSmallRegions(imgOut, minRegionArea);