Deep MIB - segmentation using Deep Learning
The deep learning tool (Deep MIB) provides access to training of deep convolutional networks over the user data and utilization of those networks for image segmentation tasks.
Back to Index --> User Guide --> Menu --> Tools Menu
Contents
Overview
For details of deep learning with DeepMIB please refer to the following tutorials:
DeepMIB: 2D U-net for image segmentation
DeepMIB: 3D U-net for image segmentation
DeepMIB, features and updates in MIB 2.80
The typical workflow consists of two parts:
- network training
- image prediction
The pretrained network is saved to disk and can be distributed to predict (the Predict tab) unseen datasets.
Please refer to the documentation below for details of various options available in MIB.

Network panel
The upper part of Deep MIB is occupied with the Network panel. This
panel is used to select one of the available architectures.
Always start a new project with selection of the architecture:
- 2D U-net, is a convolutional neural network that was
developed for biomedical image segmentation at the Computer Science
Department of the University of Freiburg, Germany. Segmentation of a 512
x 512 image takes less than a second on a modern GPU (Wikipedia)
References:
- Ronneberger, O., P. Fischer, and T. Brox. "U-Net: Convolutional Networks for Biomedical Image Segmentation." Medical Image Computing and Computer-Assisted Intervention (MICCAI). Vol. 9351, 2015, pp. 234-241 (link)
- Create U-Net layers for semantic segmentation (link)
- 2D SegNet, is a convolutional network that was
developed for segmentation of normal images University of Cambridge, UK.
It is less applicable for the microscopy dataset than U-net.
References:
- Badrinarayanan, V., A. Kendall, and R. Cipolla. "Segnet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation." arXiv. Preprint arXiv: 1511.0051, 2015 (link)
- Create SegNet layers for semantic segmentation (link) - 3D U-net, a variation of U-net, suitable for for semantic
segmentation of volumetric images.
References:
- Cicek, Ö., A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger. "3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation." Medical Image Computing and Computer-Assisted Intervention, MICCAI 2016. MICCAI 2016. Lecture Notes in Computer Science. Vol. 9901, pp. 424-432. Springer, Cham (link)
- Create 3-D U-Net layers for semantic segmentation of volumetric images (link) - 3D U-net anisotropic, a hybrid U-net that is a combination of
2D and 3D U-nets. The top layer of the network has 2D convolutions and 2D
max pooling operation, while the rest of the steps are done in 3D. As
result, it is better suited for datasets with anisotropic voxels
This button is only available when either the Train or Predict tab is selected.
- When the Train tab is selected, press of the this button defines a file for saving the network
- When the Predict tab is selected, a user can choose a file with the pretrained network to be used for prediction. For ease of navigation the button is color-coded to match the active tab
define execution environment for training and prediction
- Name of a GPU to use, the dropdown menu starts with the list of available GPUs to use. Select the one that should be used for deep learning application
- Multi-GPU, use multiple GPUs on one machine, using a local parallel pool based on your default cluster profile. If there is no current parallel pool, the software starts a parallel pool with pool size equal to the number of available GPUs. This option is only shown when multiple GPUs are present on the system
- CPU only, do calculation using only a single available CPU
- Parallel, use a local or remote parallel pool based on your default cluster profile. If there is no current parallel pool, the software starts one using the default cluster profile. If the pool has access to GPUs, then only workers with a unique GPU perform training computation. If the pool does not have GPUs, then training takes place on all available CPU workers instead

Directories and Preprocessing tab
This tab allows choosing directories with images for training and prediction as well as various preprocessing parameters. During preprocessing the images and model files are processed and converted to a mibImg format that is used for training and prediction. However, in some situations, preprocessing step can be omitted. In this case, DeepMIB will used with original image files (see below for details).
Image files used in DeepMIB workflows can be arranged in 3 different ways:
- Preprocessing mode, this mode is enabled when
Preprocess for: Training and prediction option is
selected. This mode is recommended for the general use. The scheme below
demonstrates organization of directories compatible with this mode. For
the legend see a figure below.
-
Without preprocessing, when datasets are manually split into training
and validation sets, the modes without preprocessing may be
beneficial in situations when the input patch size of the network is
matching the image size in files. This option is suitable, when the files
were manually split into the training and validation sets.
-
Without preprocessing, with automatic splitting of datasets into training and validation sets,
the modes without preprocessing may be
beneficial in situations when the input patch size of the network is
matching the image size in files. In this mode, the files are randomly
split into the train and validation sets. The split is done upon press of
the Preprocess button, when Preprocess for: Split for training and validation

- Directory with images and models for training [used
only for training]:
use these widgets to select directory that contain images and model to be used for training. For the organization of directories see the schemes above.
For 2D networks the files should contain individual 2D images, while for 3D networks individual 3D datasets.
The extension dropdown menu on the right-hand side can be used to specify extension of the image files. The Bio checkbox toggles standard or Bio-format readers for reading the images. If the Bio-Format file is a collection of image, the Index edit box can be used to specify an index of the file within the container
- Important notes:
- Number of model or mask files should match the number of image files (with one exception for 2D networks, where it is allowed to have a single model file in MIB *.model format, when Single MIB model file: ticked)
- For models in standard image formats it is important to specify number of classes including the Exterior into the Number of classes edit box
- Important! It is not possible to use numbers as names of materials, please name materials in a sensible way!
Tip! if you have only one segmented dataset you can split it into several datasets using Menu->File->Chopped images->Export operation.
- Directory with images for prediction
use these widgets to specify directory with images for prediction (in the schemes above 2_Prediction). The image files should be placed under Images subfolder. Optionally, when the ground truth models for prediction images is available, it can be placed under Labels subfolder.
When the preprocessing mode is used the images from this folder are converted and saved to 3_Results\Prediction images directory.
For 2D networks the files should contain individual 2D images, while for 3D networks individual 3D datasets.
When the ground truth models are present, they are also processed and copied to 3_Results\PredictionImages\GroundTruthLabels. These models can be used for evaluation of results (see the Predict tab below for details).
The extension dropdown menu on the right-hand side can be used to specify extension of the image files. The Bio checkbox toggles standard or Bio-format readers for reading the images.
- Single MIB model file checkbox, (only for 2D networks) tick it, when using a single model file with segmentations
- Model extension dropdown, (only for 2D networks) is used to select extension of files containing models. For 3D network MIB model format is used.
- Number of classes edit box, (TIF or PNG formats only) is used to define number of classes (including Exterior) in models. For model files in MIB *.model format, this field is not used.
- Use masking checkbox is used when some parts of the training
data should be excluded from training. The masks may be provided in various formats
and number of mask files should match the number of image files.
Note! masking may give drop in precision of training due to inconsistency within the image patches. - Mask extension is used to select extension for files that contain masks. For 3D network only MIB *.mask format is supported
- Directory with resulting images
use these widgets to specify the main output directory; results and all preprocessed images are stored there.
All subfolders inside this directory are automatically created by Deep MIB:
Details of directories (see also the Legend figure above)
- PredictionImages, place for the prepocessed images for prediction
- PredictionImages\GroundTruthLabels, place for ground truth models for prediction images, when available
- PredictionImages\ResultsModels, the main outout directory with generated models after prediction. The 2D models can be combined in MIB by selecting the files using the Shift+left mouse click during loading
- PredictionImages\ResultsScores, folder for generated prediction scores (probability) for each material. The score values are scaled between 0 and 255
- ScoreNetwork, for accuracy and loss score plots, when the Export training plots option of the Train tab is ticked and for storing checkpoints of the network after each epoch, when the Save progress after each epoch checkbox is ticked
- TrainImages, images to be used for training (only for preprocessing mode)
- TrainLabels, models accompanying images to be used for training (only for preprocessing mode)
- ValidationImages, images to be used for validation during training (only for preprocessing mode)
- ValidationLabels, models accompanying images for validation (only for preprocessing mode)
- Compress processed images checkbox, tick to compress the processed images.
The processed images are stored in *.mibImg format that can be loaded in MIB.
*.mibImg is a variation of standard MATLAB format and can also be directly loaded into MATLAB
using similar to this command: res = load('img01.mibImg, '-mat');.
Compression of images slows down performance! - Compress processed models, tick to compress models during preprocessing.
The processed models are stored in *.mibCat format that can be loaded in MIB (Menu->Models->Load model).
It is a variation of a standard MATLAB format, where the model is encoded using categorical class of MATLAB.
Compression of models slows down performance but brings significant benefit of small file sizes - Use parallel processing, when ticked DeepMIB is using multiple cores to process images. Number of cores can be specified using the Workers edit box. The parallel processing during preprocessing operation brings significant decrease in time required for preprocessing.
- Fraction of images for validation, define fraction of images that will be randomly (Random generator seed) assigned into the validation set. When set to 0, the validation option will not be used during the training
- Random generator seed, number to initialize random seed generator, which defines how the images for training and validation are split. For reproducibility of tests keep value fixed
- Preprocess for, select mode of operation upon press of the Preprocess button. Results of the preprocessing operation for each mode are presented in schemes above
Train tab
This tab contains settings for generating deep convolutional network and training. Before processing further please finish the preprocessing part, see above.
Before starting the training process it is important to check and if needed modify the settings. Also, use the Network filename button in the Network panel to select filename for the resulting network.
- Input patch size, this is important field that has to be defined
based on available memory of GPU, dimensions of the training dataset and
number of color channels.
The patch size defines dimensions of a single image block that will be directed into the network for training. The dimensions are always defined with 4 numbers representing height, width, depth, colors of the image patch (for example, type "572 572 1 2" to specify a 2D patch of 572x572 pixels, 1 z-slice and 2 color channels).
The patches are taken randomly from the volume/image and number of those patches can be specified in the Patches per image field. - Padding defines type of the convolution padding, depending on
the selected padding the Input patch size may have to be adjusted.
- same - zero padding is applied to the inputs to convolution layers such that the output and input feature maps are the same size
- valid - zero padding is not applied to the inputs to convolution layers. The convolution layer returns only values of the convolution that are computed without zero padding. The output feature map is smaller than the input feature map.
- Number of classes - number of materials of the model including Exterior, specified as a positive number
- Encoder depth - number of encoding and decoding layers of the network. U-Net is composed of an encoder subnetwork and a corresponding decoder subnetwork. The depth of these networks determines the number of times the input image is downsampled or upsampled during processing. The encoder network downsamples the input image by a factor of 2^D, where D is the value of EncoderDepth. The decoder network upsamples the encoder network output by a factor of 2^D.
- Patches per image - specifies number of patches that will be taken from each an image or a 3D dataset. According to our experience, the best strategy is to take 1 patch per image and train network for larger number of epochs. When taking a single patch per image it is important to set Training settings (Training button)->Shuffling->every-epoch). However, fill free to use any sensible number
- Mini Batch Size - number of patches processed at the same time by the network. More patches speed up the process, but it is important to understand that the resulting loss is averaged for the whole mini-batch. Number of mini batches depends on amount of GPU memory
- Filters - number of output channels for the first encoder stage, i.e. number of convolitional filters used to process the input image patch during the first stage. In each subsequent encoder stage, the number of output channels doubles. The unetLayers function sets the number of output channels in each decoder stage to match the number in the corresponding encoder stage
- Filter size - convolutional layer filter size; typical values are in the range [3, 7]
- Activation layer - specifies type of the activation layers of
the network. When the layer may have additional options, a settings
button on the right-hand side becomes available.
- reluLayer - Rectified Linear Unit (ReLU) layer, it is a default activation layer of the networks, however it can be replaced with any of other layer below
- leakyReluLayer - Leaky Rectified Linear Unit layer performs a threshold operation, where any input value less than zero is multiplied by a fixed scalar
- clippedReluLayer - Clipped Rectified Linear Unit (ReLU) layer performs a threshold operation, where any input value less than zero is set to zero and any value above the clipping ceiling is set to that clipping ceiling
- eluLayer - Exponential linear unit (ELU) layer performs the identity operation on positive inputs and an exponential nonlinearity on negative inputs
- tanhLayer - Hyperbolic tangent (tanh) layer applies the tanh function on the layer inputs
- Segmentation layer - specifies the output layer of the
network; depending on selection a settings button on the right-hand side
becomes available to bring access to additional parameters.
- pixelClassificationLayer - semantic segmentation with the crossentropyex loss function
- focalLossLayer - semantic segmentation using focal loss to deal with imbalance between foreground and background classes. To compensate for class imbalance, the focal loss function multiplies the cross entropy function with a modulating factor that increases the sensitivity of the network to misclassified observations
- dicePixelClassificationLayer - semantic segmentation using generalized Dice loss to alleviate the problem of class imbalance in semantic segmentation problems. Generalized Dice loss controls the contribution that each class makes to the loss by weighting classes by the inverse size of the expected region
- dicePixelCustomClassificationLayer - a modification of the dice loss, with better control for rare classes
- Input layer settings button - can be used to specify data normalization during training, see the info header of the dialog or press the Help button of the dialog for details
- Training settings button - define multiple parameters used for training, for details please refer to
trainingOptions
function
Tip, setting the Plots switch to "none" in the training settings may speed up the training time by up to 25%
- Check network button - press to preview and check the network.
The standalone version of MIB shows only limited information about the
network and does not check it:
- Augmentation - augment data during training. For small training sets
augmentation provides an easy way to extend amount of training data using
various filters. Depending on the selected 2D or 3D network architecture
a different sets of augmentation filters is available. These operations are configurable
using the 2D and 3D settings buttons right under the
augmentation checkbox.
There are 17 augmentation operations for 2D network and 5 augmentation operations for 3D networks. It is also possible to specify fraction of images that have to be augmented.
2D augmentations specified with 2 or 3 values, where the last value defines probability of each particular augmentation to be triggered. When the augmentation is defined with 2 values, the first value specifies whether it is on (==1) or off (==0), alternatively, the first two values define the variation range, a random number will be picked between these numbers and used as the parameter for the filter. Each specific augmentation may be turned off either by setting its probability (the last value) to 0 or by setting its variation range to be as shown in the "off=[x,x,x]" text
The 2D augmentation settings can be reset to default by pressing Options tab->Reset 2D augmentation button
or disabled by pressing Options tab->Disable 2D augmentation button
- Preview button is used to preview input image patches that are
generated using augmentor. It is useful for evaluation of augmenter
operations and understanding performance. Number of patches to show and
tweaking of various additional settings are possible by pressing the
Settigns button on the right-hand side of the Preview button.
- Save progress after each epoch when ticked Deep MIB stores training checkpoints after each epoch to 3_Results\ScoreNetwork directory. It will be possible to choose any of those networks and continue training from that checkpoint. If the checkpoint networks are present, a choosing dialog is displayed upon press of the Train button
- Export training plots when ticked accuracy and loss scores are saved to 3_Results\ScoreNetwork directory. Deep MIB uses the network filename as a template and generates a file in MATLAB format (*.score) and several files in CSV format
- Random seed set a seed for random number generator used during initialization of training. Use 0 for random initialization each time or any other number for reproducibility
To start training press the Train button highlighted under the panel. If a network already existing under the provided Network filename it is possible to continue training from that point (a dialog with possible options appears upon restart of training).
Upon training a plot with accuracy and loss is shown; it is possible to stop training at any moment by pressing the Stop or Emergency brake buttons. When the emergency brake button is pressed DeepMIB will stop the training as fast as possible, which may lead to not finalized network in situations when the batch normalization layer is used.
Please note that by default DeepMIB is using a custom progress plot. If you want to use the progress plot provided with MATLAB (available only in MATLAB version of MIB), navigate to Options tab->Custom training plot->Custom training progress window: uncheck
The plot can be completely disabled to improve performance: Train tab->Training->Plots, plots to display during network training->none
The right bottom corner of the window displays used input image and model patches. Display of those decrease training performace, but the frequency of the patch updates can be modified in Options tab->Custom training plot->Preview image patches and Fraction of images for preview. When fraction of image for preview is 1, all patches are shown. If the value is 0.01 only 1% of patches is displayed.

After the training, the network is saved to a file specified in the Network filename editbox of the Network panel.
Predict tab
The trained networks can be loaded to Deep MIB and used for prediction of
new datasets.
To start with prediction:
- select a file with the desired network in the Network filename editbox of the Network panel. Upon loading, the corresponding fields of the Train panel will be updated with the settings used for training of the loaded network
- specify correct directory with the images for prediction: Directories and Preprocessing tab -> Directory with images for prediction
- specify directory for the results: Directories and Preprocessing tab -> Directory with resulting images
- press the Preprocess button to perform data preprocessing
- finally switch back to the Predict tab and press the Predict button
- Overlapping tiles, available for the same
convolutional padding, during prediction crops the edges of the predicted
patches, which improves the segmentation, but takes more time. See comparison of results on the image below:
- Explore activations
Activations explorer brings the possibility for detailed evaliation of the network. The images processed images should be located in 3_Results\PredictionImages directory.
Here is the description of the options:
- Image has a list of all preprocessed images for prediction.
Selection of an image in this list will load a patch, which is equal to
the input size of the selected network
The arrows on the right side of the dropdown can be used to load previous or next image in this list - Layer contains a list of all layers of the selected network. Selection of a layer, starts prediction and acquiry of activation images
- Z1, X1, Y1, this spinners make possible to shift the patch across the image. Shifting of the patch does not automatically update the activation image. To update the activation image press the Update button
- Patch Z, change the z value within the loaded activation patch, it is used only for 3D networks
- Filter Id, change of this spinner brings various activation layers into the view
- Update press to calculate the activation images for the currently displayed patch
- Collage press to make a collage image of the current network
layer activations:
- Image has a list of all preprocessed images for prediction.
Selection of an image in this list will load a patch, which is equal to
the input size of the selected network
- Load images and models press this button after the prediction to open original images and result of the segmentation in the currently active buffer of MIB
- Load prediction scores press to load the resulting score images (predictions) into the currently active buffer of MIB
- Evaluate segmentation when the datasets for prediction are accompanied with
ground truth models ( requires a model file in the directory with Prediction images,
it is important that the model materials names match those for the training data!).
- Press the button to calculate various precision metrics
- As result of the evaluation a table with the confusion matrix will be
shown. The confusion matrix displays how well the predicted classes are
matching classes defined in the ground truth models. The values are
scaled from 0 (bad) to 100 (excelent):
- In addition, it is possible to calculate occurrence of labels and Sørensen-Dice similarity coefficient in the
generated and ground truth models. These options are available from a
dropdown located in the right-bottom corner of the Evaluation
results window:
- The evaluation results can be exported to MATLAB or
saved in MATLAB, Excel or CSV
formats to 3_Results\PredictionImages\ResultsModels directory, see more in the
Directories and Preprocessing section above.
For details of the metrics refer to MATLAB documentation for evaluatesemanticsegmentation function
- Press the button to calculate various precision metrics
Options tab
Some additional options and settings are available in this tab
Config files panel
This panel brings access to loading or saving Deep MIB config files.
The config files contain all settings of Deep MIB including the network name and input and output directories but excluding the actual trained network. Normally, these files are automatically created during the training process and stored next to the network *.mibDeep files also in MATLAB format using the *.mibCfg extension.
Alternatively, the files can be saved manually by pressing the Save button.
Custom training plot
Settings for the custom training progress plot showing accuracy and loss during training.- Custom training progress plot, when checked the custom training plot is used, when unchecked a standard MATLAB training plot is displayed (the standard MATLAB plot is only available for MATLAB version of MIB)
- Refresh rate, update the plot after the specified number of interations. Adjust the value to improve performace, when with larger values the plot will be updated more rare, improving performance of training.
- Number of points, number of points that is used to show the plot. Decrease the value to improve drawing performance, increase the value to see more points
- Preview image patches, when checked the custom training plot will show input image and model patches for evaluation. Plotting of patches decrease performance, it is possible to use Fraction of images for preview to specify fraction of patches that have to be shown.
- Fraction of images for preview, specify fraction of input patches that are shown in the custom training plot. Use 1 to see all input patches and small values to see only a fraction of those. For example, the value 0.01 specifies that only 1% of image patches are displayed in the progress plot window
Other buttons
- Reset 2D augnentation, press to reset 2D augmentation settings to default values
- Export network to ONNX, (only MATLAB version of MIB, requires installation of ONNX Model Format support package) converts the network file to ONNX format. Please note that some of networks can't be converted yet.
Back to Index --> User Guide --> Menu --> Tools Menu