Interactive, GPU-accelerated 3D segmentation

Summary

In this tutorial, we will learn how to combine two napari-threedee plugins with napari-accelerated-pixel-and-object-classification (napari-apoc) to perform interactive 3D segmentation of nuclei. This demonstrates how the interoperatibility of napari-threedee plugins allows users to create human in the loop workflows. You will use the render plane manipulator plugin to view arbitrary slices of a 3D image of nuclei and then label annotator to paint labels in 3D for ground truth annotations for the napari-apoc segmentation algorithm. We use the napari-apoc GPU-accelerated random forrest classifier to predict which pixels in the image belong to nuclei usign only a few sparse labels.

Prerequisites

To perform this tutorial, you need the following:

Basic familiarity with napari.
A computer with a working conda/mamba installation. We will use conda/mamba to manage the environment we install the dependencies into.

Setup

Before trying the tutorial, please initialize your python environment as described below. Note that we recommend using mamba to install/manage dependencies. If you are using conda, you can simply replace mamba with conda in the commands below, but we recommend you that the conda version is >23.10

Create a new environment called seg-env with python 3.9. In your terminal, enter the following command.
```
mamba create -n seg-env -c conda-forge python=3.9
```
Activate your new environment. If you successfully activated the environment, you should see the name of your environment ((seg-env)) to the left of your prompt.
```
mamba activate seg-env
```

Install the dependencies.

mamba install -c conda-forge napari pyqt napari-threedee napari-accelerated-pixel-and-object-classification

If your are on Mac OS, install this openCL wrapper (used by APOC for GPU acceleration):
```
mamba install -c conda-forge ocl_icd_wrapper_apple
```
If you are on linux, install this openCL wrapper (used by APOC for GPU acceleration):
```
mamba install -c conda-forge ocl-icd-system
```

Instructions

Activate the environment you created during the setup above. If you successfully activated the environment, you should see the name of your environment ((seg-env)) to the left of your prompt.
```
mamba activate seg-env
```
Launch napari from the command line using the command below. The first time you launch napari, it may take a few moments to open.
```
napari
```
Open the 3D cells sample image from the File menu. File -> Open Sample -> napari builtins -> Cells (3D+2Ch). This will load two image layers: DAPI-stained nuclei and the cell membranes.
Delete the membrane layer. First select the membrane layer from the layer list and then click the "garbage can" button.
Create a new label layer by clicking the new label layer button above the layer list. This will add a new layer called "Label".
Switch to 3D rendering mode. Click the rending toggle (square button in the lower left corner of the viewer. You can scroll up/down in the canvas to zoom in/out and click and drag to rotate the view.

7. Switch the depiction to "plane". This will render only one plane from the 3D volume. First select the "nuclei" layer from the layer list. Then change the depiction to "plane" in the layer controls.
Use the napari-threedee render plane manipulator to move the rendering plane and explore the image. Open the render plane manipulator from the plugin menu: Plugins -> napari-threedee -> render plane manipulator. Click the "activate" button the start the manipulator. Then use it to rotate the rendering plane to find a plane of interest. Note: you can double-click with the mouse to re-position the manipulator on the plane.
Next, we will use the label annotator to paint labels directly on the rendering plane. We will use these labels to train a random forest classifier to segment the nuclei. Open the label annotator plugin from the plugins menu: Plugins -> napari-threedee -> label annotator. Activate the label annotator by clicking the "activate" button.
Switch to painting mode. First select the labels layer by clicking on the "Label" layer in the layer list. Then click on the paint brush icon in the layer controls.
We will label the image as either background or nuclei so that we can train the random forest classifier to predict if each pixel is belongs to the background or a nucleus. We will start by labeling some pixels as background by giving them the value of 1. Random forest classifiers can be trained with small numbers of annotations. You can label some background pixels by clicking and dragging on the rendered plane.
Next, we will add some nuclei labels. First change the label index from 1 to 2 in the layer controls (upper left hand corner of the window). Then paint some of the nuclei. Like with the background, a couple of lines is sufficient.
We can also add labels to other planes. To do so, select the "nuclei" layer and use the render plane manipulator to select a new plane. Then re-select the "Label" layer and paint as you did before. Remember that label value 1 is used for background and label value 1 is used for nuclei.
Finally, we can train the random forest classifier. Open the napari-APOC semantic segmentation plugin: Plugins -> napari-accelerated-object-and-pixel-classification -> Semantic Segmentation. Note: for the other two widgets, you can click the middle, eye icon to hide them. First, select the "nuclei" layer in the "Select images (channels) used for training" box at the top. Then select the "Label" layer in the "Select ground truth annotation" combobox in the lower portion of the widget. Finally, click the "Train" button—we will leave the other parameters as defaults. When the training has completed, the result will be added as a new layer called "Result of PixelClassifier". Notice that it is hard to see the segmented nuclei because both the background and nuclei predictions are shown together.
We can view just the predicted nuclei labels by selecting the nucleus label value (2) in the label box in the layer controls and then clicking the "show selected" checkbox. Note that we have obtained a pretty good segmentation with just a few labels. You can also compare the segmentation result to the raw image by toggling visibility of the different layers.
If you are not happy with your segmentation, you can return to the labeling and training steps above to add more labels and improve the segmentation algorithm.