AI-Assisted Segmentation in Medical Imaging: Practical Steps and Case Studies

The official announcement from 3DSlicer states that Nvidia AI-assisted segmentation is now available in the latest Slicer preview version (rev28686 or higher). This extension is developed by Nvidia and provided by the core Slicer developers. While there are other AI-assisted segmentation modules in Slicer (such as DeepInfer , TOMAAT , SlicerCIP ), the newly added feature uses Nvidia Clara, which is a toolkit with strong industrial support and sufficient openness for researchers.

With the rise of deep learning research in medical imaging, more effective and improved methods are being developed to achieve AI-assisted workflows through GPU-supported modes. With the latest NVIDIA Clara application framework, developers have the necessary tools and APIs for AI development and for integrating AI capabilities into medical devices through embedded hardware.

The video below shows MRI brain tumor and liver segmentation, as well as fully automated liver, tumor, and spleen segmentation:

How Does It Work?

The input images (and input points in the case of guided segmentation) are sent to a computer equipped with an Nvidia GPU running the Linux operating system and Nvidia Clara software. The server uses the selected AI model to compute the segmentation and sends the results back to Slicer for display and further processing.

The Official Demo Server Set Up by PerkLab

is to make it easier for Slicer users to get started without setting up their own processing computer. We have uploaded several AI models developed by Nvidia. We provide these models and processing services as-is, and we do not guarantee the quality of this service (validity of segmentation results, speed, server uptime, etc.). No patient information will be sent to the processing server, and processed images and results will be deleted from the server; however, users need to ensure compliance with their data management guidelines while using our server. If there are any confidentiality issues, publicly available images can be used for testing: see Slicer’s sample data module or TCIA browser extension, or download from other sites such as Medical Decathlon.

If Anyone Wants to Share Their AI Model for Segmentation

As long as the model is compatible with Nvidia Clara, it should be possible to install the model on the server and make it available to the Slicer community.

Nvidia AI-assisted segmentation is available in [3D Slicer], a popular free open-source medical image visualization and analysis application. The tool is available in the application’s “Segment Editor” module.

The tool has two modes:

Fully Automated Segmentation: No user input required. In the “Automatic Segmentation” section, select the model and click “Start.” The segmentation process may take a few minutes (for large datasets on computers with slower network upload speeds, this process may take about 5-10 minutes).

Boundary Point-Based Segmentation: Requires the user to specify input points near the edges of the structure of interest, one on each side. The segmentation usually takes less than a minute.

Example results of automated segmentation:

Setup

Download and install the latest 3D Slicer Preview version (4.11.x) from http://download.slicer.org/.

Launch 3D Slicer and then open the Extension Manager.

Install the NvidiaAIAssistedAnnotation extension (in the “Segmentation” category), wait for the installation to complete, and then click “Restart”.

Optional: Set the annotation server settings in the menu: Edit/Application Settings/NVidia. If the server address is empty, the default public server will be used. Availability of the server or quality of the provided models cannot be guaranteed.

To set up your own segmentation server, follow these instructions.

A mid-range desktop computer with a powerful NVidia GPU (for example, an older computer with 8GB of memory and RTX 2070 GPU) is sufficient for running pre-trained models. The server needs to run a Linux operating system.

The computer running 3D Slicer has no special requirements for running AI-assisted segmentation. For large datasets, a fast network upload speed is recommended, as segmentation usually only takes a few seconds, while image data transfer can take several minutes.

Tutorials and Examples

Boundary Point-Based MRI Brain Tumor Segmentation:

Go to the Sample Data module and load the “MRBrainTumor1” dataset.

Go to Segment Editor.

Create a new Segment.

Click the “Nvidia AIAA” tool.

In the “Boundary Point-Based Segmentation” section, select “annotation_mri_brain_tumors_t1ce_tc” (a model trained to segment tumors on contrast-enhanced brain MRI).

Click the “Place Markers” button, then click around the tumor edges on all six sides in the slice view, and click “Start” (if a popup appears about sending image data to a remote server, click “OK”).

• In about 10 seconds, the automated segmentation result will appear in the slice view. To display the result in 3D, click the “Show 3D” button above the segmentation list. To display the slice images in 3D, click the “pin” icon in the upper left corner of the slice view, then click the “eye” icon.

Boundary Point-Based Segmentation of the Liver on CT:

Go to the Sample Data module and load the “CTACardio” dataset.

Go to Segment Editor.

Create a new segmentation.

Double-click the segment name and enter “liver” to specify the segmentation content.

Click the “Nvidia AIAA” effect in the “Boundary Point-Based Segmentation” section.

Optional: Click the “Filter” icon to list only those models containing “liver” in their names.

Select “annotation_ct_liver” (trained to segment the liver in portal venous phase CT images).

Click the “Place Markers” button, then click around the liver edges on all six sides in the slice view, and click “Start”.

• 

• In about 30 seconds, the automated segmentation result will appear in the slice view. To display the result in 3D, click the “Show 3D” button above the Segment Editor list.
  To adjust previously placed boundary points after segmentation is complete, click the “Edit” icon next to the “Start” button.

Fully Automated Segmentation of the Liver and Tumor on CT:

Download the Task03_Liver imagesTr liver_102.nii.gz dataset from http://medicaldecathlon.com/ and load it into 3D Slicer.

Go to Segment Editor.

Click the “Nvidia AIAA” effect.

In the “Automatic Segmentation” section, select the “segmentation_ct_liver_and_tumor” model and click “Start”.

The automated segmentation result should be displayed within 3-5 minutes.

Optional: Go to the Segments module to edit display settings.

Advanced

For local setups or computers with fast upload speeds, disable compression in “Application Settings” / “NVidia” (as the time spent compressing may exceed the time saved by transferring less data).

To filter models based on labels on the tags, set the label text in the “Models” section of the Nvidia AIAA effect user interface.

For Developers, This Plugin Can Be Downloaded and Installed Directly from GitHub:

git clone https://github.com/NVIDIA/ai-assisted-annotation-client.git

Open 3D Slicer: Go to Edit -> Application Settings -> Modules -> Other Module Paths

1. Add a new module path: <FULL_PATH> / slicer-plugin / NvidiaAIAA

2. Restart

To build the extension package, build 3D Slicer, then configure the AI-assisted annotation client project using CMake, and define the following variables: -DSlicer_DIR: PATH = … -DNvidiaAIAssistedAnnotation_BUILD_SLICER_EXTENSION: BOOL = ON