Add support for ultrasound data

[KumoLiu]

Unlike other imaging modalities like MRI or CT, ultrasound images have distinct characteristics, including high noise levels, unique texture patterns, and variable contrast depending on the specific tissue being examined. Processing ultrasound data requires specialized algorithms, such as speckle reduction and boundary enhancement, which differ significantly from standard image processing techniques.

To integrate ultrasound data into our existing pipelines, we need to add flexible support options. These might include handling raw ultrasound data, incorporating specialized preprocessing steps, or enabling data streaming. Supporting raw data could open possibilities for real-time analysis, while collaboration with platforms like Holoscan could further enhance our capabilities in handling and processing ultrasound imaging data effectively.

[mingxin-zheng]

Hi @KumoLiu , thanks for the proposal. I think this task lacks some definition right now, as post-processed ultrasound data in DICOM format is not that special. If we are trying to work with raw ultrasound data (e.g. phase array or curvilinear array), then it is a very different story.

Maybe we need some ultrasound experts to suggest what the community may need. cc @ericspod @aylward @Nic-Ma Thanks!

[ericspod]

Hi @mingxin-zheng I think we do need some specialist opinions here for sure! I can imagine needing support for loading/saving new data formats but also transforms for working specifically with US data. These would cover common processing operations but also generating realistic noise or other signal degradation specific to US for data augmentation.

[mingxin-zheng]

Hi @ericspod , it's a good point to bring the up the noise augmentation! As far as I know there are at least two special kinds there: noise introduced by speckle and imaging-depth.

On the other hand, what specific data formats have you seen with ultrasound systems besides DICOM?

[ericspod]

I'm not well versed in what common practices are with US so can't really comment on what formats there would be. We will discuss with colleagues involved with US research and see what requirements they come up with.

[jamesobutler]

Coming from an Ultrasound pre-clinical imaging group we use PlusToolkit (main source repo: https://github.com/PlusToolkit/PlusLib) along with 3D Slicer (main source repo: https://github.com/Slicer/Slicer) to collect ultrasound data from various ultrasound hardware and save to MetaIO and NRRD formats. In this case we are using PlusToolKit as an open-source solution for ultrasound streaming and recording. PlusToolKit’s PlusServer is the primary tool, but there are some other ultrasound processing tools (see http://perk-software.cs.queensu.ca/plus/doc/nightly/user/Tools.html).

The NA-MIC project weeks have also had ultrasound AI based projects such as https://projectweek.na-mic.org/PW39_2023_Montreal/Projects/LiveTrackedUltrasoundProcessingWithPytorch/ which will lead you down links to things such as https://github.com/SlicerIGT/aigt/tree/master/UltrasoundSegmentation.

[mingxin-zheng]

Thanks @jamesobutler for the great inputs!

Today I met @bcjiang who also works in this domain. Wondering if you can share some experiences too

[bcjiang]

Hi @mingxin-zheng, we typically obtain the ultrasound raw channel data from the Verasonix machines. It has a MATLAB interface and the channel data is saved as a multi-dimensional array with 2D Frame index, A-line index and Signal data in each dimension. Each element in Signal data is in 14-bit signed encoding (-8192 to 8191). We can save these data to any file type and load them into python.
I think prior DL works leveraging the raw channel US data were mostly aiming at beamforming/imaging quality improvement, so data can be easily acquired by just scanning different subjects continuously. I didn't see much augmentation was applied in these works, but I'm also interested to learn some other augmentation methods for raw data in classification/segmentation scenarios.

[mingxin-zheng]

Thanks @bcjiang ! The raw data is interesting. Does the .mat file also include meta data to convert a raw data to a DICOM-level image (e.g. probe apeture information for scan conversion), or that's a seperate piece of data?

[bcjiang]

Hi @mingxin-zheng, yes there is also a header structure array in MATLAB workspace that we can export, which is created by a self-defined setup script before running the Verasonix data acquisition. The header should contain all probe information and we can choose which part to export and specify the export file format.

[mingxin-zheng]

Thanks @bcjiang for the explanation !

Now I'm just imagining if we need to implement such conversion for ultrasound data and here could be one thing for MONAI MetaTensor to improve:
Typically, the MetaTensor class contains an affine matrix to represent the spatial transform applied to an image (2D/3D). But ultrasound's scan conversion, cann't be represented by one single affine operation. Each data line has its own spatial transform (e.g. angle to steer), when it needs to be interpolated to a orthogonal raster.

Here is an illustration of such process. It needs to be noted that B is the RF raw data and A is what we see after scan-conversion and interpolation.

[jamesobutler]

Is MONAI considering tasks associated with RF to scan converted images? Or taking Ultrasound scan converted images from ultrasound hardware devices and then processing them for various labeling tasks? Often times Ultrasound hardware provides both scan converted and the raw RF data.

[mingxin-zheng]

I think this is still open to discussion here @jamesobutler . It is still a research topic how much RF data can provide beyond scan-converted images. I personally think MONAI can consider features to accelerate such explorations. But what features are needed remains undefined at the moment.