Image I/O Proposal - Feedback Welcomed

[aylward]

We have posted a draft proposal for Image I/O in MONAI, and we welcome developer and community feedback:

Image I/O Proposal

This is a high-level draft, and significant input and creativity are needed to turn it into something concrete and feasible. Please give us your expert input by adding comments to this issue.

A brief summary of the proposal is

Proposal Goal

Support reading and writing common research image and data file formats: NRRD, Nifti, DICOM (objects), JPG, PNG, TIFF, MetaIO, ... without re-implementation or significant support burden. Additionally, enable a user to override default I/O methods on an experiment-by-experiment basis, e.g., to ensure a specific version of a specific reader is used so the desired, reproducible behavior is achieved for that specific experiment.

Proposal Summary

MONAI should offer an extensible framework for research image I/O and rely on ITK for the default handling of the file formats that it supports.

[kayhan-batmanghelich]

MONAI will be a huge contribution to the community. Thank you for initiating it.

A couple of suggestions:

• I understand that this I/O is mostly about image data but is it possible to include a similar interface to read deformation field?
• Would you please include Freesurfer file format to your list (in case it is not already there).
• Could you please include histological formats in addition to the radiology format images?
• Current Dicom reader in ITK is sufficient for most cases but there always cases that it failed (eg does not make a correct volume). Of course, we have more comprehensive tools for that outside of the ITK. In my opinion, when one uses loader from Dicom format, if the dicom reader is not fully successful, it is better if it fails loudly rather than silently doing what is almost right. Reading data from dicom format is still tricky and if it is done in a scale without proper inspection, it can bias the model in many different catastrophic way.

Thanks

[aylward]

Hi @kayhan-batmanghelich - Thanks for the feedback!

You raise several excellent points!

• Regarding deformation fields - With ITK you will be able to read deformation fields as vector images, and therefore I would expect that the proposed IO method would work to access them via a data loader. Clearly this is something we would need to test/investigate, but in theory, anything ITK can read should be supported for input.

• Regarding Freesurfer - ITK includes a module called the MGHIO module. I believe this is what is commonly called the freesurfer format. By using Wrapped ITK (instead of Simple ITK), we automatically get access to any and every ITK module in python, so the MGHIO module would be available and Freesurfer IO should work. Additionally, if that isn't what you were looking for, other IO methods can be added to ITK in C++ and they will be automatically wrapped and available to MONAI if this proposal is adopted. Furthermore, if there is another python library for freesurfer IO, our goal is to have it so that an "Experiment" can specify an alternative loader and library, so it could be used instead.

• Great point on histological formats. They are at the top of our list of non-neuro formats that we need to support. We've been looking at NWB and other libs for such IO. Please let us know if you have specific suggestions.

• We had a great meeting on DICOM and while we couldn't agree on the best library for DICOM that we should use, we could all agree that no single DICOM library works well for everyone! :) So, when we talk about "customization and extensibility" for I/O, we're particularly thinking about enabling alternative DICOM libraries to be used: PyDICOM, DCMTK wrapped, etc. ITK has recently updated its version of GDCM, so hopefully it is a bit better than before, but I fully agree that we need to be VERY flexible regarding DICOM libs that MONAI allows, and you raise a great point about failing subtly versus noticeably. We need to ensure such failures aren't hidden by any system we deliver.

Thanks!

[Nic-Ma]

Hi @aylward ,

Thanks so much for your great summary!
Sounds good a proposal, may I know where I can find some python examples of ITK to load Nifti, PNG, numpy NPY files? Maybe we can start to develop the IO factory from these 3 existing formats in MONAI.

Thanks.

[aylward]

Hi @Nic-Ma ,

Thanks for your input to the proposal! It has been great working with you on the working groups, and I think the proposal is coming together well, but the real challenge will be devising a clean implementation!

The documentation for Wrapped ITK is available in the Quick Start Guide.

Since ITK v5.1, reading and writing images is fairly simple and pythonic:

pip install itk

and then

#!/usr/bin/env python

import itk
import sys

input_filename = sys.argv[1]
output_filename = sys.argv[2]

image = itk.imread(input_filename)

median = itk.median_image_filter(image, radius=2)

itk.imwrite(median, output_filename)

[Nic-Ma]

Hi @aylward ,

Thanks for your sharing!
I will verify all loading logic with some test program next week, then we can try to make a design ASAP maybe.
Thanks.

[jond01]

Hi, great plan! I have some comments -

1. Choosing default conventions: from the examples I saw, MONAI currently uses NiBabel in LoadNifti and Orientation transforms. My suggestion is (a) merge these two transforms into one (also for itk-based LoadImage), (b) choose MONAI's default convention for coordinate system and orientation codes, in order to maintain reproducibility via different IO methods. Another point is to consider channeled images (channels first/last).
2. ITK vs. alternatives: ITK covers most of the medical images' formats. However, (a) its performance in python is usually inferior to NiBabel and PyDicom (especially for image series). Unfortunately, I do not have an exact report, but performance can be quite important in training and inference. (b) From my experience, the stability of the python version of itk is not great. We had problems with some the releases of itk v5.1.0 and had to downgrade the version to v5.0.1 in MedIO. There are more examples.
   Therefore, it may require maintenance along the way.
3. You will probably find the following useful:
   https://github.com/RSIP-Vision/medio/blob/master/medio/read_save.py
   https://github.com/RSIP-Vision/medio/blob/master/medio/backends/itk_io.py

More details in #827 (comment).

[Nic-Ma]

Hi @jond01 ,

Cool, thanks so much for your sharing!
Let me try to play with it this week and get back to you soon.

Thanks.

[Nic-Ma]

Hi @aylward ,

I tried to understand the ITK examples and get 2 quick questions:

1. May I know where I can find the doc-string for all ITK python APIs? I want to know what's the expected input data type and output data type of itk.imread().
2. How to get the meta_data of loaded image with python APIs? Is it a dict type object? Where can I find a list of all the expected keys in this meta_data dict for Nifti and PNG, NPY files?

Thanks.

[Nic-Ma]

Hi @aylward ,

BTW, I submitted a draft PR to develop the IO APIs: #893.
Got an error when I try to convert ITKMetaDictObject to python dict:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-2-ca80851c4218> in <module>
      1 filename = "/workspace/data/medical/Task09_Spleen/imagesTr/spleen_10.nii.gz"
      2 loader = LoadImage()
----> 3 data, meta = loader(filename)

/workspace/data/medical/MONAI/monai/transforms/io/array.py in __call__(self, filename)
     88             if not compatible_meta:
     89                 for meta_key in header.GetKeys():
---> 90                     meta_datum = header[meta_key]
     91                     if (
     92                         isinstance(meta_datum, np.ndarray)

/opt/conda/lib/python3.6/site-packages/itk/ITKCommonBasePython.py in __getitem__(self, key)
    887         import itk
    888         obj = self.Get(key)
--> 889         return itk.down_cast(obj).GetMetaDataObjectValue()
    890     def __len__(self):
    891         return self.GetKeys().size()

/opt/conda/lib/python3.6/site-packages/itkExtras.py in down_cast(obj)
   1198         raise RuntimeError(
   1199             "Can't downcast to a specialization of %s" %
-> 1200             className)
   1201     else:
   1202         return t.cast(obj)

RuntimeError: Can't downcast to a specialization of MetaDataObject

Do I misunderstand anything?

Thanks.

[aylward]

@Nic-Ma - Great work!!!

Matt McCormick ( @thewtex ) is the local ITK guru (and he leads most ITK development, including ITK Python), and he provided the following answers for your two questions:

1. How to read doc-strings for imread() and all ITK python APIs?

• ITK Python API's have standard doc-strings. They can read with help(itk.imread) in a standard Python interpreter and itk.imread? in an IPython interpreter.
• ITK 5.1.1 (just pushed to PyPI but not yet announced) has improved doc-strings for all Python filter functional interfaces.
• itk.imread will read the image file in with the native pixel type store in the file by default. A second argument can be supplied to force reading into a specific pixel type.

2. How to get the meta_data of loaded image with python APIs?

• itk.imread returns an itk.Image data structure that has the associated metadata, e.g. pixel spacing, in addition to the pixel data (we have found that this is helpful because they are always available and coupled, which makes it easier to generate processing pipelines and avoid errors).
• To get the spacing, origin of an itk.Image, currently:
  spacing = itk.spacing(image)
  origin = itk.origin(image)

For meta-data, perhaps we should consider the Python imageio library. It can use the ITK Python wrappings as an IO backend. pip install imageio itk-io. They are continuously improving a community standard meta-data dictionary for image readers to return: imageio/imageio#362

[Nic-Ma]

Hi @thewtex,

Thanks very much for your explanation, it's much more clear for me now.
I think the imageio feature is still under development, we may consider it as a good standard meta-data when it's ready to use later.
Now I want to use python dict to store meta data in MONAI, which can be compatible with other readers, how can we easily convert ITKMetaDictObject to dict? I faced an issue as I pasted in the previous comment, can't get all the values in Nifti meta data with specified key?
The test code is in PR: #893.

Thanks.

[Nic-Ma]

Hi @jond01 ,

Have you faced this issue at this line of MedIO?
https://github.com/RSIP-Vision/medio/blob/master/medio/backends/itk_io.py#L56
I tried exact same logic as yours to load the meta data of 1 Nifti image of spleen dataset, got:

RuntimeError: Can't downcast to a specialization of MetaDataObject

The test code is in PR: #893:

https://github.com/Project-MONAI/MONAI/pull/893/files#diff-a1e4249922432d53e4bb126d43ab9bcbR64

Thanks.