Image I/O Proposal - Feedback Welcomed

[aylward]

We have posted a draft proposal for Image I/O in MONAI, and we welcome developer and community feedback:

Image I/O Proposal

This is a high-level draft, and significant input and creativity are needed to turn it into something concrete and feasible. Please give us your expert input by adding comments to this issue.

A brief summary of the proposal is

Proposal Goal

Support reading and writing common research image and data file formats: NRRD, Nifti, DICOM (objects), JPG, PNG, TIFF, MetaIO, ... without re-implementation or significant support burden. Additionally, enable a user to override default I/O methods on an experiment-by-experiment basis, e.g., to ensure a specific version of a specific reader is used so the desired, reproducible behavior is achieved for that specific experiment.

Proposal Summary

MONAI should offer an extensible framework for research image I/O and rely on ITK for the default handling of the file formats that it supports.

[kayhan-batmanghelich]

MONAI will be a huge contribution to the community. Thank you for initiating it.

A couple of suggestions:

• I understand that this I/O is mostly about image data but is it possible to include a similar interface to read deformation field?
• Would you please include Freesurfer file format to your list (in case it is not already there).
• Could you please include histological formats in addition to the radiology format images?
• Current Dicom reader in ITK is sufficient for most cases but there always cases that it failed (eg does not make a correct volume). Of course, we have more comprehensive tools for that outside of the ITK. In my opinion, when one uses loader from Dicom format, if the dicom reader is not fully successful, it is better if it fails loudly rather than silently doing what is almost right. Reading data from dicom format is still tricky and if it is done in a scale without proper inspection, it can bias the model in many different catastrophic way.

Thanks

[aylward]

Hi @kayhan-batmanghelich - Thanks for the feedback!

You raise several excellent points!

• Regarding deformation fields - With ITK you will be able to read deformation fields as vector images, and therefore I would expect that the proposed IO method would work to access them via a data loader. Clearly this is something we would need to test/investigate, but in theory, anything ITK can read should be supported for input.

• Regarding Freesurfer - ITK includes a module called the MGHIO module. I believe this is what is commonly called the freesurfer format. By using Wrapped ITK (instead of Simple ITK), we automatically get access to any and every ITK module in python, so the MGHIO module would be available and Freesurfer IO should work. Additionally, if that isn't what you were looking for, other IO methods can be added to ITK in C++ and they will be automatically wrapped and available to MONAI if this proposal is adopted. Furthermore, if there is another python library for freesurfer IO, our goal is to have it so that an "Experiment" can specify an alternative loader and library, so it could be used instead.

• Great point on histological formats. They are at the top of our list of non-neuro formats that we need to support. We've been looking at NWB and other libs for such IO. Please let us know if you have specific suggestions.

• We had a great meeting on DICOM and while we couldn't agree on the best library for DICOM that we should use, we could all agree that no single DICOM library works well for everyone! :) So, when we talk about "customization and extensibility" for I/O, we're particularly thinking about enabling alternative DICOM libraries to be used: PyDICOM, DCMTK wrapped, etc. ITK has recently updated its version of GDCM, so hopefully it is a bit better than before, but I fully agree that we need to be VERY flexible regarding DICOM libs that MONAI allows, and you raise a great point about failing subtly versus noticeably. We need to ensure such failures aren't hidden by any system we deliver.

Thanks!

[Nic-Ma]

Hi @aylward ,

Thanks so much for your great summary!
Sounds good a proposal, may I know where I can find some python examples of ITK to load Nifti, PNG, numpy NPY files? Maybe we can start to develop the IO factory from these 3 existing formats in MONAI.

Thanks.

[aylward]

Hi @Nic-Ma ,

Thanks for your input to the proposal! It has been great working with you on the working groups, and I think the proposal is coming together well, but the real challenge will be devising a clean implementation!

The documentation for Wrapped ITK is available in the Quick Start Guide.

Since ITK v5.1, reading and writing images is fairly simple and pythonic:

pip install itk

and then

#!/usr/bin/env python

import itk
import sys

input_filename = sys.argv[1]
output_filename = sys.argv[2]

image = itk.imread(input_filename)

median = itk.median_image_filter(image, radius=2)

itk.imwrite(median, output_filename)

[Nic-Ma]

Hi @aylward ,

Thanks for your sharing!
I will verify all loading logic with some test program next week, then we can try to make a design ASAP maybe.
Thanks.

[jond01]

Hi, great plan! I have some comments -

1. Choosing default conventions: from the examples I saw, MONAI currently uses NiBabel in LoadNifti and Orientation transforms. My suggestion is (a) merge these two transforms into one (also for itk-based LoadImage), (b) choose MONAI's default convention for coordinate system and orientation codes, in order to maintain reproducibility via different IO methods. Another point is to consider channeled images (channels first/last).
2. ITK vs. alternatives: ITK covers most of the medical images' formats. However, (a) its performance in python is usually inferior to NiBabel and PyDicom (especially for image series). Unfortunately, I do not have an exact report, but performance can be quite important in training and inference. (b) From my experience, the stability of the python version of itk is not great. We had problems with some the releases of itk v5.1.0 and had to downgrade the version to v5.0.1 in MedIO. There are more examples.
   Therefore, it may require maintenance along the way.
3. You will probably find the following useful:
   https://github.com/RSIP-Vision/medio/blob/master/medio/read_save.py
   https://github.com/RSIP-Vision/medio/blob/master/medio/backends/itk_io.py

More details in #827 (comment).

[Nic-Ma]

Hi @jond01 ,

Cool, thanks so much for your sharing!
Let me try to play with it this week and get back to you soon.

Thanks.

[Nic-Ma]

Hi @aylward ,

I tried to understand the ITK examples and get 2 quick questions:

1. May I know where I can find the doc-string for all ITK python APIs? I want to know what's the expected input data type and output data type of itk.imread().
2. How to get the meta_data of loaded image with python APIs? Is it a dict type object? Where can I find a list of all the expected keys in this meta_data dict for Nifti and PNG, NPY files?

Thanks.

[Nic-Ma]

Hi @aylward ,

BTW, I submitted a draft PR to develop the IO APIs: #893.
Got an error when I try to convert ITKMetaDictObject to python dict:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-2-ca80851c4218> in <module>
      1 filename = "/workspace/data/medical/Task09_Spleen/imagesTr/spleen_10.nii.gz"
      2 loader = LoadImage()
----> 3 data, meta = loader(filename)

/workspace/data/medical/MONAI/monai/transforms/io/array.py in __call__(self, filename)
     88             if not compatible_meta:
     89                 for meta_key in header.GetKeys():
---> 90                     meta_datum = header[meta_key]
     91                     if (
     92                         isinstance(meta_datum, np.ndarray)

/opt/conda/lib/python3.6/site-packages/itk/ITKCommonBasePython.py in __getitem__(self, key)
    887         import itk
    888         obj = self.Get(key)
--> 889         return itk.down_cast(obj).GetMetaDataObjectValue()
    890     def __len__(self):
    891         return self.GetKeys().size()

/opt/conda/lib/python3.6/site-packages/itkExtras.py in down_cast(obj)
   1198         raise RuntimeError(
   1199             "Can't downcast to a specialization of %s" %
-> 1200             className)
   1201     else:
   1202         return t.cast(obj)

RuntimeError: Can't downcast to a specialization of MetaDataObject

Do I misunderstand anything?

Thanks.

[aylward]

@Nic-Ma - Great work!!!

Matt McCormick ( @thewtex ) is the local ITK guru (and he leads most ITK development, including ITK Python), and he provided the following answers for your two questions:

1. How to read doc-strings for imread() and all ITK python APIs?

• ITK Python API's have standard doc-strings. They can read with help(itk.imread) in a standard Python interpreter and itk.imread? in an IPython interpreter.
• ITK 5.1.1 (just pushed to PyPI but not yet announced) has improved doc-strings for all Python filter functional interfaces.
• itk.imread will read the image file in with the native pixel type store in the file by default. A second argument can be supplied to force reading into a specific pixel type.

2. How to get the meta_data of loaded image with python APIs?

• itk.imread returns an itk.Image data structure that has the associated metadata, e.g. pixel spacing, in addition to the pixel data (we have found that this is helpful because they are always available and coupled, which makes it easier to generate processing pipelines and avoid errors).
• To get the spacing, origin of an itk.Image, currently:
  spacing = itk.spacing(image)
  origin = itk.origin(image)

For meta-data, perhaps we should consider the Python imageio library. It can use the ITK Python wrappings as an IO backend. pip install imageio itk-io. They are continuously improving a community standard meta-data dictionary for image readers to return: imageio/imageio#362

[Nic-Ma]

Hi @thewtex,

Thanks very much for your explanation, it's much more clear for me now.
I think the imageio feature is still under development, we may consider it as a good standard meta-data when it's ready to use later.
Now I want to use python dict to store meta data in MONAI, which can be compatible with other readers, how can we easily convert ITKMetaDictObject to dict? I faced an issue as I pasted in the previous comment, can't get all the values in Nifti meta data with specified key?
The test code is in PR: #893.

Thanks.

[Nic-Ma]

Hi @jond01 ,

Have you faced this issue at this line of MedIO?
https://github.com/RSIP-Vision/medio/blob/master/medio/backends/itk_io.py#L56
I tried exact same logic as yours to load the meta data of 1 Nifti image of spleen dataset, got:

RuntimeError: Can't downcast to a specialization of MetaDataObject

The test code is in PR: #893:

https://github.com/Project-MONAI/MONAI/pull/893/files#diff-a1e4249922432d53e4bb126d43ab9bcbR64

Thanks.

[Nic-Ma]

Hi @thewtex and @aylward ,

I tried to load a Nifti image and check the meta data, even I still can't convert the meta data to dict, I print out several information and found that:
The shape of numpy array is [55, 512, 512] and dim[1]=512, dim[2]=512, dim[3]=55.
May I know what's the dim order in ITK? It's DHW for 3D images? Should we unifty to HWD for all kinds of readers?
Really need you guys' help on this topic.

Thanks in advance.

[jond01]

Hey @Nic-Ma, in medio you can get the metadata dictionary (called there header) with

array, metadata = medio.read_img('path/to/img.nii.gz', backend='itk', header=True)
print(metadata)  # prints also the metadata.header dictionary

---

Hi @jond01 ,

Have you faced this issue at this line of MedIO?
https://github.com/RSIP-Vision/medio/blob/master/medio/backends/itk_io.py#L56
I tried exact same logic as yours to load the meta data of 1 Nifti image of spleen dataset, got:

RuntimeError: Can't downcast to a specialization of MetaDataObject

The test code is in PR: #893:

https://github.com/Project-MONAI/MONAI/pull/893/files#diff-a1e4249922432d53e4bb126d43ab9bcbR64

Thanks.

It happens in the new version of itk for 'ITK_original_direction' and 'ITK_original_spacing' keys.
In medio we had to fix the version of itk:

(b) From my experience, the stability of the python version of itk is not great. We had problems with some the releases of itk v5.1.0 and had to downgrade the version to v5.0.1 in MedIO. There are more examples.

My best suggestion, for now, is to use a try-except approach in get_meta_dict method.
If you really want to dive deeper, try to translate itk::ExposeMetaData code into python.

---

Hi @thewtex and @aylward ,

I tried to load a Nifti image and check the meta data, even I still can't convert the meta data to dict, I print out several information and found that:
The shape of numpy array is [55, 512, 512] and dim[1]=512, dim[2]=512, dim[3]=55.
May I know what's the dim order in ITK? It's DHW for 3D images? Should we unifty to HWD for all kinds of readers?
Really need you guys' help on this topic.

Thanks in advance.

The image array in itk is array[k, j, i], so if you want the standard order - array[i, j, k] - you need to take the transpose. See:
https://github.com/RSIP-Vision/medio/blob/master/medio/backends/itk_io.py#L162
This is one example of conventions issue (see point 1 here).
Note that it is relevant also for 2D images, but RGB[A] images may require additional care.

---

Anyway, the cooperation offer with recognition of contribution is still actual if you are interested. There is still some work and we'll be happy to be involved.

[Nic-Ma]

Hi @jond01 ,

Thanks very much for your detailed explanation!!
I will try to use ITK 5.0.1 as you suggested.
About our current plan, first step is to develop ImageReader, ITKReader, NibabelReader, LoadImage transform.
NibabelReader will keep the existing features in MONAI, ITKReader will be an experimental version for the first step.
We will release the proposal and development plan for next steps then.
If you can help contribute more things later, that would be fantastic!

Thanks in advance.

[thewtex]

The shape of numpy array is [55, 512, 512] and dim[1]=512, dim[2]=512, dim[3]=55.
May I know what's the dim order in ITK? It's DHW for 3D images?

The image array in itk is array[k, j, i], so if you want the standard order - array[i, j, k] - you need to take the transpose

A few clarifications on this commonly encountered, but important set of conventions:

• The shape of the numpy array produced by ITK, available with image.shape, np.array(image).shape, itk.array_from_image(image) is DHW / k,j,i. This is because of the default indexing for NumPy array's is C-order. It is important to use this order for performance reasons -- in C-order indexing, the last index is the fastest-moving-index where data elements are contiguous in memory, and memory caching makes leveraging this performance critical. See also the scikit-image notes on order of array dimensions. Yet, i,j,k may feel more natural for imaging, and indeed, in the ITK API's, e.g. the origin and spacing order, is WHD / i,j,k. This is no magic fix or right or wrong for this topic, but the convention taken should be as consistent as possible and well documented.

It happens in the new version of itk for 'ITK_original_direction' and 'ITK_original_spacing' keys.
In medio we had to fix the version of itk:

(b) From my experience, the stability of the python version of itk is not great. We had problems with some the releases of itk v5.1.0 and had to downgrade the version to v5.0.1 in MedIO. There are more examples.

My best suggestion, for now, is to use a try-except approach in get_meta_dict method.
If you really want to dive deeper, try to translate itk::ExposeMetaData code into python.

@jond01 sorry this is not more apparent; ITK_original_direction and ITK_original_spacing in the ITK MetaDataDictionary were deprecated long ago and are not intended to be the source of this information; image.GetDirection(), image.GetSpacing() are where to find this information.

In fact, these are what the cause of the failure in the metadata conversion. The following works on the spleen nifti data:

img_meta_dict = image.GetMetaDataDictionary()
meta_dict = {}
for key in img_meta_dict.GetKeys():
    # Ignore deprecated, legacy members that cause issues
    if key.startswith('ITK_original_'):
        continue
    meta_dict[key] = img_meta_dict[key]

We could add this as the __dict__ method on itk.MetaDataDictionary -- @Nic-Ma @jond01 what do you think?

To add origin, spacing, direction:

img_meta_dict = image.GetMetaDataDictionary()
meta_dict = {}
for key in img_meta_dict.GetKeys():
    # Ignore deprecated, legacy members that cause issues
    if key.startswith('ITK_original_'):
        continue
    meta_dict[key] = img_meta_dict[key]
meta_dict['origin'] = np.asarray(img.GetOrigin())
meta_dict['spacing'] = np.asarray(img.GetSpacing())
meta_dict['direction'] = itk.array_from_matrix(img.GetDirection())

For the spleen data, this results in:

{'ITK_FileNotes': '5.0.10',
 'aux_file': '',
 'bitpix': '32',
 'cal_max': '0',
 'cal_min': '0',
 'datatype': '16',
 'descrip': '5.0.10',
 'dim[0]': '3',
 'dim[1]': '512',
 'dim[2]': '512',
 'dim[3]': '55',
 'dim[4]': '1',
 'dim[5]': '1',
 'dim[6]': '1',
 'dim[7]': '1',
 'dim_info': '0',
 'intent_code': '0',
 'intent_name': '',
 'intent_p1': '0',
 'intent_p2': '0',
 'intent_p3': '0',
 'nifti_type': '1',
 'pixdim[0]': '0',
 'pixdim[1]': '0.976562',
 'pixdim[2]': '0.976562',
 'pixdim[3]': '5',
 'pixdim[4]': '0',
 'pixdim[5]': '0',
 'pixdim[6]': '0',
 'pixdim[7]': '0',
 'qform_code': '1',
 'qform_code_name': 'NIFTI_XFORM_SCANNER_ANAT',
 'qoffset_x': '-499.023',
 'qoffset_y': '-499.023',
 'qoffset_z': '0',
 'quatern_b': '0',
 'quatern_c': '0',
 'quatern_d': '0',
 'scl_inter': '0',
 'scl_slope': '1',
 'sform_code': '1',
 'sform_code_name': 'NIFTI_XFORM_SCANNER_ANAT',
 'slice_code': '0',
 'slice_duration': '0',
 'slice_end': '0',
 'slice_start': '0',
 'srow_x': '0.976562 0 0 -499.023',
 'srow_y': '0 0.976562 0 -499.023',
 'srow_z': '0 0 5 0',
 'toffset': '0',
 'vox_offset': '352',
 'xyzt_units': '10',
 'origin': array([499.02319336, 499.02319336,   0.        ]),
 'spacing': array([0.97656202, 0.97656202, 5.        ]),
 'direction': array([[-1.,  0.,  0.],
        [ 0., -1.,  0.],
        [ 0.,  0.,  1.]])}

[Nic-Ma]

Hi @thewtex ,

Thanks for your clarification!
I will try to update and test soon.

1. __dict__ sounds good a API to have.
2. Could you please help add an argument for the order of dims for numpy array? Default can be DHW?

Thanks.

[jond01]

1. __dict__ sounds good a API to have.

+1

2. Could you please help add an argument for the order of dims for numpy array? Default can be DHW?

https://discourse.itk.org/t/importing-image-from-array-and-axis-reorder/1192/5
I think that you can use keep_axes=True - but as @thewtex wrote, it is discouraged performance-wise, though necessary for consistency with nibabel for example.

[Nic-Ma]

Hi @jond01 ,

Thanks for your suggestion! I updated the code in the latest commit.
And then I want to raise another request for ITK, do you guys think it's necessary to export all docs of python API at webpage? Like MONAI api-docs: https://docs.monai.io/en/latest/ or PyTorch docs: https://pytorch.org/docs/stable/index.html

Thanks.

[thewtex]

1. __dict__ sounds good a API to have.

+1

Great, I have added an issue to track this here: InsightSoftwareConsortium/ITK#1957 . We can use these implementations directly, though, until that is out.

1. Could you please help add an argument for the order of dims for numpy array? Default can be DHW?

https://discourse.itk.org/t/importing-image-from-array-and-axis-reorder/1192/5
I think that you can use keep_axes=True - but as @thewtex wrote, it is discouraged performance-wise, though necessary for consistency with nibabel for example.

👍

And then I want to raise another request for ITK, do you guys think it's necessary to export all docs of python API at webpage? Like MONAI api-docs: https://docs.monai.io/en/latest/ or PyTorch docs: https://pytorch.org/docs/stable/index.html

Yes, good idea 💡 I added this issue to track this: InsightSoftwareConsortium/ITKPythonPackage#148

I am mostly AFK for the next month, but I will be able to implement these after.

[thewtex]

Great, I have added an issue to track this here: InsightSoftwareConsortium/ITK#1957 .

@Nic-Ma @jond01 implementation is available in InsightSoftwareConsortium/ITK#2039 , please take a look

[wyli]

several follow-ups were created and sub-tasks completed, I'm closing this ticket. Please create new discussion thread if needed.