Got errors in dynunet_pipeline: 'RuntimeError: result type Float can't be cast to the desired output type Byte'

[yhuang1997]

Describe the bug
Intutorials/modules/dynunet_pipeline, I got RuntimeError when codes run on monai/data/utils.py: function collate_meta_tensor. I have tried to solve this, but debugging the recurse function collate_meta_tensor is hard for me. I found the data type of label will be converted into np.uint8 after resampling, so the error information is 'RuntimeError: result type Float can't be cast to the desired output type Byte.' And if I converted into np.int16, the error information will change with 'RuntimeError: result type Float can't be cast to the desired output type Short.' And so, finally, I converted the label into np.float32, the code works.

Here's the main log for codes running on Task02

2022-10-07 21:27:14,971 - dev_collate - CRITICAL - >> collate dict key "slice_start" out of 43 keys
2022-10-07 21:27:14,971 - dev_collate - CRITICAL - >>> collate/stack a list of numpy arrays
2022-10-07 21:27:14,972 - dev_collate - CRITICAL - >>> collate/stack a list of tensors
2022-10-07 21:27:14,972 - dev_collate - CRITICAL - >> collate dict key "pixdim" out of 43 keys
2022-10-07 21:27:14,973 - dev_collate - CRITICAL - >>> collate/stack a list of numpy arrays
2022-10-07 21:27:14,974 - dev_collate - CRITICAL - >>> collate/stack a list of tensors
2022-10-07 21:27:14,974 - dev_collate - CRITICAL - >> collate dict key "vox_offset" out of 43 keys
2022-10-07 21:27:14,974 - dev_collate - CRITICAL - >>> collate/stack a list of numpy arrays
2022-10-07 21:27:14,974 - dev_collate - CRITICAL - >>> collate/stack a list of tensors
2022-10-07 21:27:14,974 - dev_collate - CRITICAL - >> collate dict key "scl_slope" out of 43 keys
2022-10-07 21:27:14,975 - dev_collate - CRITICAL - >>> collate/stack a list of numpy arrays
2022-10-07 21:27:14,975 - dev_collate - CRITICAL - >>> collate/stack a list of tensors
2022-10-07 21:27:14,975 - dev_collate - CRITICAL - >> collate dict key "scl_inter" out of 43 keys
2022-10-07 21:27:14,976 - dev_collate - CRITICAL - >>> collate/stack a list of numpy arrays
2022-10-07 21:27:14,976 - dev_collate - CRITICAL - >>> collate/stack a list of tensors
2022-10-07 21:27:14,976 - dev_collate - CRITICAL - >> collate dict key "slice_end" out of 43 keys
2022-10-07 21:27:14,977 - dev_collate - CRITICAL - >>> collate/stack a list of numpy arrays
2022-10-07 21:27:14,977 - dev_collate - CRITICAL - >>> collate/stack a list of tensors
2022-10-07 21:27:14,977 - dev_collate - CRITICAL - >> collate dict key "slice_code" out of 43 keys
2022-10-07 21:27:14,977 - dev_collate - CRITICAL - >>> collate/stack a list of numpy arrays
2022-10-07 21:27:14,977 - ignite.engine.engine.DynUNetTrainer - ERROR - Current run is terminating due to exception: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/yhuang/pycharm_projects/MONAI/monai/data/utils.py", line 466, in list_data_collate
ret = collate_meta_tensor(data)
File "/home/yhuang/pycharm_projects/MONAI/monai/data/utils.py", line 435, in collate_meta_tensor
return {k: collate_meta_tensor([d[k] for d in batch]) for k in elem_0}
File "/home/yhuang/pycharm_projects/MONAI/monai/data/utils.py", line 435, in
return {k: collate_meta_tensor([d[k] for d in batch]) for k in elem_0}
File "/home/yhuang/pycharm_projects/MONAI/monai/data/utils.py", line 429, in collate_meta_tensor
collated = default_collate(batch)
File "/home/yhuang/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 56, in default_collate
return torch.stack(batch, 0, out=out)
File "/home/yhuang/pycharm_projects/MONAI/monai/data/meta_tensor.py", line 249, in torch_function
ret = super().torch_function(func, types, args, kwargs)
File "/home/yhuang/anaconda3/lib/python3.8/site-packages/torch/_tensor.py", line 1023, in torch_function
ret = func(*args, **kwargs)
RuntimeError: result type Float can't be cast to the desired output type Byte

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/yhuang/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/home/yhuang/anaconda3/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
return self.collate_fn(data)
File "/home/yhuang/pycharm_projects/MONAI/monai/data/utils.py", line 479, in list_data_collate
raise RuntimeError(re_str) from re
RuntimeError: result type Float can't be cast to the desired output type Byte

To Reproduce
I didn't change any line for dynunet_pipeline.

Environment (please complete the following information):

• OS Ubuntu 22.04.1 LTS
• Python version 3.9
• MONAI version 1.0.0+29.g2f16f012
• torch 1.12.1 with cuda11.3

[Nic-Ma]

Hi @yiheng-wang-nv ,

Could you please help take a look at this issue?

Thanks in advance.

[yiheng-wang-nv]

Hi @yhuang1997 , thanks for posting this issue. I checked the code and found that last changes which remove the cast type transforms may raise this issue on some of the decathlon datasets. I will create a PR and change back to use the cast transforms.