Affine Transform does not return target affine

Describe the bug
The output image of the Affine transform does not have the correct affine.

To Reproduce

Load this example data:
image.nii.gz
labels.nii.gz
Run the following script:

# %%
from pathlib import Path

from monai.data.meta_tensor import MetaTensor
from monai.transforms.spatial.array import Affine
import nibabel as nib
import numpy as np
import torch

# %%
folder = Path('./example_data')


def load(path: Path):
    image_ni = nib.load(path)
    image_np = np.asarray(image_ni.get_fdata(), dtype=np.float64)
    image_affine = np.asarray(image_ni.affine, dtype=np.float64)
    image_mt = MetaTensor(torch.from_numpy(image_np).unsqueeze(0), torch.from_numpy(image_affine))
    return image_ni, image_np, image_affine, image_mt


source_ni, source_np, source_affine, source_mt = load(folder / 'labels.nii.gz')
target_ni, target_np, target_affine, target_mt = load(folder / 'image.nii.gz')

# %%#
resample_affine = np.linalg.inv(source_affine) @ target_affine

affine_transform = Affine(
    affine=resample_affine,
    image_only=True,
    mode=0,
    padding_mode="zeros",
    dtype=np.float64,
)
resampled_mt = affine_transform(source_mt)

# %%
diff = target_affine - resampled_mt.affine.numpy()
with np.printoptions(precision=3, suppress=True):
    print(target_affine)
    print(source_affine @ resample_affine)
    print(resampled_mt.affine.numpy())
    print(diff)

target_affine

[[   1.      -0.       0.    -130.245]
 [  -0.       1.       0.    -163.11 ]
 [   0.       0.       1.    -112.138]
 [   0.       0.       0.       1.   ]]

source_affine @ resample_affine

[[   1.      -0.      -0.    -130.245]
 [  -0.       1.       0.    -163.11 ]
 [   0.       0.       1.    -112.138]
 [   0.       0.       0.       1.   ]]

resampled_mt.affine.numpy()

[[   1.      -0.      -0.    -139.079]
 [  -0.       1.       0.    -231.945]
 [   0.       0.       1.     -62.873]
 [   0.       0.       0.       1.   ]]

diff

[[  0.      0.      0.      8.835]
 [ -0.     -0.     -0.     68.835]
 [  0.     -0.     -0.    -49.265]
 [  0.      0.      0.      0.   ]]

As we can see, source_affine @ resample_affine is equal to target_affine.
But resampled_mt.affine.numpy() is different and there is a diff.

Expected behavior
I expected resampled_mt.affine.numpy() to be equal to source_affine @ resample_affine (or target_affine).

Environment

================================
Printing MONAI config...
================================
MONAI version: 1.2.0
Numpy version: 1.22.3
Pytorch version: 2.1.0+cu121
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: c33f1ba588ee00229a309000e888f9817b4f1934
MONAI __file__: /home/spe/.pyenv/versions/datascience/lib/python3.10/site-packages/monai/__init__.py

Optional dependencies:
Pytorch Ignite version: 0.4.11
ITK version: 5.3.0
Nibabel version: 5.1.0
scikit-image version: 0.22.0
Pillow version: 10.0.1
Tensorboard version: 2.14.1
gdown version: 4.7.1
TorchVision version: 0.16.0+cu121
tqdm version: 4.66.1
lmdb version: 1.4.1
psutil version: 5.9.5
pandas version: 2.0.3
einops version: 0.7.0
transformers version: 4.21.3
mlflow version: 2.7.1
pynrrd version: 1.0.0

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies


================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 20.04.6 LTS
Platform: Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
Processor: x86_64
Machine: x86_64
Python version: 3.10.12
Process name: python
Command: ['/home/spe/.pyenv/versions/datascience/bin/python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: [popenfile(path='/home/spe/.vscode-server/data/logs/20231005T155144/ptyhost.log', fd=19, position=0, mode='a', flags=33793), popenfile(path='/home/spe/.vscode-server/data/logs/20231005T155144/remoteagent.log', fd=21, position=13979, mode='a', flags=33793), popenfile(path='/home/spe/.vscode-server/data/logs/20231005T155144/network.log', fd=23, position=0, mode='a', flags=33793)]
Num physical CPUs: 6
Num logical CPUs: 12
Num usable CPUs: 12
CPU usage (%): [10.5, 4.5, 5.8, 2.1, 5.6, 2.6, 5.9, 3.1, 6.1, 2.1, 11.1, 90.8]
CPU freq. (MHz): 3492
Load avg. in last 1, 5, 15 mins (%): [5.2, 3.4, 3.8]
Disk usage (%): 53.4
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 15.6
Available memory (GB): 8.5
Used memory (GB): 6.8

================================
Printing GPU config...
================================
Num GPUs: 1
Has CUDA: True
CUDA version: 12.1
cuDNN enabled: True
cuDNN version: 8902
Current device: 0
Library compiled for CUDA architectures: ['sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_90']
GPU 0 Name: Quadro K2200
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 5
GPU 0 Total memory (GB): 4.0
GPU 0 CUDA capability (maj.min): 5.0

the Affine transform assumes that the origin is at the image spatial center when applying resample_affine:

MONAI/monai/transforms/spatial/array.py

Line 2285 in 100db27

mat = shift_1 @ convert_data_type(mat, np.ndarray)[0] @ shift_2

the Affine transform assumes that the origin is at the image spatial center when applying resample_affine:

MONAI/monai/transforms/spatial/array.py

Line 2285 in 100db27

mat = shift_1 @ convert_data_type(mat, np.ndarray)[0] @ shift_2

Good that you are pointing this out. I was already working on a second bug report with respect to the offset that output images have.

If I do something like this:

offset = create_translate(len(image_shape), [float(d - 1) / 2 for d in image_shape])
resample_affine = np.linalg.inv(offset) @ resample_affine @ offset

then it works and the output is correct, but it's also rather unexpected that this is necessary.
Note that compute_w_affine shifts in the opposite direction, which results in broken images. Following the code of affine_func looks to me as if that is not actually used as output affine (at least I can not see where within the code).

I'm also confused on when the normalized parameter is necessary then. I thought something like that is the use-case for that parameter? As if I my self have to shift to a center, I'm doing the normalization, which I expected Affine to do.

normalized: indicating whether the provided affine is defined to include a normalization
transform converting the coordinates from [-(size-1)/2, (size-1)/2] (defined in create_grid) to
[0, size - 1] or [-1, 1] in order to be compatible with the underlying resampling API.
If normalized=False, additional coordinate normalization will be applied before resampling.
See also: :py:func:monai.networks.utils.normalize_transform.

So in case of normalized=False I understood this to not have this center origin requirement.

Also why is this necessary at all or rather if the computation requires that center origin, why doesn't the Affine transform perform that shift?

The documentation on these functions and parameters could be improved. If there is a center origin requirement, it would be good if the Affine transform doc string would note that and potentially even give instruction on how to center it if not already done. As said, I'm not able to understand what the normalized parameter is for then. The compute_w_affine function has no doc string explaining what it is useful for.

I don't know if it's just the data we are working with (MRI images) but we already spend many hours struggling with unexpected behavior of functions in this area. Maybe it's because all libraries seem to impose different conventions. ITK is especially terrible and we try to avoid it in our code base where ever we can.

Maybe it's just a learning on our side, or we can make some useful improvements out of this, what do you thing?

thanks, agree that the documentation could be largely improved..

I would gladly contribute, but I first need to solve my confusion expressed above. Let me know if and when I can help (would also be available for a call if that's useful).

Improve Affine transform documentation

Improve Affine transform documentation and add compute_w_affine tests

Improve Affine transform documentation and add compute_w_affine tests (…

Navigation Menu

Repository navigation

Affine Transform does not return target affine #7092

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Affine Transform does not return target affine #7092

Description

Activity

wyli commented on Oct 6, 2023

Spenhouet commented on Oct 6, 2023

wyli commented on Oct 6, 2023

Spenhouet commented on Oct 6, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions