AssertionError in test_load_image

[KumoLiu]

[2024-12-28T19:19:03.755Z] ======================================================================

[2024-12-28T19:19:03.755Z] FAIL: test_nibabel_reader_gpu_3 (tests.test_load_image.TestLoadImage)

[2024-12-28T19:19:03.755Z] ----------------------------------------------------------------------

[2024-12-28T19:19:03.755Z] Traceback (most recent call last):

[2024-12-28T19:19:03.755Z]   File "/usr/local/lib/python3.10/dist-packages/parameterized/parameterized.py", line 620, in standalone_func

[2024-12-28T19:19:03.755Z]     return func(*(a + p.args), **p.kwargs, **kw)

[2024-12-28T19:19:03.755Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/tests/test_load_image.py", line 236, in test_nibabel_reader_gpu

[2024-12-28T19:19:03.755Z]     self.assertTrue(torch.equal(result_cpu, result.cpu()))

[2024-12-28T19:19:03.755Z] AssertionError: False is not true

[KumoLiu]

Hi @yiheng-wang-nv, could you please take a look at this issue? The test case encountered a random failure in the 24.08 on the A100. Thanks.

[KumoLiu]

Still randomly failed. cc @yiheng-wang-nv

[2025-01-03T19:18:59.181Z] ======================================================================

[2025-01-03T19:18:59.181Z] FAIL: test_nibabel_reader_gpu_3 (tests.test_load_image.TestLoadImage)

[2025-01-03T19:18:59.181Z] ----------------------------------------------------------------------

[2025-01-03T19:18:59.181Z] Traceback (most recent call last):

[2025-01-03T19:18:59.181Z]   File "/usr/local/lib/python3.10/dist-packages/parameterized/parameterized.py", line 620, in standalone_func

[2025-01-03T19:18:59.181Z]     return func(*(a + p.args), **p.kwargs, **kw)

[2025-01-03T19:18:59.181Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/tests/test_load_image.py", line 236, in test_nibabel_reader_gpu

[2025-01-03T19:18:59.181Z]     self.assertTrue(torch.allclose(result_cpu, result.cpu(), atol=1e-6))

[2025-01-03T19:18:59.181Z] AssertionError: False is not true

[2025-01-03T19:18:59.181Z] 

[2025-01-03T19:18:59.181Z] ----------------------------------------------------------------------

[KumoLiu]

More detailed error. cc @yiheng-wang-nv

[2025-01-07T19:18:57.654Z] ======================================================================

[2025-01-07T19:18:57.654Z] FAIL: test_nibabel_reader_gpu_3 (tests.test_load_image.TestLoadImage)

[2025-01-07T19:18:57.654Z] ----------------------------------------------------------------------

[2025-01-07T19:18:57.654Z] Traceback (most recent call last):

[2025-01-07T19:18:57.654Z]   File "/usr/local/lib/python3.10/dist-packages/parameterized/parameterized.py", line 620, in standalone_func

[2025-01-07T19:18:57.654Z]     return func(*(a + p.args), **p.kwargs, **kw)

[2025-01-07T19:18:57.654Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/tests/test_load_image.py", line 236, in test_nibabel_reader_gpu

[2025-01-07T19:18:57.654Z]     assert_allclose(result_cpu, result.cpu(), atol=1e-6)

[2025-01-07T19:18:57.654Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/tests/utils.py", line 135, in assert_allclose

[2025-01-07T19:18:57.654Z]     np.testing.assert_allclose(actual, desired, *args, **kwargs)

[2025-01-07T19:18:57.654Z]   File "/usr/local/lib/python3.10/dist-packages/numpy/testing/_private/utils.py", line 1592, in assert_allclose

[2025-01-07T19:18:57.654Z]     assert_array_compare(compare, actual, desired, err_msg=str(err_msg),

[2025-01-07T19:18:57.654Z]   File "/usr/lib/python3.10/contextlib.py", line 79, in inner

[2025-01-07T19:18:57.654Z]     return func(*args, **kwds)

[2025-01-07T19:18:57.654Z]   File "/usr/local/lib/python3.10/dist-packages/numpy/testing/_private/utils.py", line 783, in assert_array_compare

[2025-01-07T19:18:57.654Z]     flagged = func_assert_same_pos(x, y, func=isnan, hasval='nan')

[2025-01-07T19:18:57.654Z]   File "/usr/local/lib/python3.10/dist-packages/numpy/testing/_private/utils.py", line 753, in func_assert_same_pos

[2025-01-07T19:18:57.654Z]     raise AssertionError(msg)

[2025-01-07T19:18:57.654Z] AssertionError: 

[2025-01-07T19:18:57.654Z] Not equal to tolerance rtol=1e-07, atol=1e-06

[2025-01-07T19:18:57.654Z] 

[2025-01-07T19:18:57.654Z] x and y nan location mismatch:

[2025-01-07T19:18:57.654Z]  x: array([[[[0.896333, 0.001718, 0.608176, ..., 0.833388, 0.786462,

[2025-01-07T19:18:57.654Z]           0.769311],

[2025-01-07T19:18:57.654Z]          [0.26893 , 0.231931, 0.654519, ..., 0.170201, 0.532211,...

[2025-01-07T19:18:57.654Z]  y: array([[[[ 4.876519e-43,  0.000000e+00,  0.000000e+00, ...,

[2025-01-07T19:18:57.654Z]            1.786100e+00, -2.456252e+16,  1.419553e+00],

[2025-01-07T19:18:57.654Z]          [-2.201973e-17,  1.678210e+00,  8.583682e-25, ...,...

[2025-01-07T19:18:57.654Z] 

[2025-01-07T19:18:57.654Z] ----------------------------------------------------------------------

[2025-01-07T19:18:57.654Z] Ran 16089 tests in 2528.825s

[yiheng-wang-nv]

some values loaded by GPU are significantly small, I checked recently nightly tests, the error always happen in 24.08 base container with A100. Hi @KumoLiu , I think the issue may come from other softwares instead of code here.

To solve the issue, I hope to do more tests, do you know if we can manually trigger the test in the same machine on a different branch?