Different behavior of bilinear interpolation between resize_images and ONNX Upsample

[syoyo]

I have implemented exporting resize_images to ONNX by lowering it into Upsample ONNX op as did in unpooling_2D.

https://github.com/syoyo/onnx-chainer/tree/resize_images

and wanted to submit PR, but found a unit test fails due to different behavior of bilinear interpolation in Chainer and onnxruntime(and also TensorFlow/OpenCV which is a reference).

I use resize_images for pyramid pooling, thus need to embed resizing op into a model file.
So, I've tracked down this problem and here is a summary:

Input and output

Input shape: (2x2). [[64, 32], [64, 32]]
Output shape: (4x4) (Scaling factor is 2x)
Result: [64, ***, ***, 32] (Just check the first row)

========================================================= FAILURES ==========================================================
_______________________________________________ TestResizeImages.test_output ________________________________________________

self = <tests.functions_tests.test_arrays.TestResizeImages testMethod=test_output>

    def test_output(self):
    
        # FIXME(syoyo): Currently the test will fail due to different behavior
        # of bilinear interpolation between Chainer and onnxruntime.
        #
        # Currently Chainer will give [64, 53.333336, 42.666668, 32]
        # (same result with tensorflow r1.13.1 with `align_corners=True`),
        # while onnxruntime gives [64, 48, 32, 32]
        # (same result with tensorflow r1.13.1 with `align_corners=False`)
        #
        # Even though, expected bevhavior will be [64, 54, 40, 32].
        # (cv2.resize and tensorflow master(r1.14 or r2.0) after this fix:
        #  https://github.com/tensorflow/tensorflow/issues/6720)
    
        # TODO(hamaji): onnxruntime does not support Upsample-9 yet.
        # https://github.com/chainer/onnx-chainer/issues/111
>       self.expect(self.model, self.x, name='resize_images', skip_opset_version=[9])

tests/functions_tests/test_arrays.py:240: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/helper.py:101: in expect
    self.check_out_values(test_path, input_names=graph_input_names)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

test_path = 'out/opset7/test_resize_images', input_names = ['Input_0']

    def check_model_expect(test_path, input_names=None):
        if not ONNXRUNTIME_AVAILABLE:
            raise ImportError('ONNX Runtime is not found on checking module.')
    
        model_path = os.path.join(test_path, 'model.onnx')
        with open(model_path, 'rb') as f:
            onnx_model = onnx.load_model(f)
        sess = rt.InferenceSession(onnx_model.SerializeToString())
        rt_input_names = [value.name for value in sess.get_inputs()]
        rt_output_names = [value.name for value in sess.get_outputs()]
    
        # To detect unexpected inputs created by exporter, check input names
        if input_names is not None:
            assert list(sorted(input_names)) == list(sorted(rt_input_names))
    
        test_data_sets = sorted([
            p for p in os.listdir(test_path) if p.startswith('test_data_set_')])
        for test_data in test_data_sets:
            test_data_path = os.path.join(test_path, test_data)
            assert os.path.isdir(test_data_path)
            inputs, outputs = load_test_data(
                test_data_path, rt_input_names, rt_output_names)
    
            rt_out = sess.run(list(outputs.keys()), inputs)
            for cy, my in zip(outputs.values(), rt_out):
>               np.testing.assert_allclose(cy, my, rtol=1e-5, atol=1e-5)
E               AssertionError: 
E               Not equal to tolerance rtol=1e-05, atol=1e-05
E               
E               Mismatch: 50%
E               Max absolute difference: 10.666668
E               Max relative difference: 0.33333337
E                x: array([[[[64.      , 53.333336, 42.666668, 32.      ],
E                        [64.      , 53.333336, 42.666668, 32.      ],
E                        [64.      , 53.333332, 42.666668, 32.      ],
E                        [64.      , 53.333336, 42.666668, 32.      ]]]], dtype=float32)
E                y: array([[[[64., 48., 32., 32.],
E                        [64., 48., 32., 32.],
E                        [64., 48., 32., 32.],
E                        [64., 48., 32., 32.]]]], dtype=float32)

onnx_chainer/testing/test_onnxruntime.py:62: AssertionError

Chainer(v5.4)

it results in [64, 53.333336, 42.666668, 32].
(can be reproducable by running tests/functions_tests/test_arrays.py)

onnxruntime(v0.3.0)

it results in [64, 48, 32, 32].
(can be reproducable by running tests/functions_tests/test_arrays.py)

TensorFlow

import tensorflow as tf
import numpy as np

print(tf.__version__)

# NCHW
img = np.array([[[[64, 32],
                  [64, 32]]]], np.float32)
print(img.shape)

# to NHWC
img = tf.transpose(img, [0, 2, 3, 1])
print(img.shape)

align_corners = False # or True
img = tf.image.resize_bilinear(img, size=[4, 4], align_corners=align_corners)

sess = tf.Session()
ret = sess.run(img)
print(ret)

• [64, 48, 32, 32] for TensorFlow r1.13.1(default behavior)
• [64, 53.333332. 42.666664, 32] for TensorFlow r1.13.1 + align_corners=True
• [64, 56, 40, 32] for TensorFlow master after this fix: tensorflow/tensorflow@371c96d https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/image_ops_test.py#L2463

cv2.resize(should be the ground truth)

import cv2
import numpy as np

width = 2
height = 2

img = np.array([[64.0, 64.0],
                [32.0, 32.0]], np.float32)

size = (4, 4)

resized_img = cv2.resize(img, size)

print(resized_img)
 
# => (64, 56, 40, 32). FYI interpolator is [0, 0.25, 0.75, 1.0]

Current Chainer's behavior is same with the result of TF r1.13.1 but there was a bug

tensorflow/tensorflow#6720

and recently there had a fix to TF's resize_images with this commit: tensorflow/tensorflow@371c96d

What I should do?

Correct behavior will be the one in cv2.resize and tf r2.0([64, 56, 40, 32]), so we are better to move towards this direction in the last(i.e. add a fix or implement new resize_images into Chainer), but for a while there will be some options:

• Disable unit test for resize_images and send PR
• Write custom ONNX op or compare with manually supplied exected values
• Wait until onnx opset 10 support in onnx-chainer(Resize op. since Upsample will be deprecated in opset 10)

References

• How Tensorflow’s tf.image.resize stole 60 days of my life https://hackernoon.com/how-tensorflows-tf-image-resize-stole-60-days-of-my-life-aba5eb093f35

[disktnk]

Thank you for detail report. I saw your commits master...syoyo:resize_images roughly and looks fine.

wanted to submit PR, but found a unit test fails due to different behavior of bilinear interpolation in Chainer and onnxruntime

Please void output value check between Chainer and ONNXRuntime, like this

def test_output(self):

    self.check_out_values = None  # skip output value check
    self.expect(self.model, self.x)

Test tool does not have "skip all output value check" option, so by adding self.check_out_values = None, skip it (hacky a little...), and only check output ONNX graph.