Upsample support NHWC #10554

yihonglyu · 2022-02-14T22:57:39Z

Implement bilinear interpolation for Upsample (Resize) 4-D input with the
outermost and innermost scale (usually channel of NHWC) as 1.

Besides, I revert the HandleResize back to the original implementation for
TransposeOptimizerTests.TestResize* tests.

askhade · 2022-02-22T19:41:42Z

onnxruntime/core/providers/cpu/tensor/upsample.cc

+          height_scale = scales[0];
+          width_scale = scales[1];
+        } else {
+          if (scales[1] == 1.0f) {


Is this a good enough indicator to determine the layout of the tensor?

Before entering this method, ScalesValidation is invoked.

ORT_ENFORCE(scales.size() == 2 || (scales.size() == 4 && scales[0] == 1 && scales[1] == 1) || (scales.size() == 4 && scales[0] == 1 && scales[3] == 1) || scales.size() == 3 || (scales.size() == 5 && scales[0] == 1 && scales[1] == 1), "'Linear' mode only support:\n" " * 2-D inputs or\n" " * 3-D inputs ('Bilinear', 'Trilinear') or\n" " * 4-D inputs with the corresponding outermost 2 scale values being 1" " or the corresponding outermost and innermost scale values being 1 or\n" " * 5-D inputs with the corresponding outermost 2 scale values being 1" "in the ", is_resize_ ? "Resize operator" : "Upsample operator");

So this is enough to determine the layout of the tensor.

could you please add comment here to avoid this kind of confusion?

askhade · 2022-02-22T19:55:49Z

What is the motivation for this change?
Layout transformer considers this op as layout sensitive because cpu and nnapi treat this as a layout senstive op. Changes are required in layout transformer to handle the case where resize may be in nhwc format. see https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/core/optimizer/transpose_optimizer/optimizer_api_impl.cc#L852

yihonglyu · 2022-02-22T22:19:00Z

What is the motivation for this change? Layout transformer considers this op as layout sensitive because cpu and nnapi treat this as a layout senstive op. Changes are required in layout transformer to handle the case where resize may be in nhwc format. see https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/core/optimizer/transpose_optimizer/optimizer_api_impl.cc#L852

The motivation of this patch is to handle #9857. To work around the issue, the commit

commit 16bfd3c771c417fa047aa0e551dc493818c2bb4c
Author: RandySheriffH <48490400+RandySheriffH@users.noreply.github.com>
Date:   Mon Nov 29 22:30:16 2021 -0800

    Cancel transpose optimizer for resize (#9870)

comment out the transpose optimizer for resize and its tests temporally. With your change

commit f436d3437e85fa0c28a6264dd361958ae4d63e08
Author: Ashwini Khade <askhade@microsoft.com>
Date:   Tue Feb 15 20:25:29 2022 -0800

    Add layout transformer for NNAPI (#10371)

All TransposeOptimizerTests.Test* test:

1: [ FAILED ] TransposeOptimizerTests.TestResize
1: [ FAILED ] TransposeOptimizerTests.TestResizeOpset11
1: [ FAILED ] TransposeOptimizerTests.TestResizeOpset15
1: [ FAILED ] TransposeOptimizerTests.TestResizeSizeRoi
1: [ FAILED ] TransposeOptimizerTests.TestResizeRoiScalesZeroRank0
1: [ FAILED ] TransposeOptimizerTests.TestResizeNonconst
1: [ FAILED ] TransposeOptimizerTests.TestResizeNonconstOpset13

fails. Could you give me some insight about it?

yihonglyu · 2022-02-23T06:55:48Z

After revert the HandleResize back to the original implementation, all tests passes.

yufenglee

This reverts commit bd08f11.

This reverts commit bd08f11. Co-authored-by: Yufeng Li <liyufeng1987@gmail.com>

@yufenglee

* Update to flatbuffers v2.0.0 (#10866) * Fix Reduced ops pipeline (#10861) * Fix a couple of issues with the python package tools (#10858) * Tweaks to the model utils * Add handling for a dim_value of -1 when replacing the entire input shape. This occurs in models exported from PaddlePaddle * make pytorch helpers accessible in package * make QDQ helpers accessible in package * Fix wrong percentile values returned during calibration (#10847) * Use numpy.percentile to get the lookup value. * Use 1.0 as float value rather than integer. * Add missing cdf parameter for `np.percentile`. * Use 100. instead of 1.0 * Remove print. * Update from @yufenglee * Add support for opset 16 to transpose optimizer. (#10841) * Add support for opset 16 to transpose optimizer. Only change required is for GridSample to be added to the layout sensitive ops. The existing handling for layout transpose works with that as the first input and first output are layout sensitive. Update the optimize to be able to return an error message if it fails. * Use separate build directories for full and mobile iOS packages. (#10835) * Address performance issue with abseil flat_hash_table. (#10819) When returning by value in a cross DLL call, the hash table even though containing all the entries that are originally there can not find at least some of them. Reverting to std::unordered_set pending further investigation. * Mark end of version 11 C API. (#10803) * Mark end of version 11 C API * Add static_assert * avoid using LocalFree on FormatMessageW buffer (#10796) * remove local free * Remove local free from onnxruntime * don't allocate * Change to use constexpr to satisfy CPU build warning * Integrate C-API tests into Pipelines for release packages (#10794) * add c-api test for package * fix bug for running c-api test for package * refine run application script * remove redundant code * include CUDA test * Remove testing CUDA EP temporarily * fix bug * Code refactor * try to fix YAML bug * try to fix YAML bug * try to fix YAML bug * fix bug for multiple directories in Pipelines * fix bug * add comments and fix bug * Update c-api-noopenmp-packaging-pipelines.yml * Remove failOnStandardError flag in Pipelines * Detect runtime CUDA JIT and warn the user (#10781) * Use cudaMalloc vs cudaDeviceSynchronize and show the total time * Update convert_onnx_models_to_ort.py to support runtime optimizations. (#10765) Add runtime optimization support to ONNX -> ORT format conversion script. Replace `--optimization_level`, `--use_nnapi`, and `--use_coreml` with a new `--optimization_style` option. * Add multithreading test and put a lock on nvinfer1::createInferRuntime() for TRT EP (#10714) * Add multithread unit test and put lock on library call * update code * remove debug code * add comment * add one session multi-threads inference * Put lock for build engine all the time * Update naming and comment * remove unnecessary lock * Revert "remove unnecessary lock" This reverts commit 9c2317b. * Fix handling of nodes inserted by NHWC transformer. (#10904) (#10925) * Revert "Upsample support NHWC (#10554)" (#10917) This reverts commit bd08f11. Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> * [python API] Change raise import error when `C:\Windows\System32\vcruntime140_1.dll` is not found to warning (#10927) * remove throw if C:\\Windows\\System32\\vcruntime140_1.dll cannot be found * Add comments and update warning message * adding back accidentally removed line Co-authored-by: gwang0000 <62914304+gwang0000@users.noreply.github.com> * [js] Create npm packaging pipeline (#10886) * create npm packaging pipeline * fix indentations * Update npm-packaging-pipeline.yml for Azure Pipelines * Update npm-packaging-pipeline.yml for Azure Pipelines * Update npm-packaging-pipeline.yml for Azure Pipelines * react-native-ci as a template * fix typos * fix template paths * add a depencendy * change a stage name * set different artifact name for each package * fix typo * Update npm-packaging-pipeline.yml for Azure Pipelines Set a build Id for node npm package as a parameter * Update npm-packaging-pipeline.yml for Azure Pipelines Set a build Id for node npm package as a parameter * Update npm-packaging-pipeline.yml for Azure Pipelines * Follow up update for python API checking if `vcruntime140_1.dll` is available (#10927) (#10933) Co-authored-by: Hariharan Seshadri <hasesh@microsoft.com> Co-authored-by: Scott McKay <skottmckay@gmail.com> Co-authored-by: Funtowicz Morgan <mfuntowicz@users.noreply.github.com> Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Pranav Sharma <prs@microsoft.com> Co-authored-by: Ryan Lai <rylai@microsoft.com> Co-authored-by: Ryan Hill <38674843+RyanUnderhill@users.noreply.github.com> Co-authored-by: Yi-Hong Lyu <yilyu@microsoft.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Guoyu Wang <62914304+gwang-msft@users.noreply.github.com> Co-authored-by: gwang0000 <62914304+gwang0000@users.noreply.github.com> Co-authored-by: Sunghoon <35605090+hanbitmyths@users.noreply.github.com>

Implement bilinear interpolation for Upsample (Resize) 4-D input with the outermost and innermost scale (usually channel of NHWC) as 1. Besides, I revert the HandleResize back to the original implementation for TransposeOptimizerTests.TestResize* tests.

This reverts commit bd08f11. Co-authored-by: Yufeng Li <liyufeng1987@gmail.com>

Implement bilinear interpolation for Upsample (Resize) 4-D input with the outermost and innermost scale (usually channel of NHWC) as 1. Besides, I revert the HandleResize back to the original implementation for TransposeOptimizerTests.TestResize* tests.

yihonglyu requested review from snnn, chenfucn, zhanghuanrong and yufenglee February 14, 2022 22:57

askhade reviewed Feb 22, 2022

View reviewed changes

yihonglyu added 2 commits February 22, 2022 19:29

Upsample support NHWC

96ebf45

Revert HandleResize

f996937

yihonglyu force-pushed the yilyu/nhwc_upsample branch from ea6a701 to f996937 Compare February 23, 2022 04:15

yufenglee approved these changes Feb 23, 2022

View reviewed changes

yihonglyu merged commit bd08f11 into master Feb 23, 2022

yihonglyu deleted the yilyu/nhwc_upsample branch February 23, 2022 22:27

yufenglee mentioned this pull request Mar 16, 2022

The ONNX graph converted from tf.image.crop_and_resize() function will cause a different result by ORT nightly build. #10710

Closed

yufenglee added a commit that referenced this pull request Mar 17, 2022

Revert "Upsample support NHWC (#10554)"

2fa38e7

This reverts commit bd08f11.

yufenglee added a commit that referenced this pull request Mar 17, 2022

Revert "Upsample support NHWC (#10554)" (#10917)

a46b004

This reverts commit bd08f11. Co-authored-by: Yufeng Li <liyufeng1987@gmail.com>

chilo-ms pushed a commit that referenced this pull request Mar 17, 2022

Revert "Upsample support NHWC (#10554)" (#10917)

4b84d0c

This reverts commit bd08f11. Co-authored-by: Yufeng Li <liyufeng1987@gmail.com>

skottmckay mentioned this pull request Mar 17, 2022

Release 1.11.0 cherry pick round 1 #10915

Merged

lavanyax pushed a commit to intel/onnxruntime that referenced this pull request Mar 29, 2022

Revert "Upsample support NHWC (microsoft#10554)" (microsoft#10917)

e0d8203

This reverts commit bd08f11. Co-authored-by: Yufeng Li <liyufeng1987@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Upsample support NHWC #10554

Upsample support NHWC #10554

Uh oh!

yihonglyu commented Feb 14, 2022 •

edited

Loading

Uh oh!

askhade Feb 22, 2022

Uh oh!

yihonglyu Feb 22, 2022

Uh oh!

yufenglee Feb 23, 2022

Uh oh!

askhade commented Feb 22, 2022

Uh oh!

yihonglyu commented Feb 22, 2022 •

edited

Loading

Uh oh!

yihonglyu commented Feb 23, 2022 •

edited

Loading

Uh oh!

yufenglee left a comment

Uh oh!

Uh oh!

Upsample support NHWC #10554

Upsample support NHWC #10554

Uh oh!

Conversation

yihonglyu commented Feb 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

askhade Feb 22, 2022

Choose a reason for hiding this comment

Uh oh!

yihonglyu Feb 22, 2022

Choose a reason for hiding this comment

Uh oh!

yufenglee Feb 23, 2022

Choose a reason for hiding this comment

Uh oh!

askhade commented Feb 22, 2022

Uh oh!

yihonglyu commented Feb 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yihonglyu commented Feb 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yufenglee left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yihonglyu commented Feb 14, 2022 •

edited

Loading

yihonglyu commented Feb 22, 2022 •

edited

Loading

yihonglyu commented Feb 23, 2022 •

edited

Loading