Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CoreML ML Program Resize #21370

Merged
merged 11 commits into from
Jul 19, 2024
Prev Previous commit
Next Next commit
Fix some CI issues.
Remove redudant test.
  • Loading branch information
skottmckay committed Jul 17, 2024
commit 9d8be9d4439140e667a21a01fa7af17aeef2bc5b
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

#include <math.h>
#include <cmath>

#include "core/framework/tensorprotoutils.h"
#include "core/optimizer/initializer.h"
Expand Down Expand Up @@ -192,6 +192,19 @@
"Error getting validated scales");
num_scales = output_scales.size();

// special case linear downsample.
// the CoreML implementation seems to be flaky and gives different outputs on different OS versions.
// use bilinear_resize instead. we check in IsOpSupportedImpl that the downsample input is evenly
// divisible by the output size so there's no rounding involved.
if (is_linear && (output_scales[num_scales - 1] < 1.f || output_scales[num_scales - 2] < 1.f)) {
using_scales = false;
using_sizes = true;
num_sizes = num_scales;
output_sizes = input_shape;
// only the last to dims have their size changed
skottmckay marked this conversation as resolved.
Show resolved Hide resolved
output_sizes[input_rank - 2] = static_cast<int64_t>(input_shape[input_rank - 2] * output_scales[num_scales - 2]);
output_sizes[input_rank - 1] = static_cast<int64_t>(input_shape[input_rank - 1] * output_scales[num_scales - 1]);
}
} else {
ORT_RETURN_IF_NOT(GetValidatedResizeSizes(graph_viewer, node, input_shape, axes, output_sizes, logger),
"Error getting validated sizes");
Expand All @@ -200,7 +213,7 @@

#if defined(COREML_ENABLE_MLPROGRAM)
if (model_builder.CreateMLProgram()) {
using namespace CoreML::Specification::MILSpec;

Check warning on line 216 in onnxruntime/core/providers/coreml/builders/impl/resize_op_builder.cc

View workflow job for this annotation

GitHub Actions / Lint C++

[cpplint] reported by reviewdog 🐶 Do not use namespace using-directives. Use using-declarations instead. [build/namespaces] [5] Raw Output: onnxruntime/core/providers/coreml/builders/impl/resize_op_builder.cc:216: Do not use namespace using-directives. Use using-declarations instead. [build/namespaces] [5]

std::string_view coreml_op_type;
if (using_scales) {
Expand Down Expand Up @@ -265,7 +278,7 @@

AddOperationOutput(*op, *output_defs[0]);
model_builder.AddOperation(std::move(op));
} else

Check warning on line 281 in onnxruntime/core/providers/coreml/builders/impl/resize_op_builder.cc

View workflow job for this annotation

GitHub Actions / Lint C++

[cpplint] reported by reviewdog 🐶 If an else has a brace on one side, it should have it on both [readability/braces] [5] Raw Output: onnxruntime/core/providers/coreml/builders/impl/resize_op_builder.cc:281: If an else has a brace on one side, it should have it on both [readability/braces] [5]
#endif
{
std::unique_ptr<COREML_SPEC::NeuralNetworkLayer> layer = model_builder.CreateNNLayer(node);
Expand Down Expand Up @@ -308,6 +321,16 @@
return false;
}

// as we allow empty shapes in the checks done by BaseOpBuilder::HasSupportedInputs we explicitly check for an empty
// an empty input here to be consistent.
// this should never happen in a real model though as a dim with value 0 (i.e. no input data) would typically be a
// dynamic dimension where a previous step had no output (e.g. Loop of zero interations, NonZero with no matches,
// NonMaxSupression with no boxes).
if (std::find(input_shape.begin(), input_shape.end(), 0) != input_shape.end()) {
LOGS(logger, VERBOSE) << "Resize input shape has with dimension values of 0 which is not supported.";
return false;
}

const auto input_rank = input_shape.size();
if (input_params.create_mlprogram) {
if (input_rank < 3 || input_rank > 5) {
Expand Down Expand Up @@ -405,13 +428,13 @@
auto h_out = h_in * scale_h;
auto w_out = w_in * scale_w;

if (std::fmodf(float(h_in), h_out) != 0.f) {
if (std::fmod(float(h_in), h_out) != 0.f) {

Check warning on line 431 in onnxruntime/core/providers/coreml/builders/impl/resize_op_builder.cc

View workflow job for this annotation

GitHub Actions / Lint C++

[cpplint] reported by reviewdog 🐶 Using deprecated casting style. Use static_cast<float>(...) instead [readability/casting] [4] Raw Output: onnxruntime/core/providers/coreml/builders/impl/resize_op_builder.cc:431: Using deprecated casting style. Use static_cast<float>(...) instead [readability/casting] [4]
edgchen1 marked this conversation as resolved.
Show resolved Hide resolved
LOGS(logger, VERBOSE) << "Resize: downsampling output height: " << h_out
<< " is not a factor of input height: " << h_in;
skottmckay marked this conversation as resolved.
Show resolved Hide resolved
return false;
}

if (std::fmodf(float(w_in), w_out) != 0.f) {
if (std::fmod(float(w_in), w_out) != 0.f) {

Check warning on line 437 in onnxruntime/core/providers/coreml/builders/impl/resize_op_builder.cc

View workflow job for this annotation

GitHub Actions / Lint C++

[cpplint] reported by reviewdog 🐶 Using deprecated casting style. Use static_cast<float>(...) instead [readability/casting] [4] Raw Output: onnxruntime/core/providers/coreml/builders/impl/resize_op_builder.cc:437: Using deprecated casting style. Use static_cast<float>(...) instead [readability/casting] [4]
LOGS(logger, VERBOSE) << "Resize: downsampling output width: " << w_out
<< " is not a factor of input width: " << w_in;
return false;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ CoreMLExecutionProvider::CoreMLExecutionProvider(uint32_t coreml_flags)
: IExecutionProvider{onnxruntime::kCoreMLExecutionProvider},
coreml_flags_(coreml_flags),
coreml_version_(coreml::util::CoreMLVersion()) {
LOGS_DEFAULT(VERBOSE) << "CoreML version: " << coreml_version_;
if (coreml_version_ < MINIMUM_COREML_VERSION) {
LOGS_DEFAULT(ERROR) << "CoreML EP is not supported on this platform.";
}
Expand Down
22 changes: 21 additions & 1 deletion onnxruntime/core/providers/xnnpack/tensor/resize.cc
Original file line number Diff line number Diff line change
Expand Up @@ -68,9 +68,29 @@
InlinedVector<float> scale(4, 1.0F);
if (scale_tensor) {
const Initializer scale_val(*scale_tensor, node_unit.ModelPath());
if (scale_val.DataAsSpan<float>()[1] != 1.0F) {
const auto scales = scale_val.DataAsSpan<float>();
if (scales[1] != 1.0F) {
break;
}

// downsampling output seems to require the output size to be a factor of the input to match ONNX
if (scales[2] < 1.0f || scales[3] < 1.0f) {
// we also require input_shape to be known to check
int64_t h_in = x_shape->dim(2).dim_value();
int64_t w_in = x_shape->dim(3).dim_value();
if (h_in < 0 || w_in < 0) {
break;
}

float scale_h = scales[2];
float scale_w = scales[3];
float h_out = h_in * scale_h;
float w_out = w_in * scale_w;
if (std::fmod(float(h_in), h_out) != 0.f ||

Check warning on line 89 in onnxruntime/core/providers/xnnpack/tensor/resize.cc

View workflow job for this annotation

GitHub Actions / Lint C++

[cpplint] reported by reviewdog 🐶 Using deprecated casting style. Use static_cast<float>(...) instead [readability/casting] [4] Raw Output: onnxruntime/core/providers/xnnpack/tensor/resize.cc:89: Using deprecated casting style. Use static_cast<float>(...) instead [readability/casting] [4]
std::fmod(float(w_in), w_out) != 0.f) {

Check warning on line 90 in onnxruntime/core/providers/xnnpack/tensor/resize.cc

View workflow job for this annotation

GitHub Actions / Lint C++

[cpplint] reported by reviewdog 🐶 Using deprecated casting style. Use static_cast<float>(...) instead [readability/casting] [4] Raw Output: onnxruntime/core/providers/xnnpack/tensor/resize.cc:90: Using deprecated casting style. Use static_cast<float>(...) instead [readability/casting] [4]
break;
}
}
}

if (size_tensor) {
Expand Down
31 changes: 3 additions & 28 deletions onnxruntime/test/providers/cpu/tensor/resize_op_test.cc
Original file line number Diff line number Diff line change
Expand Up @@ -256,32 +256,6 @@ TEST(ResizeOpTest, ResizeOpLinearDownSampleTest_4DBilinear) {
run_test(true);
}

// Primarily to test CoreML using upsample_bilinear when the input size is evenly divisible by the output size
TEST(ResizeOpTest, ResizeOpLinearDownSampleTest_4DBilinear_EvenlyDivisible) {
OpTester test("Resize", 13);
std::vector<float> roi{};
std::vector<float> scales{1.0f, 1.0f, 0.5f, 0.5f};

test.AddAttribute("mode", "linear");

constexpr int64_t N = 1, C = 1, H = 2, W = 4;
std::vector<float> X = {
1.0f, 2.0f, 3.0f, 4.0f,
5.0f, 6.0f, 7.0f, 8.0f};

test.AddInput<float>("X", {N, C, H, W}, X);
test.AddInput<float>("roi", {0}, roi);
test.AddInput<float>("scales", {4}, scales, /*scales_in_initializer*/ true);

std::vector<float> Y = {3.5f, 5.5f};

test.AddOutput<float>("Y", {N, C, static_cast<int64_t>(H * scales[2]), static_cast<int64_t>(W * scales[3])}, Y);
// QNN: result diff
// TRT: Segmentation fault in A100
std::unordered_set<std::string> excluded_providers({kQnnExecutionProvider});
test.Run(OpTester::ExpectResult::kExpectSuccess, "", ExcludeTrtOnA100(excluded_providers));
}

TEST(ResizeOpTest, NhwcResizeOpLinearDownSampleTest_4DBilinear) {
OpTester test("Resize", 13);
std::vector<float> roi{};
Expand Down Expand Up @@ -358,11 +332,12 @@ TEST(ResizeOpTest, NhwcResizeOpLinearDownSampleTest_4DBilinear_int8) {
// Since NNAPI(TFLite) only using the scale calculate using the input/output size
// For the above test (ResizeOpLinearDownSampleTest_4DBilinear)
// The output size is [1,1,2,4].*[1,1,0.6,0.6]=[1,1,1,2]
// NNAPI will recaluclate the scales as the output size divided by input size
// NNAPI will recalculate the scales as the output size divided by input size
// scales = [1,1,1,2]./[1,1,2,4] = [1,1,0.5,0.5]
// See:https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/kernels/internal/reference/reference_ops.h
// So the result of the above example will be different than CPU EP
// Add the following 2 tests to test with scales valid to NNAPI
// Add the following 2 tests to test with scales valid to NNAPI.
// CoreML also doesn't handle a scale that doesn't divide the input size evenly.
TEST(ResizeOpTest, ResizeOpLinearDownSampleTest_4DBilinear1) {
// To test NNAPI EP, we need the scales/sizes to be in initializers
auto run_test = [](bool scales_in_initializer) {
Expand Down
Loading