"Onnx Type not supported" error while importing Custom Vision ONNX model

[venkateshkrc]

System information

• OS version/distro: Win 10 Pro Build 17763.503
• .NET Version (eg., dotnet --info): .Net FrameWork 4.7.2
• ML.NET Version:1.1.0
• ONNX Version:: 1.0 / 1.2

Issue

• What did you do?
  Exported a image classification(General(Compact)) ONNX model from Custom Vision and attempted to generate a ONNX Estimator in ML.NET

• What happened?
  The model loading failed with error "Onnx type not supported".
  This happens with all image classification ONNX models exported from Custom Vision.
  Corresponding Tensorflow models work well in ML.NET.

• What did you expect?
  It is possible to load ONNX models exported from Tensorflow/Scikit-learn. ML.NET should be able to accept ONNX models from Custom Vision.

Source code / logs

var onnxEstimator = mLContext.Transforms.ApplyOnnxModel(OutputColumnNames, InputColumnNames, OnnxModelPath);

Files

onnx12.zip

[wschin]

With #3881, you can do something similar to the following function to get ONNX outputs.

        private class ImageDataPoint3_227_227
        {
            /// <summary>
            /// Height of <see cref="Image"/>.
            /// </summary>
            private const int height = 227;

            /// <summary>
            /// Width of <see cref="Image"/>.
            /// </summary>
            private const int width = 227;

            /// <summary>
            /// Image will be consumed by ONNX image multiclass classification model.
            /// </summary>
            [ImageType(height, width)]
            public Bitmap Image { get; set; }

            public ImageDataPoint3_227_227()
            {
                Image = null;
            }

            public ImageDataPoint3_227_227(Color color)
            {
                Image = new Bitmap(width, height);
                for (int i = 0; i < width; ++i)
                    for (int j = 0; j < height; ++j)
                        Image.SetPixel(i, j, color);
            }
        }

        public void TestCustomVisionOnnxModel()
        {
            var modelFile = Path.Combine(Directory.GetCurrentDirectory(), "customvision", "onnx12.onnx");

            // Create in-memory data points. Its Image/Scores field is the input/output of the used ONNX model.
            var dataPoints = new ImageDataPoint3_227_227[]
            {
                new ImageDataPoint3_227_227(Color.Red),
                new ImageDataPoint3_227_227(Color.Green)
            };

            // Convert training data to IDataView, the general data type used in ML.NET.
            var dataView = ML.Data.LoadFromEnumerable(dataPoints);

            // Create a ML.NET pipeline which contains two steps. First, ExtractPixle is used to convert the 224x224 image to a 3x224x224 float tensor.
            // Then the float tensor is fed into a ONNX model with an input called "data_0" and an output called "softmaxout_1". Note that "data_0" and
            // "softmaxout_1" are model input and output names stored in the used ONNX model file. Users may need to inspect their own models to
            // get the right input and output column names.
            var pipeline = ML.Transforms.ExtractPixels("data", "Image")                   // Map column "Image" to column "data_0"
                .Append(ML.Transforms.ApplyOnnxModel(new[] { "classLabel", "loss" }, new[] { "data" }, modelFile)); // Map column "data_0" to column "softmaxout_1"
            var model = pipeline.Fit(dataView);
            var transformedDataView = model.Transform(dataView);

            var predictColumn = transformedDataView.Schema["classLabel"];
            var outputColumn = transformedDataView.Schema["loss"];
            using (var curs = transformedDataView.GetRowCursor(predictColumn, outputColumn))
            {
                VBuffer<ReadOnlyMemory<char>> label = default;
                IEnumerable<IDictionary<string, float>> buffer = null;
                var getMapSequence = curs.GetGetter<IEnumerable<IDictionary<string, float>>>(outputColumn);
                var getPredictedLabel = curs.GetGetter<VBuffer<ReadOnlyMemory<char>>>(predictColumn);
                while (curs.MoveNext())
                {
                    getPredictedLabel(ref label);
                    getMapSequence(ref buffer);
                    Assert.Equal(1, label.Length);
                    Assert.Single(buffer);
                    var dictionary = buffer.First();
                    Assert.Equal(2, dictionary.Count());
                }
            }
        }

If you want to load the prediction results back to memory, you can also do

        private class ImageDataPoint3_227_227
        {
            /// <summary>
            /// Height of <see cref="Image"/>.
            /// </summary>
            private const int height = 227;

            /// <summary>
            /// Width of <see cref="Image"/>.
            /// </summary>
            private const int width = 227;

            /// <summary>
            /// Image will be consumed by ONNX image multiclass classification model.
            /// </summary>
            [ImageType(height, width)]
            public Bitmap Image { get; set; }

            public ImageDataPoint3_227_227()
            {
                Image = null;
            }

            public ImageDataPoint3_227_227(Color color)
            {
                Image = new Bitmap(width, height);
                for (int i = 0; i < width; ++i)
                    for (int j = 0; j < height; ++j)
                        Image.SetPixel(i, j, color);
            }
        }

        private class TransformedImageDataPoint3_227_227
        {
            public string[] classLabel { get; set; }

            [OnnxSequenceType(typeof(IDictionary<string, float>))]
            public IEnumerable<IDictionary<string, float>> loss { get; set; }
        }

        /// <summary>
        /// A test to check if sequence output works.
        /// </summary>
        [OnnxFact]
        public void TestCustomVisionOnnxModel()
        {
            var modelFile = Path.Combine(Directory.GetCurrentDirectory(), "customvision", "onnx12.onnx");

            // Create in-memory data points. Its Image/Scores field is the input/output of the used ONNX model.
            var dataPoints = new ImageDataPoint3_227_227[]
            {
                new ImageDataPoint3_227_227(Color.Red),
            };

            // Convert training data to IDataView, the general data type used in ML.NET.
            var dataView = ML.Data.LoadFromEnumerable(dataPoints);

            // Create a ML.NET pipeline which contains two steps. First, ExtractPixle is used to convert the 224x224 image to a 3x224x224 float tensor.
            // Then the float tensor is fed into a ONNX model with an input called "data_0" and an output called "softmaxout_1". Note that "data_0" and
            // "softmaxout_1" are model input and output names stored in the used ONNX model file. Users may need to inspect their own models to
            // get the right input and output column names.
            var pipeline = ML.Transforms.ExtractPixels("data", "Image")                   // Map column "Image" to column "data_0"
                .Append(ML.Transforms.ApplyOnnxModel(new[] { "classLabel", "loss" }, new[] { "data" }, modelFile)); // Map column "data_0" to column "softmaxout_1"
            var model = pipeline.Fit(dataView);
            var transformedDataView = model.Transform(dataView);

            // Convert IDataView back to IEnumerable<ImageDataPoint> so that user can inspect the output, column "softmaxout_1", of the ONNX transform.
            // Note that Column "softmaxout_1" would be stored in ImageDataPont.Scores because the added attributed [ColumnName("softmaxout_1")]
            // tells that ImageDataPont.Scores is equivalent to column "softmaxout_1".
            var transformedDataPoints = ML.Data.CreateEnumerable<TransformedImageDataPoint3_227_227>(transformedDataView, false).ToList();

            for (int i = 0; i < transformedDataPoints.Count; ++i)
            {
                Assert.Single(transformedDataPoints[i].loss);
                var dictionary = transformedDataPoints[i].loss.First();
                Assert.Equal(2, dictionary.Count());
            }

[venkateshkrc]

@wschin Thanks for the update.