Fault convert to ONNX object detection model

[ogoun]

System Information:

• OS & Version: Windows 10
• ML.NET Version: ML.NET v3.0.0-preview.23266.6
• .NET Version: .NET 7.0

Describe the bug
Fault conversion to ONNX

To Reproduce
Steps to reproduce the behavior:

1. Train object detection model by tutorial (https://devblogs.microsoft.com/dotnet/object-detection-ml-dotnet-model-builder/)
2. Try convert model

public class ModelInput
        {
            [LoadColumn(0)]
            [ColumnName(@"Labels")]
            public string[] Labels { get; set; }

            [LoadColumn(1)]
            [ColumnName(@"Image")]
            [Microsoft.ML.Transforms.Image.ImageType(640, 640)]
            public MLImage Image { get; set; }

            [LoadColumn(2)]
            [ColumnName(@"Box")]
            public float[] Box { get; set; }

        }

public static void Export()
        {
            var onnxPath = "test.onnx";
            var mlContext = new MLContext();
            ITransformer mlModel = mlContext.Model.Load(MLNetModelPath, out var _);

            var input = new ModelInput[30];
            for (int i = 0; i < input.Length; i++)
            {
                input[i] = new ModelInput
                {
                    Image = MLImage.CreateFromFile(@"demo.jpg")
                };
            }
            IDataView trainingData = mlContext.Data.LoadFromEnumerable(input);
            using (var onnx = File.Open(onnxPath, FileMode.OpenOrCreate))
            {
                mlContext.Model.ConvertToOnnx(mlModel, trainingData, onnx);
            }
        }

3. Got error
   System.InvalidOperationException: 'Unsupported type: Microsoft.ML.Data.MLImage'

Expected behavior
Got onnx model

Additional context
If I try to represent the input as an array [B,3,W,H], it says that it expects an image, when I pass the image, it says that it does not support it.

[dhienoz]

I also got the same issue. Any updates on this?

[MasterNaz]

Based on my research, there is no documented way to convert an ML.NET model for object detection into the ONNX format. The issue seems to be the MLImage object. The reason for this appears to be that the ONNX Estimator expects the input to be in certain formats, and MLImage isn't one of them (see also the reply from @LittleLittleCloud here to an simlar question).

After looking into ONNX, I had an idea of how it could work. Unfortunately, I lack some knowledge about how the training process of ModelBuilder works, but I have a hunch that the trained model's architecture in the saved ML.NET file contains the model input. From the documentation, it seems that simply converting an ML.NET model that contains an MLImage to the ONNX format would be impossible. Therefore, I think the ModelInput needs to be changed before training the model.

If retraining your model is an option:

My approach would be to modify the code generated by ModelBuilder so that our model input contains only data types that the ONNX Estimator can handle. We would then retrain our ML.NET model and save it in the ONNX format.

The steps to do this would be:

1. Modifying our ModelInput/ModelOutput
   Convert the MLImage into a data type that the ONNX Estimator can handle. Perhaps we could change/convert the image/model input and output to something like:

   public float[] ImageAsTensor { get; set; }
   // Additionally, if we want to avoid changing the logic of some functions in the generated code, it would be wise to save the image dimensions.
   public float ImageWidth { get; set; }
   public float ImageHeight { get; set; }

2. Writing additional helper functions
   Create a function like ConvertMLImageToTensor.

3. Extensive modification of the code generated by ModelBuilder
   Modify the code to handle the datatype float[] ImageAsTensor instead of MLImage Image.

4. Updating the save method
   Update the save method to use Microsoft.ML.Model.OnnxConverter to save an ONNX file.

5. Calling the Train function

I haven't tried this, so there is no guarantee that it will work.

Alternatively, instead of making so many modifications to your code and training the model locally on the CPU/GPU, it would likely be much more time-efficient to simply select Azure in the environments. This option directly outputs a model in the ONNX format.

[vpenades]

I am having the same problem, I think ML should consider being more ONNX export friendly on the template builders.

So, for example, before training step, to have an option like this:

☑ do all neccesary changes to ensure the trained model is exportable to ONNX

So if that option is checked, the input columns are modified to be Onnx friendly, as @MasterNaz has suggested.

And afterwards, if the trained model is Onnx friendly, then add another option to include the code to export to ONNX

Another option would be, given that MLImage is specific type with a well known conversion to a tensor, to reinterpret Microsoft.ML.Data.MLImage as a tensor when creating the ONNX model instead of throwing an exception