Create an ML.Net sample showing image classification using images already loaded in memory, rather than from disk

[Webreaper]

I'm trying to put together an ML.Net image classification app. However, the samples I can find all assume that the images being classified are a folder of disk-based image files. This doesn't really help when processing already-in-memory images (for example, streaming video frames, etc).

I've tried adapting the example here to use bytes from a Bitmap pre-loaded from disk, but I don't seem to be getting the setup of the tensor data right, which means that I just get NaN results from the scoring.

It would be really useful if the ML.Net team could put together a sample where the input was a binary image - so for example, in the ScoreTensorFlowModel example above, if the GetTensorData method loaded a couple of JPEGs from disk, and then passed back the TensorData[] structure as an array of binary data from the images.

It seems, while googling, that a lot of people would find such an example extremely helpful - as there's a lot of people trying (and failing) to image-process video streams and other in-memory images using ML.Net and TF.

Thanks!

[yangpei1010110]

agree, I need detect a Mat from opencvsharp model in real time, but I don't know how to do it.

[michaelgsharp]

@JakeRadMSFT @luisquintanilla don't we already have this sample somewhere? Is it in one of your notebooks jake?

[luisquintanilla]

We don't have an E2E sample that I'm aware of, but we do have the one in the reference docs

machinelearning/docs/samples/Microsoft.ML.Samples/Dynamic/Trainers/MulticlassClassification/ImageClassification/ImageClassificationDefault.cs

Line 108 in 52ddbcd

TrySinglePrediction(fullImagesetFolderPath, mlContext, loadedModel);

@jwood803 is also working on a doc to provide guidance for image scenarios.

dotnet/docs#26537

[JakeRadMSFT]

Yeah I have a notebook ... let me track it down.

[JakeRadMSFT]

ImageClassificationBytesAndFile.txt

Attached is a Notebook (you can rename the extension to ipynb from txt) and then open it in VS with this extension:
https://marketplace.visualstudio.com/items?itemName=MLNET.notebook

The file should be compatible with any other IPYNB notebook editor (VS Code, Jupyter, etc ...)

I'll also include Code Snippets here ... let me know if you have any questions:

using Microsoft.ML.Data;

public class ModelInput
{
    [ColumnName(@"Label"), LoadColumn(0)]
    public string Label { get; set; }

    [ColumnName(@"ImageSource"), LoadColumn(1)]
    public string ImageSource { get; set; }

}

using Microsoft.ML.Data;

public class ModelInputBytes
{
    [ColumnName(@"Label")]
    public string Label { get; set; }

    [ColumnName(@"Features")]
    public byte[] ImageBytes { get; set; }

}

using Microsoft.ML.Data;

public class ModelOutput
{
    [ColumnName("PredictedLabel")]
    public string Prediction { get; set; }

    public float[] Score { get; set; }
}

using Microsoft.ML;

public class ConsumeClass
{
    public static ModelOutput Predict(ModelInput input, string MLNetModelPath)
    {
        MLContext mlContext = new MLContext();

        // Load model & create prediction engine
        ITransformer mlModel = mlContext.Model.Load(MLNetModelPath, out var modelInputSchema);

        ITransformer dataPreProcessTransform = LoadImageFromFileTransformer(input, mlContext);

        var predEngine = mlContext.Model.CreatePredictionEngine<ModelInput, ModelOutput>(dataPreProcessTransform.Append(mlModel));

        ModelOutput result = predEngine.Predict(input);
        return result;
    }

    public static ITransformer LoadImageFromFileTransformer(ModelInput input, MLContext mlContext)
    {
        var dataPreProcess = mlContext.Transforms.Conversion.MapValueToKey(@"Label", @"Label")
                                .Append(mlContext.Transforms.LoadRawImageBytes(@"ImageSource_featurized","", @"ImageSource"))
                                .Append(mlContext.Transforms.CopyColumns(@"Features", @"ImageSource_featurized"));

        var dataView = mlContext.Data.LoadFromEnumerable(new[] { input });
        var dataPreProcessTransform = dataPreProcess.Fit(dataView);
        return dataPreProcessTransform;
    }

    public static ModelOutput PredictFromBytes(ModelInputBytes input, string MLNetModelPath)
    {
        MLContext mlContext = new MLContext();

        // Load model & create prediction engine
        ITransformer mlModel = mlContext.Model.Load(MLNetModelPath, out var modelInputSchema);

        var predEngine = mlContext.Model.CreatePredictionEngine<ModelInputBytes, ModelOutput>(mlModel);

        ModelOutput result = predEngine.Predict(input);
        return result;
    }
}

using Microsoft.ML;

public class DataLoadClass
{
    public static IDataView LoadImageFromFile(MLContext mlContext, string trainDataPath)
    {
        IDataView trainData = mlContext.Data.LoadFromTextFile<ModelInput>(
        path: trainDataPath,
        hasHeader: true,
        separatorChar: '\t',
        allowQuoting: true,
        allowSparse: false);

        return trainData;
    }
}

using Microsoft.ML;

public class TrainClass
{
    public static ITransformer RetrainPipeline(MLContext context, IDataView trainData)
    {


        var dataPreProcess = context.Transforms.Conversion.MapValueToKey(@"Label", @"Label")
                                .Append(context.Transforms.LoadRawImageBytes(@"ImageSource_featurized", "", @"ImageSource"))
                                .Append(context.Transforms.CopyColumns(@"Features", @"ImageSource_featurized"));

        var trainingPipeline = context.MulticlassClassification.Trainers.ImageClassification(labelColumnName: @"Label")
                                .Append(context.Transforms.Conversion.MapKeyToValue(@"PredictedLabel", @"PredictedLabel"));

        var processedData = dataPreProcess.Fit(trainData).Transform(trainData);

        var model = trainingPipeline.Fit(processedData);

        return model;
    }
}

// See https://aka.ms/new-console-template for more information

using Microsoft.ML;
using SixLabors.ImageSharp;
using SixLabors.ImageSharp.PixelFormats;
using SixLabors.ImageSharp.Processing;
using System;
using System.IO;

var mlContext = new MLContext();
var imageData = DataLoadClass.LoadImageFromFile(mlContext, "ImageTrainData.tsv");

var rows = mlContext.Data.CreateEnumerable<ModelInput>(imageData, reuseRowObject: false)
                    .Take(5)
                    .ToList();

display(rows);

var pipeline = TrainClass.RetrainPipeline(mlContext, imageData);


mlContext.Model.Save(pipeline, imageData.Schema, "model.mlnet");

ModelInput modelInput = new ModelInput()
{
    ImageSource = @"C:\dev\datasets\weather-small\Cloudy\cloudy1.jpg",
};

ModelOutput prediction = ConsumeClass.Predict(modelInput, "model.mlnet");

Console.WriteLine("Prediction: " + prediction.Prediction);

var image = Image.Load<SixLabors.ImageSharp.PixelFormats.Rgb24>(@"C:\dev\datasets\weather\Sunrise\sunrise103.jpg");
image.Mutate(x => x.Resize(224, 224));
byte[] imageBytes;

using (MemoryStream memoryStream = new MemoryStream())
{
    image.SaveAsJpeg(memoryStream);
    imageBytes = memoryStream.ToArray();
}

ModelInputBytes modelInputBytes = new ModelInputBytes()
{
    ImageBytes = imageBytes,
};

var prediction2 = ConsumeClass.PredictFromBytes(modelInputBytes, "model.mlnet");
Console.WriteLine("Prediction: " + prediction2.Prediction);

[vpenades]

@JakeRadMSFT I'm having a similar problem, but in this case I need this for training.

The key difference is that when you do inferences, you usually pass one image at a time, which is fine to do it from memory.

But if you need to do training, things get more complicated, because you probably need thousands of input instances, and if each one has a bitmap, you can easily run out of memory just preparing the training data.

my training case scenarios are: taking frames from a video, and downloading images on demand from a private server.

As I propose in #6204 (close it if you feel it's redundant), I think what would be needed is some sort of callback to retrieve images on demand, or some interface that we would need to implement and pass to the loader, so the bitmaps would be loaded on demand from whatever source we need

[michaelgsharp]

Closing as the original question has already been answered and the question about the callback has an answer in #6204