Problem

With 1.0.0-preview bits, it is currently harder than it needs to be to use ML.NET inside an ASP.NET service or application. The first problem users hit is whether they can cache a PredictionEngine statically and reuse it for multiple requests. As described in #1789, you cannot use a PredictionEngine on multiple threads at the same time. Doing so will cause problems in your application.

Thus the recommendation is to use a pooling technique, but writing one from scratch is rather hard and potentially error prone.

Also, by default the MLContext's Log operation is not aware of any logging infrastructure currently used by ASP.NET apps/services. Thus the log goes nowhere, and is lost.

Proposal

We propose to add a new library (Microsoft.ML.Extensions?, Microsoft.Extensions.ML?) that is aware of both Microsoft.ML and Microsoft.Extensions.DependencyInjection/Microsoft.Extensions.Logging and glues the two together. This should make it much easier to consume ML.NET models inside ASP.NET apps/services, as well as any other app model that integrates with the Microsoft.Extensions.* libraries.

Usage

Adding a new ML.NET model into an ASP.NET application could be as simple as two steps:

1. Add a PredictionEnginePool in your Startup.cs:

    // This method gets called by the runtime. Use this method to add services to the container.
    public void ConfigureServices(IServiceCollection services)
    {
        services
            .AddPredictionEnginePool<SentimentIssue, SentimentPrediction>("SentimentModel.zip");

        // other service configuration
    }

2. In any controller that needs to make a prediction, inject the PredictionEngine pool in the constructor, and use it where necessary:

[ApiController]
public class PredictionController : ControllerBase
{
    private PredictionEnginePool<SentimentIssue, SentimentPrediction> _predictionEnginePool;

    public PredictionController(PredictionEnginePool<SentimentIssue, SentimentPrediction> predictionEnginePool)
    {
        _predictionEnginePool = predictionEnginePool;
    }

    [HttpGet()]
    public ActionResult<SentimentPrediction> GetSentiment([FromQuery]SentimentIssue input)
    {
        return _predictionEnginePool.Predict(input);
    }
}

Other potential scenarios

1. Being able to add a model .zip file from sources other than a file path
   1. Azure Blob Storage
   2. A SQL Database
   3. Any URL
2. Being able to automatically reload an updated model, if the file/URL changes (using FileWatcher or ETag or some other mechanism).
3. Being able to have different "named" models for scenarios like A/B testing where you want 90% of users to get Model A and 10% to get Model B.

@glennc @CESARDELATORRE @glebuk @TomFinley