Text loader v.s in-memory data structure in API reference samples

[wschin]

We often starts our trainer examples with text loader but recently I feel loading text into IDataView is not directly related to the actual training procedure. If we use

/// <summary>
/// Example with one binary label and 10 feature values.
/// </summary>
public class BinaryLabelFloatFeatureVectorSample
{
    public bool Label;

    [VectorType(_simpleBinaryClassSampleFeatureLength)]
    public float[] Features;
}

as our in-memory example, we can create more flexible examples like scikit-learn ones (where data matrix is float matrix) and make ML.NET's learning curve smoother (because users don't need to learn text loader, the loaded data, and trainer at the same time).

cc @shmoradims, @rogancarr, @sfilipi, @shauheen

#2780 shows a scikit-learn-style example for ML.NET. It is

• Self-contained --- To understand it, user doesn't need to look another document or use Visual Studio to search for those used functions. Notice that we can't rely on Visual Studio because not everyone is using it (1st partry and 3rd party experiences should the same!).
• End-to-end to C# developers --- because it trains a model over a C# List and get the prediction back as a C# List (The two ends are not IDataView so user doesn't need to learn IDataView to play with that API).
• Independent to external packages --- We shouldn’t expect a user who needs doc knows SamplesUtils.
• Production-friendly --- Doing prediction with C# data structure is included. That's how a trained model will be used in production.

[shmoradims]

I chatted with Wei-Sheng offline about this. I can summarize the cons and pros as follows:

• Pros of in-memory (Cons of Textloader): Simpler data pipeline and more focused on the actual trainer that's being documented. It doesn't involve downloading, TextLoader, featurization, etc., so there's less friction for users to understand the sample and adapt it to their scenario.
• Cons of in-memory (Pros of Textloader): More meaningful data (Adult UCI vs randomly generated data)

Wei-Sheng has strong preference for adopting in-memory data for our samples. I'm ambivalent. @rogancarr @sfilipi what do you think?

[bartczernicki]

I will add that one of the ways I pitch ML.NET to customers is that it allows you to put models directly into memory right next to your existing business logic/rules engines (Web APIs, ASP.NET MVC/Web Forms, Win Forms etc.). I realize this suggestion was for trainers and not inference, but once I amplify to developers (keep in mind they are not data scientists) you don't need to load new data and you probably already have it in-memory/in-process already it's one of the things I had to explain more than once.

[wschin]

@bartczernicki, Thanks a lot for your input. Looks like in-memory scenario is more closer to C# developers, right?

[rogancarr]

Let's do both kinds of examples, but I think we should define what kinds of samples that we're building.

1. API Docs: The samples in docs/samples/Microsoft.ML.Samples/ appear on https://docs.microsoft.com in-lined into the API documentation (e.g. for GAMs). These samples should focus on the learner and do as little data manipulation as possible. We have been moving the data loading into helper functions like this

var data = LoadHousingRegressionDataset(mlContext);

so that we don't bog down the documentation on how a learner works and what produces with tons of lines describing data loading. I am hesitant to have these focus on more than just the API in question.

2. Samples: We have a sample repository, https://github.com/dotnet/machinelearning-samples, that has long-form samples showing how to use ML.NET for long-form, end-to-end samples, and are pulled into the samples portion of https://docs.microsoft.com. This repository is where we should focus on building out copy-and-paste examples that people can use for file-based and memory-based training. I'd focus energy here if we want to give a focus for in-memory training and push for documentation similar to sklearn.

[wschin]

@rogancarr, the GAM example has too many thing not directly related to the trainer. Moving things into a function doesn't really increase the flexibility. Assume that I already have a data matrix and label vector like those in many scikit-learn trainer examples:

X = [[0], [1], [2], [3]]
Y = [0, 1, 2, 3]

What is the gap between GAM example and the training pipeline I want? I'd imagine I need to

• Look into the data structure returned by

var data = LoadHousingRegressionDataset(mlContext);

If I were smart enough, I will clone ML.NET, open Visual Studio, search for the definition of LoadHousingRegressionDataset, go through all lines inside, realize all I need is just an IDataView, and finally define my own class like this

        /// <summary>
        /// Example with one binary label and 10 feature values.
        /// </summary>
        public class BinaryLabelFloatFeatureVectorSample
        {
            public bool Label;

            [VectorType(_simpleBinaryClassSampleFeatureLength)] // _simpleBinaryClassSampleFeatureLength =  10, for example.
            public float[] Features;
        }

Oh wait, how should I know I need to define my own class? Why should I have Visual Studio? Note that finding an example class with a vector field (i.e., public float[] Features here) was a nightmare back to months ago (I guess we have a better documentation now but I still want to emphasize that any small hole here could block a new user forever).

• Then, I need to realize that I need to use this undocumented (I call everything without an example undocumented) function.

Those points mentioned above might explain why scikit-learn trainer examples always do things as simple as

X = [[0], [1], [2], [3]]
Y = [0, 1, 2, 3]
clf = svm.SVC(gamma='scale', decision_function_shape='ovo')
clf.fit(X, Y)

We have many end-to-end examples. However, that end-to-end doesn't really mean end-to-end if we define an end is something easily-understandable to users.

[TomFinley]

Without interfering too much in matters of samples and documentation, my own sympathies at least at first glance are strongly with @wschin.

As @bartczernicki points out, it is inevitable even from the first example that you need to introduce the subject of in memory consumption anyway, since that is the most plausible way the trained models will be consumed. When doing predictions, you're not going to be consuming from a file. So, you need to have that.

Now IDataLoader is an essential thing to cover at some point, but I agree that it could wait till "lesson 2" or "lesson 4" or somesuch. In lesson 1 we will have our hands full enough with the "trinity" of IEstimator/ITransformer/IDataView (#581), I think.

Beyond that core and necessary part, I think about what I have to explain. I think about what I have to explain in @wschin's world: I can say, "hey look, here's an array of length 150 of IrisExample with these five fields, and you can convert these instances into one of our IDataView structures like so" and I think a C# developer will just sort of get this. But to say, "look man, we start with this file, and it has tabs, but we can sort of load it by setting the text loader like so to coinfigure it, but just be aware it's not actually in memory because these structures are lazily evaluated so it loads it when needed in a streaming fashion, but you can control that with these additional things," that's a harder conversation. That was a conversation we had to have when this code was exposed as a command-line tool, but now we're writing an API, and this is an opportunity to have an easier conversation. At least, to start.

Otherwise we inevitably have to get into conversations about in-memory vs. out-of-memory structures, the implications of lazily-evaluated structures like IDataView w.r.t. this, the whole interface for parsing out files, and all this all at once, which I think maybe might confuse people. And is not in any event how I think a C# developer thinks about their data.

(I might at least mention in @wschin 's world that this is a simple example and we have other mechanisms for handling out-of-core data -- imagining myself as an outside reader, I would get suspicious myself if I did not receive some assurance on this point --, but I wouldn't hit people over the head with the details of how that is done right off the bat.)

Again, not trying to interfere in matters of documentation and samples. Just registering my own thoughts on this subject, feel free to ignore.

[Ivanidzo4ka]

I would prefer following way:
If you have example of data transformation: use in memory format.
If you have example of trainer: use data reading.

My reasoning is following:

If I get to the learner I probably already build some pipeline for my data. So I would prefer to show user how to specify certain options and what columns learner produce, how to make prediction, how to get metrics out of it. Basically focus on learner rather than pipeline building.
I like python snippets by @wschin , but C# is not so expressive, and we will end up with lot of extra code in our samples which would take away whole point of sample.

Data transformation: I think it's necessary to have in memory examples since we transform data. So we should show before and after stages. Which is hard with data reading.At least in cases where we work with data itself rather than schema of data, for data schema I would prefer lesser footprint

Maybe bad, but still example:
https://docs.microsoft.com/en-us/dotnet/api/system.drawing.image.pixelformat?view=netframework-4.7.2
No one constructing bitmap in memory, they just read it from somewhere.
Or this one:
https://docs.microsoft.com/en-us/dotnet/api/system.speech.recognition.grammar.speechrecognized?view=netframework-4.7.2
No one even cares about data, they just trying to show what is interface and how you can use it.

@JRAlexander has probably thousand times more expertise than we are, so can be nice to add him to this discussion.

[wschin]

A trainer is definitely not a thing should start with data reading. Why should a user learn an IDataView type systems and our text format before doing training? Training a linear model is not a ML.NET specific thing; it should be easily doable to everyone who only knows C# fundamentals. Thinking about scikit-learn examples, do they ask user to learn numpy sparse data structure before training a SVM?

[zeahmed]

Thanks @wschin for bring this up.

I think this is the matter of showing ML.NET's preferred way of importing data into pipeline. Either it is using C# structure (in-memory) or from file (using text loader).

If all of our samples contain the in-memory streaming then our message to user is that the preferred way to build model in ML.NET is using in-memory streams.

Scikit example here does not make much sense because scikit does not provide data loading support (use numpy). So, for them data loading is not concern at all. If we are going to be in the same boat then there is no-doubt about using in-memory streaming.

There is also a question of how many samples will be sufficient for the training of the transform or learner. If all of learners and trainers can be trained on just 5-10 examples then it is perfect otherwise creation of data will fill up the sample.

[wschin]

C# API suppose to primarily work with C# data structures, not files! Scikit-learn example is an example for trainers, why it needs to load data? Training is just training; loading is just loading. Why do we need to mix them in a training API's example?