CountVectorizer Alternative?

[XavierGeerinck]

System information

Product Information:
Version: 2.1.200
Commit SHA-1 hash: 2edba8d7f1

Runtime Environment:
OS Name: Mac OS X
OS Version: 10.13
OS Platform: Darwin
RID: osx.10.13-x64
Base Path: /usr/local/share/dotnet/sdk/2.1.200/

Microsoft .NET Core Shared Framework Host

Version : 2.0.7
Build : 2d61d0b043915bc948ebf98836fefe9ba942be11

Issue

• What did you do?
  I am trying to create a model that based on tags (such as sunny, windy, outdoor, ...) predicts the number of comments. This format looks like:

likes;tags;
123;tag1,tag2,tag3;
12;tag1;
654;tag2;tag4

I know scikit has this through the CountVectorizerModel but I'm looking for a ML.NET alternative. Is anyone aware of this?

• What happened?
  I couldn't find an alternative

• What did you expect?
  Being able to use a CountVectorizerModel

Source code / logs

See above for more information

[TomFinley]

Hi @thebillkidy , thanks for the question... Python's CountVectorizer reads a file, tokenizes. If you're looking for an equivalent workflow in ML.NET, I guess we'd first have the TextLoader to get the pipeline started loading it from a file, then some mechanism to tokenize it and vectorize it -- the "simplest" of these would be the text transform, but you would I suppose have the ability to do something a bit more custom by stringing together a set of tokenizers.

We've generally tried to have composable components -- in particular, relevant to this sklearn component, we feel pretty strongly like treating the loading of data and featurization as two separate steps is important.

[XavierGeerinck]

Hi Tom, thanks a lot for your reply! What I am currently really struggling with is how I can test this pipeline without actually training all the time? As in, I would like to run the pipeline and just view the output of let's say my transformers. Is there a way on how I can do this? :) because I didn't see a method in the Pipeline learner

[shauheen]

EDIT;
Hi,

Maybe something like the following code could help you get started with the training. Just keep in mind that the small data that you provided above, is probably insufficient for any meaningful results. Also you want to make sure that the format is correct (i.e. separator for lines and tag values are correct).

public class LikesDataOut
{
    [ColumnName("Score")]
    public float score;
}

public class LikesDataIn
{
	[Column(ordinal: "0", name: "Label")]
	public float likes;

	[Column(ordinal: "1")]
    public string tags;
}

public static void Main(string[] args)
{
	string dataPath = "/test.txt";
	var pipeline = new LearningPipeline();
	pipeline.Add(new TextLoader(dataPath).CreateFrom<LikesDataIn>(useHeader: true, separator: ';'));
	pipeline.Add(new WordTokenizer("tags") { TermSeparators = "," });
	pipeline.Add(new CategoricalOneHotVectorizer("tags"){ OutputKind = CategoricalTransformOutputKind.Bag } );
	pipeline.Add(new ColumnConcatenator(outputColumn: "Features", "tags"));
	pipeline.Add(new StochasticDualCoordinateAscentRegressor());
	PredictionModel<LikesDataIn, LikesDataOut> model = pipeline.Train<LikesDataIn, LikesDataOut>();
	model.WriteAsync("likes_model.zip");

}

Once you train the model and save it, you can then load that model and just score to get results. If you are looking for interactive debugging you can do that in Visual studio by stepping through the training code.

[XavierGeerinck]

Hi @shauheen thanks a lot for this example! However when I run this, I do get this error: "System.ArgumentOutOfRangeException: Source column 'Features' not found" is that normal?

[codemzs]

@thebillkidy can you please paste a lines from the dataset? From the code that you sent me your data path is "static string dataPath = "data.csv" which means it is comma separated but the TextLoader separator is ";", can you please change that to a comma?
pipeline.Add(new TextLoader(dataPath).CreateFrom(useHeader: true, separator: ','));

[shauheen]

@thebillkidy, Sorry, I edited the code snippet above slightly, this should work now. Please let me know.

[TomFinley]

There was also the question I believe from @thebillkidy about checking this out without training. While you do not have to train a predictor (so, a transform-only pipeline is possible), many of the transforms do still require training (e.g., CategoricalOneHotVectorizer as used in @shauheen 's example is building up a dictionary of observed terms). Since you're familiar with sklearn, think of Train as being somewhat like pipeline components having a fit function, if that helps.

[XavierGeerinck]

Awesome, thanks a lot @shauheen and @codemzs I got it working now :)

@TomFinley this was more from the insights of a developer :) I am not a data scientist (I'm a CSA actually) but wanted to try this, which is why it's weird that I can not see the transformations happening as I'm preparing data. But I guess this kind of makes sense now

[abgoswam]

DRI RESPONSE: Issue seems to have been answered. I plan to close the issue withing the next few days.

Also, with the new API (see #583), you will be able to access the output of transforms without necessarily having a predictor/trainer at the end. Please check out this unit test