CookBook sample - Could not find input column 'CategoricalOneHot'

[sdg002]

System information

• OS version/distro:Windows 10
• .NET Version (eg., dotnet --info): 4.6

Issue

• What did you do?
  I am trying to follow the sample code from the GitHub article
  https://github.com/dotnet/machinelearning/blob/master/docs/code/MlNetCookBook.md#how-do-i-train-my-model-on-categorical-data
  and get an understanding of how to get one-hot encoding to work

• What happened?
  Run time error message - Could not find input column 'CategoricalOneHot'
  Parameter name: inputSchema'

• What did you expect?
  I expected the code to run without errors and then I should be able to examine how the data has been transformed

Source code / logs

        // Build several alternative featurization pipelines.
        var pipeline =
            // Convert each categorical feature into one-hot encoding independently.
            mlContext.Transforms.Categorical.OneHotEncoding("CategoricalFeatures", "CategoricalOneHot")
            // Convert all categorical features into indices, and build a 'word bag' of these.
            .Append(mlContext.Transforms.Categorical.OneHotEncoding("CategoricalFeatures", "CategoricalBag", Microsoft.ML.Transforms.Categorical.OneHotEncodingTransformer.OutputKind.Bag))
            // One-hot encode the workclass column, then drop all the categories that have fewer than 10 instances in the train set.
            .Append(mlContext.Transforms.Categorical.OneHotEncoding("Workclass", "WorkclassOneHot"))
            .Append(mlContext.Transforms.FeatureSelection.SelectFeaturesBasedOnCount("WorkclassOneHot", "WorkclassOneHotTrimmed", count: 10));

Please paste or attach the code or logs or traces that would be helpful to diagnose the issue you are reporting.

[sfilipi]

Hi @sdg002,
thanks for reporting.

I cannot see the schema of your data, but your first pipeline estimator, the OneHotEncoding, expect the schema to have a column called "CategoricalOneHot". Is there such a column in your dataView object?

Can you do dataView.Preview() before starting the pipeline, and inspect the names of the columns?

[sdg002]

Hi @sfilipi ,

Thanks for looking into this. Here is the code snippet from the CookBook that I have been trying to run and this leads to Could not find input column 'CategoricalOneHot'.

Please note that I had to make a couple of lines of changes to the code snippet in the CookBook to get it to compile. (See below)

Link to CookBook

https://github.com/dotnet/machinelearning/blob/master/docs/code/MlNetCookBook.md#how-do-i-train-my-model-on-categorical-data
I hope I am following the correct link?

Code from CookBook

    [TestMethod]
    public void CookBook_CategoricalData()
    {
        var mlContext = new Microsoft.ML.MLContext();
        //Changed CreateTextReader to CreateTextLoader
        var reader = mlContext.Data.CreateTextLoader(new[]
            {
                new TextLoader.Column("Label", DataKind.BL, 0),
                // We will load all the categorical features into one vector column of size 8.
                new TextLoader.Column("CategoricalFeatures", DataKind.TX, 1, 8),
                // Similarly, load all numerical features into one vector of size 6.
                new TextLoader.Column("NumericalFeatures", DataKind.R4, 9, 14),
                // Let's also separately load the 'Workclass' column.
                new TextLoader.Column("Workclass", DataKind.TX, 1),
            },
                        hasHeader: true
        );
        // Read the data.
        var data = reader.Read(_dataPath);


        // Inspect the first 10 records of the categorical columns to check that they are correctly read.
        var catColumns = data.GetColumn<string[]>(mlContext, "CategoricalFeatures").Take(10).ToArray();

        // Build several alternative featurization pipelines.
        var pipeline =
            // Convert each categorical feature into one-hot encoding independently.
            mlContext.Transforms.Categorical.OneHotEncoding("CategoricalFeatures", "CategoricalOneHot")
            // Convert all categorical features into indices, and build a 'word bag' of these.
            //.Append(mlContext.Transforms.Categorical.OneHotEncoding("CategoricalFeatures", "CategoricalBag", CategoricalTransform.OutputKind.Bag)) //This line did not compile
            .Append(mlContext.Transforms.Categorical.OneHotEncoding("CategoricalFeatures", "CategoricalBag", Microsoft.ML.Transforms.Categorical.OneHotEncodingTransformer.OutputKind.Bag))
            // One-hot encode the workclass column, then drop all the categories that have fewer than 10 instances in the train set.
            .Append(mlContext.Transforms.Categorical.OneHotEncoding("Workclass", "WorkclassOneHot"))
            //.Append(mlContext.Transforms.FeatureSelection.CountFeatureSelectingEstimator("WorkclassOneHot", "WorkclassOneHotTrimmed", count: 10)); //Did not compile
            .Append(mlContext.Transforms.FeatureSelection.SelectFeaturesBasedOnCount("WorkclassOneHot", "WorkclassOneHotTrimmed", count: 10));

        // Let's train our pipeline, and then apply it to the same data.
        var transformedData = pipeline.Fit(data).Transform(data);

        // Inspect some columns of the resulting dataset.
        var categoricalBags = transformedData.GetColumn<float[]>(mlContext, "CategoricalBag").Take(10).ToArray();
        var workclasses = transformedData.GetColumn<float[]>(mlContext, "WorkclassOneHotTrimmed").Take(10).ToArray();

    }

Sample CSV from Adult dataset

Changes I had to make to the code snippet

[sfilipi]

Hi @sdg002 ,

In version 0.9 we swapped the order of the columns for the estimators, and it loos like we missed updating this snippet of the cookbook.

The order of the arguments, is OutputColumn, InputColumn: so the colum that appears second, or at the end of the estimator parameters, should exist in the dataview.

If you swap the order
mlContext.Transforms.Categorical.OneHotEncoding("CategoricalOneHot", "CategoricalFeatures")

see the test with that same pipeline here:
https://github.com/dotnet/machinelearning/blob/master/test/Microsoft.ML.Tests/Scenarios/Api/CookbookSamples/CookbookSamplesDynamicApi.cs#L371 .

Note: leaving this issue open, and we'll close it with the PR that fixes the cookbook.

[sdg002]

Yes. It worked after changing the order. Thanks for looking into this.

[sfilipi]

Resolved with #2494