Skip to content

Updated sample for Concatenate API. #3262

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Apr 10, 2019
Merged
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 44 additions & 35 deletions docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Concatenate.cs
Original file line number Diff line number Diff line change
@@ -1,64 +1,73 @@
using System;
using System.Collections.Generic;
using Microsoft.ML;
using Microsoft.ML.Data;

namespace Microsoft.ML.Samples.Dynamic
namespace Samples.Dynamic
{
public static class ConcatTransform
public static class Concatenate
{
public static void Example()
{
// Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
// as well as the source of randomness.
var mlContext = new MLContext();

// Get a small dataset as an IEnumerable and them read it as ML.NET's data type.
var data = SamplesUtils.DatasetUtils.GetInfertData();
var trainData = mlContext.Data.LoadFromEnumerable(data);
// Create a small dataset as an IEnumerable.
var samples = new List<InputData>()
{
new InputData(){ Feature1 = 0.1f, Feature2 = new[]{ 1.1f, 2.1f, 3.1f}, Feature3 = 1 },
new InputData(){ Feature1 = 0.2f, Feature2 = new[]{ 1.2f, 2.2f, 3.2f}, Feature3 = 2 },
new InputData(){ Feature1 = 0.3f, Feature2 = new[]{ 1.3f, 2.3f, 3.3f}, Feature3 = 3 },
new InputData(){ Feature1 = 0.4f, Feature2 = new[]{ 1.4f, 2.4f, 3.4f}, Feature3 = 4 },
new InputData(){ Feature1 = 0.5f, Feature2 = new[]{ 1.5f, 2.5f, 3.5f}, Feature3 = 5 },
new InputData(){ Feature1 = 0.6f, Feature2 = new[]{ 1.6f, 2.6f, 3.6f}, Feature3 = 6 },
};

// Preview of the data.
//
// Age Case Education induced parity pooled.stratum row_num ...
// 26.0 1.0 0-5yrs 1.0 6.0 3.0 1.0 ...
// 42.0 1.0 0-5yrs 1.0 1.0 1.0 2.0 ...
// 39.0 1.0 0-5yrs 2.0 6.0 4.0 3.0 ...
// 34.0 1.0 0-5yrs 2.0 4.0 2.0 4.0 ...
// 35.0 1.0 6-11yrs 1.0 3.0 32.0 5.0 ...
// Convert training data to IDataView.
var dataview = mlContext.Data.LoadFromEnumerable(samples);

// A pipeline for concatenating the Age, Parity and Induced columns together into a vector that will be the Features column.
// A pipeline for concatenating the "Feature1", "Feature2" and "Feature3" columns together into a vector that will be the Features column.
// Concatenation is necessary because learners take **feature vectors** as inputs.
Copy link
Member

@sfilipi sfilipi Apr 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

** [](start = 64, length = 2)

remove the **.

learners take vectors of floats #Resolved

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also should trainers be used instead of learners?


In reply to: 273801982 [](ancestors = 273801982)

// e.g. var regressionTrainer = mlContext.Regression.Trainers.FastTree(labelColumn: "Label", featureColumn: "Features");
string outputColumnName = "Features";
var pipeline = mlContext.Transforms.Concatenate(outputColumnName, new[] { "Age", "Parity", "Induced" });
//
// Please note that the "Feature3" column is converted from int32 to float using the ConvertType API.
Copy link
Member

@singlis singlis Apr 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ConvertType Transformer? #Resolved

// The Concatenate API requires all columns to be of same type.
Copy link
Member

@singlis singlis Apr 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

API [](start = 31, length = 3)

The Concatenate Transformer? #Resolved

var pipeline = mlContext.Transforms.Conversion.ConvertType("Feature3", outputKind: DataKind.Single)
.Append(mlContext.Transforms.Concatenate("Features", new[] { "Feature1", "Feature2", "Feature3" }));

// The transformed data.
var transformedData = pipeline.Fit(trainData).Transform(trainData);
var transformedData = pipeline.Fit(dataview).Transform(dataview);

// Now let's take a look at what this concatenation did.
// We can extract the newly created column as an IEnumerable of SampleInfertDataWithFeatures, the class we define above.
var featuresColumn = mlContext.Data.CreateEnumerable<SampleInfertDataWithFeatures>(transformedData, reuseRowObject: false);
// We can extract the newly created column as an IEnumerable of TransformedData.
var featuresColumn = mlContext.Data.CreateEnumerable<TransformedData>(transformedData, reuseRowObject: false);

// And we can write out a few rows
Console.WriteLine($"{outputColumnName} column obtained post-transformation.");
Console.WriteLine($"Features column obtained post-transformation.");
foreach (var featureRow in featuresColumn)
{
foreach (var value in featureRow.Features.GetValues())
Console.Write($"{value} ");
Console.WriteLine("");
}
Console.WriteLine(string.Join(" ", featureRow.Features));

// Expected output:
// Features column obtained post-transformation.
//
// 26 6 1
// 42 1 1
// 39 6 2
// 34 4 2
// 35 3 1
// Features column obtained post-transformation.
// 0.1 1.1 2.1 3.1 1
// 0.2 1.2 2.2 3.2 2
// 0.3 1.3 2.3 3.3 3
// 0.4 1.4 2.4 3.4 4
// 0.5 1.5 2.5 3.5 5
// 0.6 1.6 2.6 3.6 6
}

private class InputData
{
public float Feature1;
[VectorType(3)]
public float[] Feature2;
Copy link
Contributor

@Ivanidzo4ka Ivanidzo4ka Apr 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would put VectorType(3) here.
Otherwise you concat scalar+VarVector+Scalar which would be VarVector and you can't use it for trainers.
Or maybe include that in sample, two concats, one on varvector, one on vector.... #Resolved

public int Feature3;
}

private class SampleInfertDataWithFeatures
private sealed class TransformedData
{
public VBuffer<float> Features { get; set; }
public float[] Features { get; set; }
}
}
}