Skip to content

Changed Ranker to Ranking in evaluation related files. #2675

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Feb 25, 2019
Merged

Changed Ranker to Ranking in evaluation related files. #2675

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ea78c3b
Change Ranker to Ranking in evaluation related files.
zeahmed Feb 21, 2019
4b1ee70
Addressed reviewers' comments.
zeahmed Feb 21, 2019
648fe92
Merge remote-tracking branch 'upstream/master' into ranker_to_ranking
zeahmed Feb 21, 2019
6a77f70
Merge remote-tracking branch 'upstream/master' into ranker_to_ranking
zeahmed Feb 21, 2019
7d4270d
Resolved merged conflicts.
zeahmed Feb 25, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/Microsoft.ML.Core/Prediction/ITrainer.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ namespace Microsoft.ML
[BestFriend]
internal delegate void SignatureMultiOutputRegressorTrainer();
[BestFriend]
internal delegate void SignatureRankerTrainer();
internal delegate void SignatureRankingTrainer();
Copy link
Contributor

@TomFinley TomFinley Feb 21, 2019 •
edited by zeahmed
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SignatureRankingTrainer [](start = 27, length = 23)

Since this is internal it is not a big deal, but note that this is incorrect. So you see how it is SignatureBinaryClassifierTrainer above? This is because the thing that does the task binary classification is a binary classifier. So this should have been retained as signature ranker trainer. What is the task? Ranking. What is the thing doing the task? A ranker.

Sort of like how things are being transformed, and the thing that does this is called a transformer. The use of different tenses, casing, and whatnot is deliberate.

Not a huge deal, but if you have to post another iteration anyway, this is what I would call a hypercorrection that should be backcorrected. #Resolved

[BestFriend]
internal delegate void SignatureAnomalyDetectorTrainer();
[BestFriend]
Expand Down
4 changes: 2 additions & 2 deletions src/Microsoft.ML.Data/EntryPoints/InputBuilder.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -849,8 +849,8 @@ public static class PipelineSweeperSupportedMetrics
public const string RSquared = RegressionLossEvaluatorBase<MultiOutputRegressionEvaluator.Aggregator>.RSquared;
public const string LogLoss = BinaryClassifierEvaluator.LogLoss;
public const string LogLossReduction = BinaryClassifierEvaluator.LogLossReduction;
public const string Ndcg = RankerEvaluator.Ndcg;
public const string Dcg = RankerEvaluator.Dcg;
public const string Ndcg = RankingEvaluator.Ndcg;
public const string Dcg = RankingEvaluator.Dcg;
public const string PositivePrecision = BinaryClassifierEvaluator.PosPrecName;
public const string PositiveRecall = BinaryClassifierEvaluator.PosRecallName;
public const string NegativePrecision = BinaryClassifierEvaluator.NegPrecName;
Expand Down
2 changes: 1 addition & 1 deletion src/Microsoft.ML.Data/Evaluators/EvaluatorUtils.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ public static Dictionary<string, Func<IHostEnvironment, IMamlEvaluator>> Instanc
{ MetadataUtils.Const.ScoreColumnKind.Regression, env => new RegressionMamlEvaluator(env, new RegressionMamlEvaluator.Arguments()) },
{ MetadataUtils.Const.ScoreColumnKind.MultiOutputRegression, env => new MultiOutputRegressionMamlEvaluator(env, new MultiOutputRegressionMamlEvaluator.Arguments()) },
{ MetadataUtils.Const.ScoreColumnKind.QuantileRegression, env => new QuantileRegressionMamlEvaluator(env, new QuantileRegressionMamlEvaluator.Arguments()) },
{ MetadataUtils.Const.ScoreColumnKind.Ranking, env => new RankerMamlEvaluator(env, new RankerMamlEvaluator.Arguments()) },
{ MetadataUtils.Const.ScoreColumnKind.Ranking, env => new RankingMamlEvaluator(env, new RankingMamlEvaluator.Arguments()) },
Copy link
Contributor

@TomFinley TomFinley Feb 21, 2019 •
edited by zeahmed
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RankingMamlEvaluator [](start = 86, length = 20)

Note that this renaming is correct. You are not evaluating a ranker, you are evaluating a ranking, so this one is good. #Resolved

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is also you note now more consistent with its "brethren" here.

In reply to: 258754361 [](ancestors = 258754361)

{ MetadataUtils.Const.ScoreColumnKind.Clustering, env => new ClusteringMamlEvaluator(env, new ClusteringMamlEvaluator.Arguments()) },
{ MetadataUtils.Const.ScoreColumnKind.AnomalyDetection, env => new AnomalyDetectionMamlEvaluator(env, new AnomalyDetectionMamlEvaluator.Arguments()) }
};
Expand Down
10 changes: 5 additions & 5 deletions ....Data/Evaluators/Metrics/RankerMetrics.cs → ...Data/Evaluators/Metrics/RankingMetrics.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

namespace Microsoft.ML.Data
{
public sealed class RankerMetrics
public sealed class RankingMetrics
{
/// <summary>
/// Array of normalized discounted cumulative gains where i-th element represent NDCG@i.
Expand All @@ -32,15 +32,15 @@ private static T Fetch<T>(IExceptionContext ectx, DataViewRow row, string name)
return val;
}

internal RankerMetrics(IExceptionContext ectx, DataViewRow overallResult)
internal RankingMetrics(IExceptionContext ectx, DataViewRow overallResult)
{
VBuffer<double> Fetch(string name) => Fetch<VBuffer<double>>(ectx, overallResult, name);

Dcg = Fetch(RankerEvaluator.Dcg).GetValues().ToArray();
Ndcg = Fetch(RankerEvaluator.Ndcg).GetValues().ToArray();
Dcg = Fetch(RankingEvaluator.Dcg).GetValues().ToArray();
Ndcg = Fetch(RankingEvaluator.Ndcg).GetValues().ToArray();
}

internal RankerMetrics(double[] dcg, double[] ndcg)
internal RankingMetrics(double[] dcg, double[] ndcg)
{
Dcg = new double[dcg.Length];
dcg.CopyTo(Dcg, 0);
Expand Down
80 changes: 40 additions & 40 deletions ...oft.ML.Data/Evaluators/RankerEvaluator.cs → ...ft.ML.Data/Evaluators/RankingEvaluator.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,19 +16,19 @@
using Microsoft.ML.Internal.Utilities;
using Microsoft.ML.Model;

[assembly: LoadableClass(typeof(RankerEvaluator), typeof(RankerEvaluator), typeof(RankerEvaluator.Arguments), typeof(SignatureEvaluator),
"Ranking Evaluator", RankerEvaluator.LoadName, "Ranking", "rank")]
[assembly: LoadableClass(typeof(RankingEvaluator), typeof(RankingEvaluator), typeof(RankingEvaluator.Arguments), typeof(SignatureEvaluator),
"Ranking Evaluator", RankingEvaluator.LoadName, "Ranking", "rank")]

[assembly: LoadableClass(typeof(RankerMamlEvaluator), typeof(RankerMamlEvaluator), typeof(RankerMamlEvaluator.Arguments), typeof(SignatureMamlEvaluator),
"Ranking Evaluator", RankerEvaluator.LoadName, "Ranking", "rank")]
[assembly: LoadableClass(typeof(RankingMamlEvaluator), typeof(RankingMamlEvaluator), typeof(RankingMamlEvaluator.Arguments), typeof(SignatureMamlEvaluator),
"Ranking Evaluator", RankingEvaluator.LoadName, "Ranking", "rank")]

[assembly: LoadableClass(typeof(RankerPerInstanceTransform), null, typeof(SignatureLoadDataTransform),
"", RankerPerInstanceTransform.LoaderSignature)]
[assembly: LoadableClass(typeof(RankingPerInstanceTransform), null, typeof(SignatureLoadDataTransform),
"", RankingPerInstanceTransform.LoaderSignature)]

namespace Microsoft.ML.Data
{
[BestFriend]
internal sealed class RankerEvaluator : EvaluatorBase<RankerEvaluator.Aggregator>
internal sealed class RankingEvaluator : EvaluatorBase<RankingEvaluator.Aggregator>
{
public sealed class Arguments
{
Expand Down Expand Up @@ -61,7 +61,7 @@ public sealed class Arguments
private readonly bool _groupSummary;
private readonly Double[] _labelGains;

public RankerEvaluator(IHostEnvironment env, Arguments args)
public RankingEvaluator(IHostEnvironment env, Arguments args)
: base(env, LoadName)
{
// REVIEW: What kind of checking should be applied to labelGains?
Expand Down Expand Up @@ -89,13 +89,13 @@ private protected override void CheckScoreAndLabelTypes(RoleMappedSchema schema)
var t = schema.Label.Value.Type;
if (t != NumberDataViewType.Single && !(t is KeyType))
{
throw Host.ExceptSchemaMismatch(nameof(RankerMamlEvaluator.Arguments.LabelColumn),
throw Host.ExceptSchemaMismatch(nameof(RankingMamlEvaluator.Arguments.LabelColumn),
"label", schema.Label.Value.Name, "R4 or a key", t.ToString());
}
var scoreCol = schema.GetUniqueColumn(MetadataUtils.Const.ScoreValueKind.Score);
if (scoreCol.Type != NumberDataViewType.Single)
{
throw Host.ExceptSchemaMismatch(nameof(RankerMamlEvaluator.Arguments.ScoreColumn),
throw Host.ExceptSchemaMismatch(nameof(RankingMamlEvaluator.Arguments.ScoreColumn),
"score", scoreCol.Name, "R4", t.ToString());
}
}
Expand All @@ -105,7 +105,7 @@ private protected override void CheckCustomColumnTypesCore(RoleMappedSchema sche
var t = schema.Group.Value.Type;
if (!(t is KeyType))
{
throw Host.ExceptSchemaMismatch(nameof(RankerMamlEvaluator.Arguments.GroupIdColumn),
throw Host.ExceptSchemaMismatch(nameof(RankingMamlEvaluator.Arguments.GroupIdColumn),
"group", schema.Group.Value.Name, "key", t.ToString());
}
}
Expand All @@ -129,7 +129,7 @@ internal override IDataTransform GetPerInstanceMetricsCore(RoleMappedData data)
var scoreInfo = data.Schema.GetUniqueColumn(MetadataUtils.Const.ScoreValueKind.Score);
Host.CheckParam(data.Schema.Group.HasValue, nameof(data), "Schema must contain a group column");

return new RankerPerInstanceTransform(Host, data.Data,
return new RankingPerInstanceTransform(Host, data.Data,
data.Schema.Label.Value.Name, scoreInfo.Name, data.Schema.Group.Value.Name, _truncationLevel, _labelGains);
}

Expand Down Expand Up @@ -242,7 +242,7 @@ private protected override void GetAggregatorConsolidationFuncs(Aggregator aggre
/// <param name="groupId">The name of the groupId column.</param>
/// <param name="score">The name of the predicted score column.</param>
/// <returns>The evaluation metrics for these outputs.</returns>
public RankerMetrics Evaluate(IDataView data, string label, string groupId, string score)
public RankingMetrics Evaluate(IDataView data, string label, string groupId, string score)
{
Host.CheckValue(data, nameof(data));
Host.CheckNonEmpty(label, nameof(label));
Expand All @@ -256,12 +256,12 @@ public RankerMetrics Evaluate(IDataView data, string label, string groupId, stri
Host.Assert(resultDict.ContainsKey(MetricKinds.OverallMetrics));
var overall = resultDict[MetricKinds.OverallMetrics];

RankerMetrics result;
RankingMetrics result;
using (var cursor = overall.GetRowCursorForAllColumns())
{
var moved = cursor.MoveNext();
Host.Assert(moved);
result = new RankerMetrics(Host, cursor);
result = new RankingMetrics(Host, cursor);
moved = cursor.MoveNext();
Host.Assert(!moved);
}
Expand Down Expand Up @@ -374,15 +374,15 @@ public void Update(short label, Single output)

public void UpdateGroup(Single weight)
{
RankerUtils.QueryMaxDcg(_labelGains, TruncationLevel, _queryLabels, _queryOutputs, _groupMaxDcgCur);
RankingUtils.QueryMaxDcg(_labelGains, TruncationLevel, _queryLabels, _queryOutputs, _groupMaxDcgCur);
if (_groupMaxDcg != null)
{
var maxDcg = new Double[TruncationLevel];
Array.Copy(_groupMaxDcgCur, maxDcg, TruncationLevel);
_groupMaxDcg.Add(maxDcg);
}

RankerUtils.QueryDcg(_labelGains, TruncationLevel, _queryLabels, _queryOutputs, _groupDcgCur);
RankingUtils.QueryDcg(_labelGains, TruncationLevel, _queryLabels, _queryOutputs, _groupDcgCur);
if (_groupDcg != null)
{
var groupDcg = new Double[TruncationLevel];
Expand Down Expand Up @@ -539,7 +539,7 @@ public void GetSlotNames(ref VBuffer<ReadOnlyMemory<char>> slotNames)
}
}

internal sealed class RankerPerInstanceTransform : IDataTransform
internal sealed class RankingPerInstanceTransform : IDataTransform
{
public const string LoaderSignature = "RankerPerInstTransform";
private const string RegistrationName = LoaderSignature;
Expand All @@ -552,7 +552,7 @@ private static VersionInfo GetVersionInfo()
verReadableCur: 0x00010001,
verWeCanReadBack: 0x00010001,
loaderSignature: LoaderSignature,
loaderAssemblyName: typeof(RankerPerInstanceTransform).Assembly.FullName);
loaderAssemblyName: typeof(RankingPerInstanceTransform).Assembly.FullName);
}

public const string Ndcg = "NDCG";
Expand All @@ -576,25 +576,25 @@ private static VersionInfo GetVersionInfo()
/// </summary>
public DataViewSchema OutputSchema => _transform.OutputSchema;

public RankerPerInstanceTransform(IHostEnvironment env, IDataView input, string labelCol, string scoreCol, string groupCol,
public RankingPerInstanceTransform(IHostEnvironment env, IDataView input, string labelCol, string scoreCol, string groupCol,
int truncationLevel, Double[] labelGains)
{
_transform = new Transform(env, input, labelCol, scoreCol, groupCol, truncationLevel, labelGains);
}

private RankerPerInstanceTransform(IHostEnvironment env, ModelLoadContext ctx, IDataView input)
private RankingPerInstanceTransform(IHostEnvironment env, ModelLoadContext ctx, IDataView input)
{
_transform = new Transform(env, ctx, input);
}

public static RankerPerInstanceTransform Create(IHostEnvironment env, ModelLoadContext ctx, IDataView input)
public static RankingPerInstanceTransform Create(IHostEnvironment env, ModelLoadContext ctx, IDataView input)
{
Contracts.CheckValue(env, nameof(env));
var h = env.Register(RegistrationName);
h.CheckValue(ctx, nameof(ctx));
ctx.CheckAtModel(GetVersionInfo());
h.CheckValue(input, nameof(input));
return h.Apply("Loading Model", ch => new RankerPerInstanceTransform(h, ctx, input));
return h.Apply("Loading Model", ch => new RankingPerInstanceTransform(h, ctx, input));
}

void ICanSaveModel.Save(ModelSaveContext ctx)
Expand Down Expand Up @@ -801,9 +801,9 @@ protected override void ProcessExample(RowCursorState state, short label, Single
protected override void UpdateState(RowCursorState state)
{
// Calculate the current group DCG, NDCG and MaxDcg.
RankerUtils.QueryMaxDcg(_labelGains, _truncationLevel, state.QueryLabels, state.QueryOutputs,
RankingUtils.QueryMaxDcg(_labelGains, _truncationLevel, state.QueryLabels, state.QueryOutputs,
state.MaxDcgCur);
RankerUtils.QueryDcg(_labelGains, _truncationLevel, state.QueryLabels, state.QueryOutputs, state.DcgCur);
RankingUtils.QueryDcg(_labelGains, _truncationLevel, state.QueryLabels, state.QueryOutputs, state.DcgCur);
for (int t = 0; t < _truncationLevel; t++)
{
Double ndcg = state.MaxDcgCur[t] > 0 ? state.DcgCur[t] / state.MaxDcgCur[t] * 100 : 0;
Expand Down Expand Up @@ -838,7 +838,7 @@ public RowCursorState(int truncationLevel)
}

[BestFriend]
internal sealed class RankerMamlEvaluator : MamlEvaluatorBase
internal sealed class RankingMamlEvaluator : MamlEvaluatorBase
{
public sealed class Arguments : ArgumentsBase
{
Expand All @@ -855,25 +855,25 @@ public sealed class Arguments : ArgumentsBase
public string GroupSummaryFilename;
}

private readonly RankerEvaluator _evaluator;
private readonly RankingEvaluator _evaluator;
private readonly string _groupIdCol;

private readonly string _groupSummaryFilename;

private protected override IEvaluator Evaluator => _evaluator;

public RankerMamlEvaluator(IHostEnvironment env, Arguments args)
public RankingMamlEvaluator(IHostEnvironment env, Arguments args)
: base(args, env, MetadataUtils.Const.ScoreColumnKind.Ranking, "RankerMamlEvaluator")
{
Host.CheckValue(args, nameof(args));
Utils.CheckOptionalUserDirectory(args.GroupSummaryFilename, nameof(args.GroupSummaryFilename));

var evalArgs = new RankerEvaluator.Arguments();
var evalArgs = new RankingEvaluator.Arguments();
evalArgs.DcgTruncationLevel = args.DcgTruncationLevel;
evalArgs.LabelGains = args.LabelGains;
evalArgs.OutputGroupSummary = !string.IsNullOrEmpty(args.GroupSummaryFilename);

_evaluator = new RankerEvaluator(Host, evalArgs);
_evaluator = new RankingEvaluator(Host, evalArgs);
_groupSummaryFilename = args.GroupSummaryFilename;
_groupIdCol = args.GroupIdColumn;
}
Expand Down Expand Up @@ -908,14 +908,14 @@ private bool TryGetGroupSummaryMetrics(Dictionary<string, IDataView>[] metrics,
Host.AssertNonEmpty(metrics);

if (metrics.Length == 1)
return metrics[0].TryGetValue(RankerEvaluator.GroupSummary, out gs);
return metrics[0].TryGetValue(RankingEvaluator.GroupSummary, out gs);

gs = null;
var gsList = new List<IDataView>();
for (int i = 0; i < metrics.Length; i++)
{
IDataView idv;
if (!metrics[i].TryGetValue(RankerEvaluator.GroupSummary, out idv))
if (!metrics[i].TryGetValue(RankingEvaluator.GroupSummary, out idv))
return false;

idv = EvaluateUtils.AddFoldIndex(Host, idv, i, metrics.Length);
Expand All @@ -939,13 +939,13 @@ private protected override IEnumerable<string> GetPerInstanceColumnsToSave(RoleM
yield return scoreCol.Name;

// Return the output columns.
yield return RankerPerInstanceTransform.Ndcg;
yield return RankerPerInstanceTransform.Dcg;
yield return RankerPerInstanceTransform.MaxDcg;
yield return RankingPerInstanceTransform.Ndcg;
yield return RankingPerInstanceTransform.Dcg;
yield return RankingPerInstanceTransform.MaxDcg;
}
}

internal static class RankerUtils
internal static class RankingUtils
{
private static volatile Double[] _discountMap;
public static Double[] DiscountMap
Expand Down Expand Up @@ -1054,8 +1054,8 @@ private static Comparison<int> GetCompareItems(List<short> queryLabels, List<Sin

internal static partial class Evaluate
{
[TlcModule.EntryPoint(Name = "Models.RankerEvaluator", Desc = "Evaluates a ranking scored dataset.")]
public static CommonOutputs.CommonEvaluateOutput Ranking(IHostEnvironment env, RankerMamlEvaluator.Arguments input)
[TlcModule.EntryPoint(Name = "Models.RankingEvaluator", Desc = "Evaluates a ranking scored dataset.")]
public static CommonOutputs.CommonEvaluateOutput Ranking(IHostEnvironment env, RankingMamlEvaluator.Arguments input)
{
Contracts.CheckValue(env, nameof(env));
var host = env.Register("EvaluateRanker");
Expand All @@ -1068,9 +1068,9 @@ public static CommonOutputs.CommonEvaluateOutput Ranking(IHostEnvironment env, R
MatchColumns(host, input, out label, out weight, out name);
var schema = input.Data.Schema;
string groupId = TrainUtils.MatchNameOrDefaultOrNull(host, schema,
nameof(RankerMamlEvaluator.Arguments.GroupIdColumn),
nameof(RankingMamlEvaluator.Arguments.GroupIdColumn),
input.GroupIdColumn, DefaultColumnNames.GroupId);
IMamlEvaluator evaluator = new RankerMamlEvaluator(host, input);
IMamlEvaluator evaluator = new RankingMamlEvaluator(host, input);
var data = new RoleMappedData(input.Data, label, null, groupId, weight, name);
var metrics = evaluator.Evaluate(data);

Expand Down
4 changes: 2 additions & 2 deletions src/Microsoft.ML.Data/TrainCatalog.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -623,14 +623,14 @@ internal RankingTrainers(RankingCatalog catalog)
/// <param name="groupId">The name of the groupId column in <paramref name="data"/>.</param>
/// <param name="score">The name of the score column in <paramref name="data"/>.</param>
/// <returns>The evaluation results for these calibrated outputs.</returns>
public RankerMetrics Evaluate(IDataView data, string label, string groupId, string score = DefaultColumnNames.Score)
public RankingMetrics Evaluate(IDataView data, string label, string groupId, string score = DefaultColumnNames.Score)
{
Environment.CheckValue(data, nameof(data));
Environment.CheckNonEmpty(label, nameof(label));
Environment.CheckNonEmpty(score, nameof(score));
Environment.CheckNonEmpty(groupId, nameof(groupId));

var eval = new RankerEvaluator(Environment, new RankerEvaluator.Arguments() { });
var eval = new RankingEvaluator(Environment, new RankingEvaluator.Arguments() { });
return eval.Evaluate(data, label, groupId, score);
}
}
Expand Down
Loading