LightGBM Unbalanced Data Argument [Issue #3688 Fix]

[rayankrish]

Fix for issue #3688. LightGBM Multiclass Trainer can now accept unbalanced data parameter as was previously possible in the Binary Trainer. An additional argument was added to the LightGBMBinaryEstimator test.

[dnfclas]

All CLA requirements met.

[wschin]

We need two tests for UnbalancedSets=true and UnbalancedSets=false. In addition to simply run a model, we need more restrictive comparison like

        [LightGBMFact]
        public void LightGbmMulticlassEstimatorCompareOva()
        {
            float sigmoidScale = 0.5f; // Constant used train LightGBM. See gbmParams["sigmoid"] in the helper function.

            // Train ML.NET LightGBM and native LightGBM and apply the trained models to the training set.
            LightGbmHelper(useSoftmax: false, sigmoid: sigmoidScale, out string modelString, out List<GbmExample> mlnetPredictions, out double[] nativeResult1, out double[] nativeResult0);

            // The i-th predictor returned by LightGBM produces the raw score, denoted by z_i, of the i-th class.
            // Assume that we have n classes in total. The i-th class probability can be computed via
            // p_i = sigmoid(sigmoidScale * z_i) / (sigmoid(sigmoidScale * z_1) + ... + sigmoid(sigmoidScale * z_n)).
            Assert.True(modelString != null);
            // Compare native LightGBM's and ML.NET's LightGBM results example by example
            for (int i = 0; i < _rowNumber; ++i)
            {
                double sum = 0;
                for (int j = 0; j < _classNumber; ++j)
                {
                    Assert.Equal(nativeResult0[j + i * _classNumber], mlnetPredictions[i].Score[j], 6);
                    if (float.IsNaN((float)nativeResult1[j + i * _classNumber]))
                        continue;
                    sum += MathUtils.SigmoidSlow(sigmoidScale * (float)nativeResult1[j + i * _classNumber]);
                }
                for (int j = 0; j < _classNumber; ++j)
                {
                    double prob = MathUtils.SigmoidSlow(sigmoidScale * (float)nativeResult1[j + i * _classNumber]);
                    Assert.Equal(prob / sum, mlnetPredictions[i].Score[j], 6);
                }
            }

            Done();
        }

in machinelearning2\test\Microsoft.ML.Tests\TrainerEstimators\TreeEstimators.cs. Note that this test directly checks if ML.NET's LightGBM module and LightGBM's C API produce identical numbers.

[rayankrish]

Thank you. I will add and run these tests

[rayankrish]

Needed to add the unbalanced argument to LightGbmMulticlassTrainer. Ran a test to revise manifest file

Tests Added:

• Unbalanced binary trainer test
• Two tests for unbalanced = true and unbalanced = false for multiclass
• Compare multiclass .NET LightGBM output with multiclass LightGBM C output test

[eerhardt]

Use for binary classification when training data is not balanced.

It's odd to say "use for binary classification" when this is the multiclass trainer.

[rayankrish]

Is it more helpful to just say "Use for multi-class classification when training data is not balanced."

[eerhardt]

Use for binary classification when training data is not balanced.

Is putting this option on the OptionsBase class correct if it is only for binary classification?

[rayankrish]

You're right. The cause of the problem was this commit which split the trainers in to their classes (binary and multiclass). is it better not to include it as an argument in this file?

[eerhardt]

Where is this boolean being used? I don't see any code added using it.

[rayankrish]

It's added to the dictionary, NameMapping on line 39:
{nameof(OptionsBase.UnbalancedSets), "is_unbalance"},

[eerhardt]

You are declaring 2 new boolean fields. I only see 1 being used.

[codemzs]

res[GetOptionName(nameof(UnbalancedSets))] = UnbalancedSets; [](start = 16, length = 60)

how is getting mapped to is_unbalance 🔗︎, default = false, type = bool, aliases: unbalance, unbalanced_sets #Resolved

[codemzs]

nevermind I see it now:

public static string GetOptionName(string name)
{
// Otherwise convert the name to the light gbm argument
StringBuilder strBuf = new StringBuilder();
bool first = true;
foreach (char c in name)
{
if (char.IsUpper(c))
{
if (first)
first = false;
else
strBuf.Append('_');
strBuf.Append(char.ToLower(c));
}
else
strBuf.Append(c);
}
return strBuf.ToString();
}

---

In reply to: 299023864 [](ancestors = 299023864)

[codemzs]

[wschin]

Empty line.