ShuffleRows is broken (again) in 1.5.2

[ogo-adp]

System information

• OS version/distro: Windows Server 2016
• .NET Version (eg., dotnet --info): .NET 4.8

Issue

My training tasks times exploded when going from 1.4 to 1.5.2 (from seconds to tens of hours). The culprit is ShuffleRows which is now extremely slow.

ShuffleRows was already broken in previous release (see issue #5312). The fix for that issue (#5313) is broken too in my opinion. It contains a line with a Thread.Sleep(1) to wait for async completion. This is a no go ... First the Sleep should not be there. And second, sleeping 1ms is dependent on timer resolution which is in general 15ms, so a wait will be in most cases 15ms. As my learning tasks read millions of datarows, this can not work and learning time explode to the point of being unusable.

To confirm that Thread.Sleep(1) is the culprit, I changed the timer resolution to 1ms on my server and learning time improved greatly but are still very far from the times I got with version 1.4. The fix for #5312 needs to be redone properly (sorry for being a bit harsh).

So ShuffleRows needs a fix, as I suspect this bug will impact many users. I'm not an async specialist so I can't fix the code myself.

[eerhardt]

@ogo-adp - thanks for the report. Would it be possible to list your training code in order to reproduce the issue?

[ogo-adp]

Any dataset will reproduce the bug as long as it has sufficient count of rows (let's say a few hundreds of thousands). Any call to ShuffleRows will dramatically decrease training speed.

Here is a sample you can play with (it does generate a dummy training dataset with 200000 rows).

Without shuffling it needs 2.4s to run. With shuffling it needs 48s.

using Microsoft.ML;
using Microsoft.ML.Data;
using Microsoft.ML.Trainers.FastTree;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;

class SampleModel
{
class TrainRow
{
[ColumnName("Sample")]
public string Sample;

    [ColumnName("Week")]
    public float Week;

    [ColumnName("Label")]
    public float Label;
}

private static IEnumerable<TrainRow> ReadAsEnumerable()
{
    for (var i = 0; i < 200_000; i++)
        yield return new TrainRow() { Sample = i.ToString(), Week = i, Label = i / 2 };
}

public static void Train()
{
    Console.WriteLine("Training ...");

    var st = Stopwatch.StartNew();
    var mlContext = new MLContext(seed: 0);
    var fullData = mlContext.Data.LoadFromEnumerable<TrainRow>(ReadAsEnumerable());

    // comment this line to see time without ShuffleRows
    fullData = mlContext.Data.ShuffleRows(fullData, seed: 0);

    var trainTestData = mlContext.Data.TrainTestSplit(fullData, testFraction: 0.05, seed: 0);
    var trainingDataView = trainTestData.TrainSet;
    var testingDataView = trainTestData.TestSet;

    var options = new FastTreeRegressionTrainer.Options()
    {
        LabelColumnName = "Label",
    };

    var pipeline = mlContext.Transforms.Categorical.OneHotEncoding("Sample")
        .Append(mlContext.Transforms.Concatenate("Features"
        , "Sample"
        , "Week"))
        .Append(mlContext.Regression.Trainers.FastTree(options));

    var model = pipeline.Fit(trainingDataView);

    Console.WriteLine("Done ms=" + st.ElapsedMilliseconds);
}

}

[ogo-adp]

Same code with version 1.5.0 run in 2.4s (non shuffled) and 2.6s (shuffled)

[frank-dong-ms-zz]

Thanks @ogo-adp this looks problematic to me, I can repro this issue, we will start to work on a fix and update in this thread if any progress, thanks.

[jwood803]

Thanks @frank-dong-ms! If I can help at all just let me know.

@ogo-adp Sorry you've seen this and the previous issue.

[ogo-adp]

Hi,

Thanks for looking at this 👍

[ogo-adp]

Hi to the team and thanks for the fix :)