Parallel.ForEach is actually slow

[superichmann]

I have a process which I need to run very fast. when running with parallel.foreach (MaxDegreeOfParallelism = 8) (my cpu is 16) it takes 2 minutes. when changing MaxDegreeOfParallelism to 16 it takes more than 2 minutes. the cpu is not actually sweating (40%)

When splitting the data and running it from two separate EXE processes, it takes 1 minute for both (half time, same data, cpu is 90%).
Anyone knows why? or how can I fix it without running multiple processes? thanks :)

            Parallel.ForEach(sfs, new ParallelOptions { MaxDegreeOfParallelism = 8 }, sf =>
            {var success= PerformFromDatabase(sfString);}

Tagging subscribers to this area: @dotnet/area-system-linq-parallel
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I have a process which I need to run very fast. when running with parallel.foreach (MaxDegreeOfParallelism = 8) (my cpu is 16) it takes 2 minutes. when changing MaxDegreeOfParallelism to 16 it takes more than 2 minutes. the cpu is not actually sweating (40%)

When splitting the data and running it from two separate EXE processes, it takes 1 minute for both (half time, same data, cpu is 90%).
Anyone knows why? or how can I fix it without running multiple processes? thanks :)

            Parallel.ForEach(sfs, new ParallelOptions { MaxDegreeOfParallelism = 8 }, sf =>
            {var success= PerformFromDatabase(sfString);}

Author:	superichmann
Assignees:	-
Labels:	area-System.Linq.Parallel, tenet-performance, untriaged
Milestone:	-

[svick]

That depends entirely on what the body of the loop does. Without understanding that, we could only guess.

[superichmann]

Hi @svick :]

The code inside the loop is performing about 4000 SELECTs to a database on the same machine. on each SELECT it is building an ML model (using ml.net DatabaseLoader).

the sf value goes to the SELECT statement WHERE clause, I am running on ~2000 sf values. so 2000 * 4000...

I have no locks inside or stuff that should stop the thread from running.

[Clockwork-Muse]

That's not sufficient information.

to a database on the same machine

So, first, this is going to play havoc with any performance testing, because your application is going to be fighting with the database for resources.

the sf value goes to the SELECT statement WHERE clause, I am running on ~2000 sf values. so 2000 * 4000...

What this sounds like is you may have something that should be run as a single statement (ie, no loop at all on the application side), letting the database handle any potential mutlithreading aspects (which in most cases it's going to be much better at). Otherwise you're just setting yourself for RBAR (row-by-agonizing-row). We'd need to see the statements for that, though.

I have no locks inside or stuff that should stop the thread from running.

You may have written no locks. That doesn't mean there aren't locks in other parts of the application, or that other things wouldn't be the cause. We'd need actual code to diagnose this.

[stephentoub]

I suggest profiling. That should highlight the issue. Most likely there is a shared resource protected by some kind of synchronization and that is bottlenecking the operation. By splitting into two processes, as a side-effect you'd also be doubling the resources (eg a database connection that's inherently serialized), thereby doubling throughput.

[omariom]

Have a look at Concurrency Visualizer.

[superichmann]

thanks @Clockwork-Muse @stephentoub @omariom @svick
I have tried playing with the concurrency visualizer but no success :\

Here is the simplified console application code:
generating selects and performing machine learning on them with ml.net.

using Microsoft.ML;
using Microsoft.ML.AutoML;
using Microsoft.ML.Data;
using Microsoft.ML.Runtime;
using Npgsql;
using System.Collections.Concurrent;
using System.Data;

var watch = new System.Diagnostics.Stopwatch();
watch.Start();
int nAmountOfRuns = 256;
string strSelect = "SELECT rnd_double,rnd_double from long_sequence(2000)";
string connectionString = $@"host=127.0.0.1;port=8812;username=admin;password=quest;database=qdb;ServerCompatibilityMode=NoTypeLoading;Pooling=true;MaxPoolSize=99999;CommandTimeout=99999;";
MLContext mlXXX = new MLContext(666);
DatabaseLoader.Column[] columns = new DatabaseLoader.Column[2];
columns[0] = new DatabaseLoader.Column("rnd_double", DbType.Double, 0);
columns[1] = new DatabaseLoader.Column("rnd_double1", DbType.Double, 1);
ConcurrentBag<(int Key, double Value)> hScores = new ConcurrentBag<(int, double)>();
Parallel.For(0, nAmountOfRuns, new ParallelOptions { MaxDegreeOfParallelism = 16 }, (i) =>
{
    IDataView currData = LoadIntoIDataView(columns);
    var splitted = mlXXX.Data.TrainTestSplit(currData, 0.01);
    var experimentSettings = new RegressionExperimentSettings();
    experimentSettings.MaxModels = 1;
    experimentSettings.OptimizingMetric = RegressionMetric.MeanAbsoluteError;
    RegressionExperiment experiment = mlXXX.Auto().CreateRegressionExperiment(experimentSettings);
    var preDoubleToSingle = TransformDoubleToSingle(splitted.TrainSet);
    ExperimentResult<RegressionMetrics>? experimentResult = null; 
    RegressionMetrics? metrics = null;
    try
    {
        experimentResult = experiment.Execute(splitted.TrainSet, splitted.TestSet, "rnd_double1", preFeaturizer: preDoubleToSingle);
        metrics = experimentResult.BestRun.ValidationMetrics;
        Console.WriteLine($"Total models produced: {experimentResult.RunDetails.Count()}");
        Console.WriteLine($"{experimentResult.BestRun.TrainerName}\t Score:\t{experimentResult.BestRun.ValidationMetrics.MeanAbsoluteError}");
    }
    catch (Exception ex)
    {
        Console.WriteLine("Experiment is dead");
        Console.WriteLine(ex.ToString());
    }
    hScores.Add((i, experimentResult.BestRun.ValidationMetrics.MeanAbsoluteError));
    //Console.WriteLine(i);
});

var sorted = hScores.OrderBy(entry => entry.Value).Take(5).ToList();
Console.WriteLine("Best runs:");
foreach (var g in sorted)
{
    Console.WriteLine(g.Key);
}
watch.Stop();

Console.WriteLine($"Execution Time: {watch.Elapsed.TotalSeconds}");

IDataView LoadIntoIDataView(DatabaseLoader.Column[] columns)
{
    IDataView idvContents;
    DatabaseLoader loader = mlXXX.Data.CreateDatabaseLoader(columns);
    DatabaseSource dbSource = new DatabaseSource(NpgsqlFactory.Instance, connectionString, strSelect, 201);
    idvContents = loader.Load(dbSource);
    return idvContents;
}

IEstimator<ITransformer> TransformDoubleToSingle(IDataView data)
{
    var mlContext = new MLContext();

    var doubleColumns = data.Schema
        .Where(col => col.Type == NumberDataViewType.Double)
        .Select(col => new InputOutputColumnPair($"{col.Name}", col.Name));
    if (doubleColumns.Any())
        return mlContext.Transforms.Conversion.ConvertType(doubleColumns.ToArray(), DataKind.Single);
    else
        return null;
}

If you would like to actually reproduce and take a look:

• this is the database just download and run it
• Microsoft.ML and Microsoft.Auto.ML NuGet are from dotnet-libraries, stock version will not work.
• npgsql NuGet is prerelease version
• .Net 7

On my machine, running on MaxDegreeOfParallelism = 1 takes 14.9451178 seconds, running on MaxDegreeOfParallelism = 16 takes 21.1512313 seconds. and running on MaxDegreeOfParallelism = 8 takes 15.6480345 seconds.

running two instances of the same program EXE (splitting the data in two - so 128 and 128) takes 5 seconds.

@LittleLittleCloud maybe you can take a look? since its ml related.. how is DatabaseLoader acting? it might lock some operations?

[Clockwork-Muse]

Console.WriteLine

Access to the console has to be serialized, there's a lock there. A fairly long-lived one, as output here can be relatively slow. Don't do that.

I don't know enough about the ML library, but what you want to look for is some way to create a batch of experiments, not a bunch of single experiments.

string strSelect = "SELECT rnd_double,rnd_double from long_sequence(2000)";

This doesn't seem to be a thing in standard PostgreSQL, but from reading it and the rest of the code the implication is that you're getting a duplicated single column of 2000 rows each time you start a new experiment. This has problems:

1. You're making a round trip to the database just to get this data, when it would be way faster to do it in-process. (Same machine is somewhat irrelevant - you're still hitting the network stack and incurring serialization costs).
2. If those aren't actually random numbers, you're using the same data over and over again....
3. (Potentially - depends on how ML.NET handles this) The conversion single->double is possibly happening on the application side, instead of the database side.

It's also not clear to me whether the experiments need a "live" view of the data from the database, or whether you could capture it locally, or what.

var watch = new System.Diagnostics.Stopwatch();

1. Your capture is too wide. For example, if the majority of the time was spent in sorting the bag at the end, you wouldn't be able to tell. You're not testing the performance of just the parallel parts.
2. The library we generally use for performance testing is BenchmarkDotNet, not a raw Stopwatch (specifically because it handles certain trickier aspects better than doing it manually).

[superichmann]

Thanks @Clockwork-Muse with your help I think I can solve this!

• I have removed all console writes.
• The values I extract are different on each SELECT. I could make one huge select and filter everything in IDataView but I don't know how to do it :\
• Conversion Double->Single is due to the nature of ml.net and my SELECTs, moving this functionality to the database will be hard as well. Update I have moved the conversion o the db it and it is not improving performance

BenchMarkDotNet:

• Before even starting, when running the program with dotnet run -c Release as specified in the the BenchMarkDotNet page, execution time dropped in half 💯 I am not sure why. even in comparison to when running from release.
• BenchMarkDotNet just running in loop over the entire code and never ending :\ WorkloadActual 63: 1 op, 1613725600.00 ns, 1.6137 s/op each time with larger amount of time it takes to run each execution.
• ok so it finished running. what should I actually do now?

BenchmarkDotNet=v0.13.5, OS=Windows 10 (10.0.19045.2846/22H2/2022Update)
Intel Core i9-9880H CPU 2.30GHz, 1 CPU, 16 logical and 8 physical cores
.NET SDK=7.0.203
  [Host]     : .NET 7.0.5 (7.0.523.17405), X64 RyuJIT AVX2
  DefaultJob : .NET 7.0.5 (7.0.523.17405), X64 RyuJIT AVX2

Method	Mean	Error	StdDev
RunAllStuff	333.7 ms	7.88 ms	23.23 ms

I have started implementing some changes and it improves performance:

• Moving some settings to be created once at the beginning of the class and not dynamically before ml
• removing all console write lines
• removing unnecessary ml models from the experiment.

After all of this I still don't know why when changing to maximum threads (16) it gets slower but when running from external EXEs at the same time it finished much faster

[superichmann]

My guess is that AutoML class is static and have some capping on the threads when running multiple experiments, @LittleLittleCloud ?
I have as well tried to load the entire table into an IDataView and use FilterRowsByColumn and DropColumns in combination with Data.Cache but ml.net not really caches :\

var cachedTRAIN = mlx.Data.Cache(idvtrain);
var storeTRAIN = mlx.Data.FilterRowsByColumn(cachedTRAIN, "col", 1,2);
... experiment.Execute // caching not working and AutoML goes to database again :\

As well when loading the entire table into IDataView (executing an experiment without filter, it takes really long time to load everything)

[adamsitnik]

I have tried playing with the concurrency visualizer but no success :\

Could you please share the trace file?