ML.NET rolling and fixed version CI legs #988

adamsitnik · 2019-10-31T10:34:02Z

In #896 we have agreed that it would be good to have two different configs for ML.NET:

one that uses latest .NET Core and fixed ML.NET to detect regressions in .NET Core
one that uses latest version of ML.NET and LTS .NET Core to detect regressions in ML.NET

This PR tries to implement that by doing the following:

extending the benchmarks .yml template with an optional libraryVersion parameter
defining two jobs for ML.NET:
- fixed ML.NET, latest .NET Core
- latest .NET Core, fixed ML.NET
extending the python script used by CI to set env var for controlling the version

I could not read latest ML.NET version from Maestro because ML.NET is not using Arcade, so I've used an asterisk to obtain latest preview version.

<LibraryVersion Condition="'$(PERFLAB_LIBRARY_VERSION)' == 'rolling'">1.5.0-*</LibraryVersion>

This solution is simple, but requires a version bump after ML.NET increases major|minor version number (few times a year) due to NuGet/Home#912

…er time (and make it a correct name)

# Conflicts: # azure-pipelines.yml # src/benchmarks/real-world/Microsoft.ML.Benchmarks/HashBench.cs # src/benchmarks/real-world/Microsoft.ML.Benchmarks/Microsoft.ML.Benchmarks.csproj

adamsitnik · 2019-11-05T11:47:07Z

@DrewScoggins could you please take a look?

adamsitnik · 2019-11-05T11:50:49Z

eng/performance/benchmark_jobs.yml

@@ -92,9 +93,9 @@ jobs:
            ${{ each framework in parameters.frameworks }}:
              ${{ framework }}:
                _Framework: ${{ framework }}
-                _Configs: CompilationMode=Tiered RunKind="${{ parameters.kind }}"
+                _Configs: CompilationMode=Tiered RunKind="${{ parameters.kind }}${{ parameters.libraryVersion }}"


I don't know how the new uploader works so I've just extended the RunKind string with fixed|rolling This parameter is optional and null by default, so the "id" should change only for ML.NET runs

This seems like the right way to go about this.

eerhardt · 2019-11-05T15:18:46Z

scripts/benchmarks_ci.py

@@ -213,6 +220,10 @@ def __main(args: list) -> int:
        target_framework_monikers
    )

+    # set the product version using Env Vars handled by the MSBuild project file
+    if args.lib_version:
+        os.environ['PERFLAB_LIBRARY_VERSION'] = args.lib_version


Can we instead pass it in through /p on the command line?

BDN also performs dotnet restore, I would have to extend BDN to be able to parse /p parameters. With env vars it's simpler - we just set ugly env var and the child processes get the same settings out of the box.

DrewScoggins

So if I understand what is going on here we still use core-sdk as the catalyst for when runs are triggered even on runs where .NET Core is not moving. So we will not really have anyway to link back to what version of ML.NET that was running. Maybe we can add the version of ML.NET that we are testing as a config flag for the run? Otherwise I wonder how actionable this data will be.

kunalspathak · 2021-12-01T00:34:43Z

@DrewScoggins - after our today's discussion, should we re-evaluate this?

Also - Is it possible to update the ML.NEt benchmarks to the latest https://github.com/dotnet/machinelearning/tree/main/test/Microsoft.ML.PerformanceTests?

danmoseley · 2021-12-01T16:47:09Z

cc @michaelgsharp

michaelgsharp · 2021-12-01T17:30:19Z

I am happy to update to the latest and run them. Is there anything other than that I need to do? I'm not super familiar with this issue.

kunalspathak · 2021-12-01T17:49:38Z

@adamsitnik can chime in, but I want to make sure that we remove all the project references , add back nuget references and then make sure that all the benchmarks run. If you can get to that, I can verify if they run on windows/arm64 as well.

adamsitnik · 2021-12-02T13:28:33Z

Existing ML.NET benchmarks that we have in this repo are using a hardcoded version of ML.NET in order to catch regressions in .NET itself.

The purpose of this PR was to add another CI leg that would be using latest ML.NET version to detect regressions in ML.NET itself.

@kunalspathak I assume you would be interested in updating the benchmarks, ML.NET version and using the benchmarks as part of our automated regression detection suite?

kunalspathak · 2021-12-02T15:48:06Z

@kunalspathak I assume you would be interested in updating the benchmarks, ML.NET version and using the benchmarks as part of our automated regression detection suite?

That's right. It is part of our ongoing goal of having real-world benchmarks to measure the .NET performance. I also am working on adding ImageSharp to this suite in #2149. So help me understand, with the current state, if we just copy the latest code from https://github.com/dotnet/machinelearning/tree/main/test/Microsoft.ML.PerformanceTests and used nuget references to the latest ML.NET, we should be good and this PR is not blocking it, yes?

Of course, we certainly want to track ML.NET performance and in order to do that, we need this PR.

adamsitnik · 2021-12-02T16:45:54Z

we should be good and this PR is not blocking it, yes?

Yes, that is correct.

adamsitnik added 10 commits October 30, 2019 13:18

update ML.NET benchmarks to the latest version

dc5c8a0

use public APIs only

da0c319

use dirty reflection to get it working...

8cd2420

define two ML.NET CI legs: fixed and rolling ML.NET versions

270923a

CI leg must have unique ID

1c3984c

CI leg must have unique ID...

f5bddfc

use fixed|rolling instead of version numbers to keep the CI leg ID ov…

11ebf05

…er time (and make it a correct name)

Merge remote-tracking branch 'upstream/master' into mlDotNetRolling

0a853d9

# Conflicts: # azure-pipelines.yml # src/benchmarks/real-world/Microsoft.ML.Benchmarks/HashBench.cs # src/benchmarks/real-world/Microsoft.ML.Benchmarks/Microsoft.ML.Benchmarks.csproj

fix yml file after the recent merge conflict

cba0daf

add ML.NET CI feed so we can test the daily builds

0943115

adamsitnik changed the title ~~[WIP] Ml.NET rolling and fixed version CI legs~~ [WIP] ML.NET rolling and fixed version CI legs Nov 4, 2019

adamsitnik mentioned this pull request Nov 4, 2019

update the CI feed url dotnet/machinelearning#4435

Merged

adamsitnik added 4 commits November 4, 2019 00:15

bump the ML.NET versions to 1.4 and 1.5

369eca4

make sure a preview version can be restored properly

46ecb70

update the private CI legs

3b86165

the reporting system part?

b57d9dc

adamsitnik changed the title ~~[WIP] ML.NET rolling and fixed version CI legs~~ ML.NET rolling and fixed version CI legs Nov 5, 2019

adamsitnik requested a review from DrewScoggins November 5, 2019 11:46

adamsitnik marked this pull request as ready for review November 5, 2019 11:47

adamsitnik commented Nov 5, 2019

View reviewed changes

eerhardt reviewed Nov 5, 2019

View reviewed changes

DrewScoggins reviewed Nov 5, 2019

View reviewed changes

adamsitnik mentioned this pull request Nov 7, 2019

Update ML.NET to 1.4 #987

Merged

adamsitnik closed this Apr 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ML.NET rolling and fixed version CI legs #988

ML.NET rolling and fixed version CI legs #988

adamsitnik commented Oct 31, 2019 •

edited

Loading

adamsitnik commented Nov 5, 2019

adamsitnik Nov 5, 2019

DrewScoggins Nov 5, 2019

eerhardt Nov 5, 2019

adamsitnik Nov 5, 2019

DrewScoggins left a comment

kunalspathak commented Dec 1, 2021

danmoseley commented Dec 1, 2021

michaelgsharp commented Dec 1, 2021

kunalspathak commented Dec 1, 2021

adamsitnik commented Dec 2, 2021

kunalspathak commented Dec 2, 2021 •

edited

Loading

adamsitnik commented Dec 2, 2021

ML.NET rolling and fixed version CI legs #988

ML.NET rolling and fixed version CI legs #988

Conversation

adamsitnik commented Oct 31, 2019 • edited Loading

adamsitnik commented Nov 5, 2019

adamsitnik Nov 5, 2019

Choose a reason for hiding this comment

DrewScoggins Nov 5, 2019

Choose a reason for hiding this comment

eerhardt Nov 5, 2019

Choose a reason for hiding this comment

adamsitnik Nov 5, 2019

Choose a reason for hiding this comment

DrewScoggins left a comment

Choose a reason for hiding this comment

kunalspathak commented Dec 1, 2021

danmoseley commented Dec 1, 2021

michaelgsharp commented Dec 1, 2021

kunalspathak commented Dec 1, 2021

adamsitnik commented Dec 2, 2021

kunalspathak commented Dec 2, 2021 • edited Loading

adamsitnik commented Dec 2, 2021

adamsitnik commented Oct 31, 2019 •

edited

Loading

kunalspathak commented Dec 2, 2021 •

edited

Loading