Enable the running of performance tests: SDK:release/2.0.0 by johnbeisner · Pull Request #1969 · dotnet/sdk

johnbeisner · 2018-02-16T21:57:37Z

Enable the running of performance tests:

commandLine: "build.cmd -perf"
optional properties: 'PerfIterations' 'PerfOutputDirectory' 'PerfCollectionType'
update RepoToolset version to a version that contains the performance targets
first draft 'perf.groovy'
rename: "Microsoft.NET.Perf.Tests" to "Microsoft.NET.PerformanceTests"

…properties: 'PerfIterations' 'PerfOutputDirectory' 'PerfCollectionType'; update RepoToolset version to a version that contains the performance targets; first draft 'perf.groovy'.

johnbeisner · 2018-02-16T22:01:54Z

@dsplaisted

DrewScoggins · 2018-02-20T17:34:48Z

perf.groovy

+    ['Windows_NT'].each { os ->
+        ['x64', 'x86'].each { arch ->
+            def architecture = arch
+            def jobName = "perf_perflab_${os}_${arch}"


Probably want to update the name of the job to better reflect the SDK. Something like SDK_Perf_${os}_${arch}

johnbeisner · 2018-02-20T17:38:43Z

@DrewScoggins
@tannergooding
Can I get a code review on this please.

DrewScoggins · 2018-02-20T17:42:15Z

perf.groovy

+                    batchFile("if \"%GIT_BRANCH:~0,7%\" == \"origin/\" (set \"GIT_BRANCH_WITHOUT_ORIGIN=%GIT_BRANCH:origin/=%\") else (set \"GIT_BRANCH_WITHOUT_ORIGIN=%GIT_BRANCH%\")\n" +
+                    "set \"BENCHVIEWNAME=${benchViewName}\"\n" +
+                    "set \"BENCHVIEWNAME=%BENCHVIEWNAME:\"=\"\"%\"\n" +
+                    "py \"${perfWorkingDirectory}\\Microsoft.BenchView.JSONFormat\\tools\\submission-metadata.py\" --name \"%BENCHVIEWNAME%\" --user-email \"dotnet-bot@microsoft.com\" --output \"${perfWorkingDirectory}\"\n" +


I am fairly certain that when you use the output argument you need to also specify the file name, not just the out directory.

DrewScoggins · 2018-02-20T17:42:46Z

perf.groovy

+				    batchFile("\"%WORKSPACE%\\build.cmd\" -perf /p:PerfIterations=10 /p:PerfOutputDirectory=\"${perfWorkingDirectory}\" /p:PerfCollectionType=stopwatch")
+
+			       //Create submission json and upload to Benchview
+				   batchFile("for /f \"tokens=*\" %%a in ('dir /b/a-d ${perfWorkingDirectory}\\*.xml') do (py \"${perfWorkingDirectory}\\Microsoft.BenchView.JSONFormat\\tools\\measurement.py xunitscenario \"${perfWorkingDirectory}\\%%a\" --better desc --output \"${perfWorkingDirectory}\")")


Make sure that you add --append or you will overwrite the measurements

DrewScoggins · 2018-02-20T17:43:14Z

perf.groovy

+                   "--architecture \"${arch}\"" +
+                   "--machinepool \"perfsnake\"" +
+				   "--output \"${perfWorkingDirectory}\"")
+                   batchFile("py \"${perfWorkingDirectory}\\Microsoft.BenchView.JSONFormat\\tools\\upload.py \"${perfWorkingDirectory}\submission.json\" --container stability")


The container name should be coreclr

@DrewScoggins
Why is the container: "coreclr"?

DrewScoggins · 2018-02-20T17:44:10Z

perf.groovy

+
+                builder.triggerOnlyOnComment()
+			    //Phrase is "test Windows_NT x64 perf"
+                builder.setCustomTriggerPhrase("(?i).*test\\W+${os}\\W+${arch}\\W+perf.*")


This should include something about SDK in the trigger phrase

dsplaisted · 2018-02-20T19:20:31Z

build/build.ps1

    Write-Host "  -rebuild                Rebuild solution"
    Write-Host "  -deploy                 Deploy built VSIXes"
    Write-Host "  -test                   Run all unit tests in the solution"
+	Write-Host "  -perf                   Run all performance tests in the solution"


Use spaces instead of tabs for indentation

dsplaisted · 2018-02-20T19:21:20Z

build/build.sh

      echo "  --build                  Build solution"
      echo "  --rebuild                Rebuild solution"
      echo "  --test                   Run all unit tests in the solution"
+	  echo "  --perf                   Run all performance tests in the solution"


Use spaces instead of tabs

dsplaisted · 2018-02-20T19:21:54Z

build/build.sh

      shift 1
+      ;;
+	--Perf)
+      pack=true


Shouldn't this be perf=true? Looks like a copy/paste error.

Yup, done...

dsplaisted · 2018-02-20T19:24:43Z

perf.groovy

+                    "py \"${perfWorkingDirectory}\\Microsoft.BenchView.JSONFormat\\tools\\build.py\" git --branch %GIT_BRANCH_WITHOUT_ORIGIN% --type ${runType} --output \"${perfWorkingDirectory}\\build.json\"")
+                    batchFile("py \"%WORKSPACE%\\Microsoft.BenchView.JSONFormat\\tools\\machinedata.py\" --output \"${perfWorkingDirectory}\\machinedata.json\"")
+
+			        // Build solution and run the performance tests


Use spaces instead of tabs

* Update dependencies from https://github.com/dotnet/runtime build 20231017.5 Microsoft.Extensions.FileSystemGlobbing , Microsoft.Extensions.Logging From Version 8.0.0-rtm.23513.17 -> To Version 9.0.0-alpha.1.23517.5 * Update dependencies from https://github.com/dotnet/runtime build 20231023.1 Microsoft.Extensions.FileSystemGlobbing , Microsoft.Extensions.Logging From Version 8.0.0-rtm.23517.16 -> To Version 9.0.0-alpha.1.23523.1 * Update Version.Details.xml --------- Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> Co-authored-by: Jan Jones <janjones@microsoft.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>

Enable the running of performance tests: "build.cmd -perf"; optional …

8eb35d5

…properties: 'PerfIterations' 'PerfOutputDirectory' 'PerfCollectionType'; update RepoToolset version to a version that contains the performance targets; first draft 'perf.groovy'.

johnbeisner added the build label Feb 16, 2018

johnbeisner self-assigned this Feb 16, 2018

johnbeisner requested review from a team, dsplaisted and tannergooding February 16, 2018 21:57

Merge branch 'release/2.0.0' into CIPerfTests

7f0cf81

DrewScoggins reviewed Feb 20, 2018

View reviewed changes

johnbeisner requested a review from mmitche February 20, 2018 17:38

DrewScoggins reviewed Feb 20, 2018

View reviewed changes

Change 'jobName'

8b0e4b1

DrewScoggins reviewed Feb 20, 2018

View reviewed changes

Changes per code review...

77a157e

dsplaisted approved these changes Feb 20, 2018

View reviewed changes

jbeisner added 2 commits February 20, 2018 19:37

Add "-sign -ci" to the 'build.cmd' switches.

7864c5c

Changes per code review...

a49c505

johnbeisner merged commit ec2d5b1 into dotnet:release/2.0.0 Feb 21, 2018

Conversation

johnbeisner commented Feb 16, 2018

Uh oh!

johnbeisner commented Feb 16, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnbeisner commented Feb 20, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnbeisner Feb 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

johnbeisner Feb 20, 2018 •

edited

Loading