Closed
Description
Run Information
| Name | Value |
|---|---|
| Architecture | arm64 |
| OS | ubuntu 20.04 |
| Queue | AmpereUbuntu |
| Baseline | f11edcdc155c810f0f114f8b2ccb02d8e96ab82c |
| Compare | 5d54b08d5fc40d0b1c156f430a487a94c1e34f79 |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Improvements in System.Threading.Tasks.ValueTaskPerfTest
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio |
|---|---|---|---|---|---|---|---|---|
| Await_FromCompletedTask - Duration of single invocation | 35.62 ns | 28.61 ns | 0.80 | 0.27 | False | |||
| CreateAndAwait_FromCompletedTask_ConfigureAwait - Duration of single invocation | 25.14 ns | 19.92 ns | 0.79 | 0.08 | False | |||
| CreateAndAwait_FromDelayedTCS - Duration of single invocation | 222.81 ns | 211.47 ns | 0.95 | 0.42 | False | |||
| CreateAndAwait_FromResult_ConfigureAwait - Duration of single invocation | 16.25 ns | 12.61 ns | 0.78 | 0.33 | False | |||
| Await_FromResult - Duration of single invocation | 16.12 ns | 11.91 ns | 0.74 | 0.35 | False | |||
| CreateAndAwait_FromCompletedTask - Duration of single invocation | 25.34 ns | 19.62 ns | 0.77 | 0.11 | False | |||
| CreateAndAwait_FromResult - Duration of single invocation | 16.30 ns | 12.19 ns | 0.75 | 0.34 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Threading.Tasks.ValueTaskPerfTest*'Payloads
System.Threading.Tasks.ValueTaskPerfTest.Await_FromCompletedTask
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 28.61487390899056 < 33.34832279729494.
IsChangePoint: Marked as a change because one of 7/6/2023 2:30:01 AM, 7/12/2023 7:14:29 PM falls between 7/3/2023 9:53:34 PM and 7/12/2023 7:14:29 PM.
IsImprovementStdDev: Marked as improvement because 44.19649391917961 (T) = (0 -28.32453875511059) / Math.Sqrt((1.3402980768082071 / (299)) + (0.23652569379052643 / (13))) is greater than 1.9676459287777595 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (13) - 2, .975) and 0.19026524492290722 = (34.980021022333275 - 28.32453875511059) / 34.980021022333275 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromCompletedTask_ConfigureAwait
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 19.92428416614487 < 23.95357382251005.
IsChangePoint: Marked as a change because one of 7/6/2023 2:30:01 AM, 7/12/2023 7:14:29 PM falls between 7/3/2023 9:53:34 PM and 7/12/2023 7:14:29 PM.
IsImprovementStdDev: Marked as improvement because 31.241158145501796 (T) = (0 -19.955999671050797) / Math.Sqrt((0.31571572244260665 / (299)) + (0.3866985364468786 / (13))) is greater than 1.9676459287777595 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (13) - 2, .975) and 0.21553441115456992 = (25.438973939471165 - 19.955999671050797) / 25.438973939471165 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromDelayedTCS
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 211.46507138644685 < 220.31627455221306.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 4:22:12 AM, 7/6/2023 2:30:01 AM, 7/12/2023 7:14:29 PM falls between 7/3/2023 9:53:34 PM and 7/12/2023 7:14:29 PM.
IsImprovementStdDev: Marked as improvement because 16.783999421544404 (T) = (0 -205.64069899862946) / Math.Sqrt((401.8422417571078 / (299)) + (122.76542870863013 / (13))) is greater than 1.9676459287777595 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (13) - 2, .975) and 0.21139905644619017 = (260.7664886525686 - 205.64069899862946) / 260.7664886525686 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromResult_ConfigureAwait
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 12.605457328978128 < 15.172555552515997.
IsChangePoint: Marked as a change because one of 4/27/2023 10:53:57 PM, 7/6/2023 2:30:01 AM, 7/12/2023 7:14:29 PM falls between 7/3/2023 9:53:34 PM and 7/12/2023 7:14:29 PM.
IsImprovementStdDev: Marked as improvement because 20.75613035005599 (T) = (0 -11.902770511159709) / Math.Sqrt((0.7265589252259097 / (299)) + (0.6371686130458514 / (13))) is greater than 1.9676459287777595 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (13) - 2, .975) and 0.2834178394042537 = (16.610475623987178 - 11.902770511159709) / 16.610475623987178 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Threading.Tasks.ValueTaskPerfTest.Await_FromResult
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 11.911558067903377 < 15.482179970424674.
IsChangePoint: Marked as a change because one of 4/26/2023 2:24:59 PM, 7/6/2023 3:11:59 PM, 7/12/2023 7:14:29 PM falls between 7/3/2023 9:53:34 PM and 7/12/2023 7:14:29 PM.
IsImprovementStdDev: Marked as improvement because 24.361620929467236 (T) = (0 -11.811223363708569) / Math.Sqrt((0.7063935548926421 / (299)) + (0.4217720312694984 / (12))) is greater than 1.9676708853560316 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (12) - 2, .975) and 0.28544460092106877 = (16.529471863110054 - 11.811223363708569) / 16.529471863110054 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromCompletedTask
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 19.621969693199542 < 23.92709083614278.
IsChangePoint: Marked as a change because one of 7/6/2023 2:30:01 AM, 7/12/2023 7:14:29 PM falls between 7/3/2023 9:53:34 PM and 7/12/2023 7:14:29 PM.
IsImprovementStdDev: Marked as improvement because 47.5834908052072 (T) = (0 -19.63970524257827) / Math.Sqrt((0.4157620584244874 / (299)) + (0.1773791566997335 / (13))) is greater than 1.9676459287777595 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (13) - 2, .975) and 0.22903772890066643 = (25.474275433185003 - 19.63970524257827) / 25.474275433185003 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromResult
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 12.18794280495896 < 14.857679884447988.
IsChangePoint: Marked as a change because one of 4/26/2023 12:55:36 AM, 7/6/2023 2:30:01 AM, 7/12/2023 7:14:29 PM falls between 7/3/2023 9:53:34 PM and 7/12/2023 7:14:29 PM.
IsImprovementStdDev: Marked as improvement because 18.94918227330156 (T) = (0 -12.025091340092189) / Math.Sqrt((0.8187101230440123 / (299)) + (0.7171015338077503 / (13))) is greater than 1.9676459287777595 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (13) - 2, .975) and 0.27492930727599096 = (16.58471575359814 - 12.025091340092189) / 16.58471575359814 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | arm64 |
| OS | ubuntu 20.04 |
| Queue | AmpereUbuntu |
| Baseline | f11edcdc155c810f0f114f8b2ccb02d8e96ab82c |
| Compare | 5d54b08d5fc40d0b1c156f430a487a94c1e34f79 |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Improvements in System.Tests.Perf_String
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio |
|---|---|---|---|---|---|---|---|---|
| ToUpper - Duration of single invocation | 16.68 ns | 11.17 ns | 0.67 | 0.25 | False | |||
| ToLower - Duration of single invocation | 32.34 ns | 27.51 ns | 0.85 | 0.28 | False | |||
| ToLower - Duration of single invocation | 32.03 ns | 27.06 ns | 0.84 | 0.30 | False | |||
| ToUpper - Duration of single invocation | 32.32 ns | 27.12 ns | 0.84 | 0.34 | False | |||
| ToUpper - Duration of single invocation | 32.03 ns | 27.02 ns | 0.84 | 0.31 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'Payloads
System.Tests.Perf_String.ToUpper(s: "TEST")
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 11.173341397436122 < 15.640955084950953.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 7/6/2023 2:30:01 AM, 7/12/2023 7:14:29 PM falls between 7/3/2023 9:53:34 PM and 7/12/2023 7:14:29 PM.
IsImprovementStdDev: Marked as improvement because 23.648928681558807 (T) = (0 -11.798848664146071) / Math.Sqrt((8.685046794585997 / (299)) + (0.7044760600327841 / (13))) is greater than 1.9676459287777595 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (13) - 2, .975) and 0.3663951441390164 = (18.621777524278954 - 11.798848664146071) / 18.621777524278954 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Tests.Perf_String.ToLower(s: "TeSt")
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 27.508859473210574 < 30.698693552724677.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 6/22/2023 12:06:43 AM, 7/6/2023 2:30:01 AM, 7/12/2023 7:14:29 PM falls between 7/3/2023 9:53:34 PM and 7/12/2023 7:14:29 PM.
IsImprovementStdDev: Marked as improvement because 45.37365214793177 (T) = (0 -26.859124503868696) / Math.Sqrt((5.853930262406842 / (299)) + (0.315940277568661 / (13))) is greater than 1.9676459287777595 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (13) - 2, .975) and 0.2613805025478663 = (36.36395274768563 - 26.859124503868696) / 36.36395274768563 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Tests.Perf_String.ToLower(s: "TEST")
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 27.056416604588595 < 30.77977791741454.
IsChangePoint: Marked as a change because one of 4/25/2023 1:02:54 PM, 6/22/2023 12:06:43 AM, 7/6/2023 2:30:01 AM, 7/12/2023 7:14:29 PM falls between 7/3/2023 9:53:34 PM and 7/12/2023 7:14:29 PM.
IsImprovementStdDev: Marked as improvement because 43.025941345937746 (T) = (0 -27.23199572013145) / Math.Sqrt((4.672958749224636 / (299)) + (0.36404854280507326 / (13))) is greater than 1.9676459287777595 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (13) - 2, .975) and 0.24813804897669597 = (36.2194092719654 - 27.23199572013145) / 36.2194092719654 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Tests.Perf_String.ToUpper(s: "TeSt")
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 27.11879204879119 < 30.236438170933877.
IsChangePoint: Marked as a change because one of 4/25/2023 1:02:54 PM, 6/22/2023 12:06:43 AM, 7/6/2023 2:30:01 AM, 7/12/2023 7:14:29 PM falls between 7/3/2023 9:53:34 PM and 7/12/2023 7:14:29 PM.
IsImprovementStdDev: Marked as improvement because 45.95882696750559 (T) = (0 -26.566791047323022) / Math.Sqrt((6.229309952122278 / (299)) + (0.3174116356301178 / (13))) is greater than 1.9676459287777595 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (13) - 2, .975) and 0.26900211298182225 = (36.343184459385974 - 26.566791047323022) / 36.343184459385974 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Tests.Perf_String.ToUpper(s: "test")
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 27.020194936049585 < 29.84573311864761.
IsChangePoint: Marked as a change because one of 6/22/2023 12:06:43 AM, 7/6/2023 2:30:01 AM, 7/12/2023 7:14:29 PM falls between 7/3/2023 9:53:34 PM and 7/12/2023 7:14:29 PM.
IsImprovementStdDev: Marked as improvement because 44.96770790275354 (T) = (0 -26.44897786388619) / Math.Sqrt((5.623792628427194 / (299)) + (0.3796516637425127 / (13))) is greater than 1.9676459287777595 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (13) - 2, .975) and 0.27142231031783565 = (36.302206667108244 - 26.44897786388619) / 36.302206667108244 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment