Closed
Description
Run Information
| Architecture | arm64 |
|---|---|
| OS | ubuntu 20.04 |
| Baseline | 99ba09612b31e90e1e7feb5511ee541f9f3ef02b |
| Compare | e09e643ca91ab0deb427878a151875b2e5b7d53c |
| Diff | Diff |
Regressions in System.Tests.Perf_Int64
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| ToString - Duration of single invocation | 651.01 ns | 859.40 ns | 1.32 | 0.10 | False | |||||
| ToString - Duration of single invocation | 482.65 ns | 713.44 ns | 1.48 | 0.18 | False | |||||
| ToString - Duration of single invocation | 254.81 ns | 457.91 ns | 1.80 | 0.05 | False | |||||
| TryFormat - Duration of single invocation | 690.54 ns | 920.76 ns | 1.33 | 0.06 | False | |||||
| TryFormat - Duration of single invocation | 278.69 ns | 492.14 ns | 1.77 | 0.08 | False | |||||
| TryFormat - Duration of single invocation | 515.52 ns | 722.77 ns | 1.40 | 0.19 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int64*'Payloads
Histogram
System.Tests.Perf_Int64.ToString(value: -9223372036854775808)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/8/2022 2:00:38 AM, 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -33.095376716597784 (T) = (0 -865.2700159776588) / Math.Sqrt((26.7361599658936 / (3)) + (237.9035898515841 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.3342514500045192 = (648.5059588840829 - 865.2700159776588) / 648.5059588840829 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int64.ToString(value: 9223372036854775807)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/8/2022 2:00:38 AM, 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -30.066255784866158 (T) = (0 -722.7462823488792) / Math.Sqrt((51.79595366498003 / (3)) + (304.2707066250194 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.4797134624241048 = (488.43664716333495 - 722.7462823488792) / 488.43664716333495 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int64.ToString(value: 12345)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/8/2022 2:00:38 AM, 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -195.09654879432358 (T) = (0 -456.41526839401934) / Math.Sqrt((1.5554958294189751 / (3)) + (3.7873584276129653 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.7857060016433914 = (255.59373602036325 - 456.41526839401934) / 255.59373602036325 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int64.TryFormat(value: -9223372036854775808)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 10/7/2022 12:48:42 AM, 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -36.3267430311068 (T) = (0 -914.4111772346594) / Math.Sqrt((51.209639554803545 / (3)) + (128.35536810415692 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.3095657366782918 = (698.2552701432611 - 914.4111772346594) / 698.2552701432611 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int64.TryFormat(value: 12345)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -71.74210042293288 (T) = (0 -484.5741103214599) / Math.Sqrt((5.512854784108389 / (3)) + (44.16186911567508 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.7318104690814088 = (279.8078190268066 - 484.5741103214599) / 279.8078190268066 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Int64.TryFormat(value: 9223372036854775807)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -25.588873538238158 (T) = (0 -730.7510879648489) / Math.Sqrt((152.53667936500136 / (3)) + (85.95428697386826 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.38545530966248626 = (527.444720063088 - 730.7510879648489) / 527.444720063088 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
| Architecture | arm64 |
|---|---|
| OS | ubuntu 20.04 |
| Baseline | 99ba09612b31e90e1e7feb5511ee541f9f3ef02b |
| Compare | e09e643ca91ab0deb427878a151875b2e5b7d53c |
| Diff | Diff |
Regressions in System.Tests.Perf_Enum
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Enum*'Payloads
Histogram
System.Tests.Perf_Enum.InterpolateIntoString(value: 32)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -17.177807615232204 (T) = (0 -12337.08621807441) / Math.Sqrt((6185.100276183106 / (3)) + (28266.01644288279 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.12201339175610751 = (10995.489277329523 - 12337.08621807441) / 10995.489277329523 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Enum.ToString_NonFlags_Small(value: -1)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -27.541687571200402 (T) = (0 -1327.0352187605993) / Math.Sqrt((81.79439712410466 / (3)) + (423.3172028492839 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.24131592739767585 = (1069.0551772284332 - 1327.0352187605993) / 1069.0551772284332 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Enum.ToString_Flags(value: 32)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -15.43029237067653 (T) = (0 -1371.4184426303113) / Math.Sqrt((273.7942960707263 / (3)) + (899.0560605404988 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.20015025601432546 = (1142.7056201985608 - 1371.4184426303113) / 1142.7056201985608 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Enum.ToString_Format_Flags_Large(value: All, format: "d")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -13.7999906797011 (T) = (0 -1171.529776057601) / Math.Sqrt((710.6974121535924 / (3)) + (507.1937143459526 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.2613252616109037 = (928.8086203564811 - 1171.529776057601) / 928.8086203564811 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Enum.StringFormat(value: 32)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -12.2260905759331 (T) = (0 -9720.872222255932) / Math.Sqrt((6972.31771952942 / (3)) + (42234.36819946623 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.12991814547534639 = (8603.164982509816 - 9720.872222255932) / 8603.164982509816 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Enum.ToString_Flags(value: 36)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -12.60590109145653 (T) = (0 -1452.9357068415047) / Math.Sqrt((860.9770571999692 / (3)) + (967.4027692841173 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.2178846158604931 = (1192.9994745971374 - 1452.9357068415047) / 1192.9994745971374 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Enum.InterpolateIntoSpan_NonFlags(value: 42)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -7.569973995795917 (T) = (0 -11885.385673088953) / Math.Sqrt((11825.344733907059 / (3)) + (112769.48968650837 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.09913019004768546 = (10813.446651459286 - 11885.385673088953) / 10813.446651459286 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Enum.InterpolateIntoSpan_Flags(value: 32)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -7.619315676823492 (T) = (0 -11789.987124046515) / Math.Sqrt((76554.24917959468 / (3)) + (50643.905352137845 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.132448192493553 = (10411.060922871873 - 11789.987124046515) / 10411.060922871873 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Enum.InterpolateIntoStringBuilder_NonFlags(value: 42)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -12.06516781542372 (T) = (0 -12035.942375315399) / Math.Sqrt((5142.016405430455 / (3)) + (51493.64012834289 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.1055453792561089 = (10886.882258432533 - 12035.942375315399) / 10886.882258432533 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Enum.ToString_NonFlags_Large(value: 42)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -26.430989632963186 (T) = (0 -1187.904119672497) / Math.Sqrt((177.64188723732553 / (3)) + (264.38984780682273 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.28060643720911754 = (927.6106110018853 - 1187.904119672497) / 927.6106110018853 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Enum.InterpolateIntoStringBuilder_Flags(value: 32)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -11.430229941794801 (T) = (0 -11636.042278074277) / Math.Sqrt((4119.94214008344 / (3)) + (25544.61884455195 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.07482548929374494 = (10825.982816727004 - 11636.042278074277) / 10825.982816727004 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | arm64 |
|---|---|
| OS | ubuntu 20.04 |
| Baseline | 99ba09612b31e90e1e7feb5511ee541f9f3ef02b |
| Compare | e09e643ca91ab0deb427878a151875b2e5b7d53c |
| Diff | Diff |
Regressions in System.Text.Json.Tests.Perf_Deep
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| WriteDeepUtf16 - Duration of single invocation | 151.67 ms | 173.07 ms | 1.14 | 0.01 | False | |||||
| WriteDeepUtf8 - Duration of single invocation | 205.42 ms | 222.09 ms | 1.08 | 0.01 | False | |||||
| WriteDeepUtf16 - Duration of single invocation | 196.96 ms | 218.14 ms | 1.11 | 0.03 | False | |||||
| WriteDeepUtf8 - Duration of single invocation | 199.31 ms | 219.45 ms | 1.10 | 0.03 | False | |||||
| WriteDeepUtf8 - Duration of single invocation | 151.93 ms | 170.97 ms | 1.13 | 0.03 | False | |||||
| WriteDeepUtf16 - Duration of single invocation | 152.34 ms | 162.52 ms | 1.07 | 0.08 | False | |||||
| WriteDeepUtf16 - Duration of single invocation | 201.36 ms | 223.42 ms | 1.11 | 0.01 | False | |||||
| WriteDeepUtf8 - Duration of single invocation | 146.65 ms | 165.21 ms | 1.13 | 0.05 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Deep*'Payloads
Histogram
System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: False)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -20.95542695073325 (T) = (0 -171275693.1252747) / Math.Sqrt((1635224229119.0781 / (3)) + (2492561183545.021 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.13140729714259708 = (151382878.25952384 - 171275693.1252747) / 151382878.25952384 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: True, SkipValidation: False)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -16.73831426654424 (T) = (0 -224479007.76713768) / Math.Sqrt((2894850251741.879 / (3)) + (3400601215370.486 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.09867360685482479 = (204318194.56349206 - 224479007.76713768) / 204318194.56349206 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: True, SkipValidation: True)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -20.40066484030584 (T) = (0 -217825328.14201993) / Math.Sqrt((2097286694964.6772 / (3)) + (2908033348946.6255 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.1097225864985254 = (196288091.08888888 - 217825328.14201993) / 196288091.08888888 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: True, SkipValidation: True)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -17.749747299324902 (T) = (0 -217998984.19183677) / Math.Sqrt((2924868027481.7627 / (3)) + (2028955682295.9502 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.10079941052200278 = (198036973.95555556 - 217998984.19183677) / 198036973.95555556 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: False)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -18.123757176587247 (T) = (0 -170707883.54149657) / Math.Sqrt((1717438624248.9868 / (3)) + (4166032651884.2524 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.12958867949807887 = (151123932.66666666 - 170707883.54149657) / 151123932.66666666 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: True)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -6.402736807338654 (T) = (0 -164076576.29489797) / Math.Sqrt((20305267561746.72 / (3)) + (1486350044866.8816 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.11495495196796798 = (147159825.6102564 - 164076576.29489797) / 147159825.6102564 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: True, SkipValidation: False)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -19.95623596472667 (T) = (0 -224217865.9802721) / Math.Sqrt((1397914486633.5112 / (3)) + (5390138137686.538 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.10981641284739044 = (202031492.2222222 - 224217865.9802721) / 202031492.2222222 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: True)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -16.69303026034224 (T) = (0 -165218903.11206174) / Math.Sqrt((3596417388831.634 / (3)) + (1336948890361.8418 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.13521663047183105 = (145539537.28055558 - 165218903.11206174) / 145539537.28055558 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository