Closed
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | Windows 10.0.18362 |
| Baseline | 2a6613530b19fafb34c2c1dd731f7e4672d3f53c |
| Compare | 748dfb78ef87b38599b89e85d3201cdb9f9c0b33 |
| Diff | Diff |
Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Count - Duration of single invocation | 219.41 μs | 185.42 μs | 0.85 | 0.01 | False | |||||
| Count - Duration of single invocation | 127.97 μs | 120.84 μs | 0.94 | 0.01 | False | |||||
| Count - Duration of single invocation | 102.70 μs | 91.59 μs | 0.89 | 0.01 | False | |||||
| Count - Duration of single invocation | 291.96 μs | 243.59 μs | 0.83 | 0.01 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes", Options: NonBacktracking)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 185.42012363996045 < 207.46613772397868.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 5/27/2022 11:47:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsImprovementStdDev: Marked as improvement because 36.87162218217572 (T) = (0 -185154.71974726144) / Math.Sqrt((367615232.5744559 / (299)) + (5148138.573540484 / (15))) is greater than 1.9675964973877862 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (15) - 2, .975) and 0.19983030222454995 = (231394.31580827123 - 185154.71974726144) / 231394.31580827123 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock", Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 120.8367683727034 < 122.30840426997244.
IsChangePoint: Marked as a change because one of 3/29/2022 11:07:04 AM, 3/31/2022 1:45:25 PM, 5/11/2022 9:09:01 PM, 5/20/2022 10:18:01 PM, 5/24/2022 7:55:26 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsImprovementStdDev: Marked as improvement because 22.23477825246874 (T) = (0 -120874.39032711006) / Math.Sqrt((11791860.423452394 / (299)) + (1866099.0400869364 / (33))) is greater than 1.9671786747533526 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (33) - 2, .975) and 0.05391773229179572 = (127763.0862059353 - 120874.39032711006) / 127763.0862059353 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: NonBacktracking)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 91.58659595220679 < 98.61362157501536.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 5/27/2022 11:47:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsImprovementStdDev: Marked as improvement because 34.59358165224952 (T) = (0 -91478.10918640625) / Math.Sqrt((44230273.07772734 / (299)) + (937219.8397866668 / (13))) is greater than 1.9676459287777595 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (13) - 2, .975) and 0.15065829047417847 = (107704.7178543457 - 91478.10918640625) / 107704.7178543457 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson", Options: NonBacktracking)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 243.59035447154474 < 274.47274772727275.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 5/27/2022 11:47:05 PM, 5/31/2022 8:51:40 AM falls between 5/22/2022 6:37:49 PM and 5/31/2022 8:51:40 AM.
IsImprovementStdDev: Marked as improvement because 42.56087909228116 (T) = (0 -244604.69992368296) / Math.Sqrt((524686568.553174 / (299)) + (2134465.7755791806 / (15))) is greater than 1.9675964973877862 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (15) - 2, .975) and 0.19332562693373814 = (303226.0700112642 - 244604.69992368296) / 303226.0700112642 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository