Closed
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | ee6de921299d1b86a02d9049b9c8dd164cab1401 |
| Compare | 7fa0d5b5942f138c362f2753398b4a8d4f71eb73 |
| Diff | Diff |
Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Count - Duration of single invocation | 82.47 μs | 69.45 μs | 0.84 | 0.01 | False | |||||
| Count - Duration of single invocation | 573.03 μs | 456.53 μs | 0.80 | 0.01 | False | |||||
| Count - Duration of single invocation | 2.45 ms | 1.65 ms | 0.67 | 0.00 | False | |||||
| Count - Duration of single invocation | 110.87 μs | 93.13 μs | 0.84 | 0.01 | False | |||||
| Count - Duration of single invocation | 90.21 μs | 80.33 μs | 0.89 | 0.01 | False | |||||
| Count - Duration of single invocation | 720.57 μs | 652.43 μs | 0.91 | 0.02 | False | |||||
| Count - Duration of single invocation | 44.18 μs | 34.30 μs | 0.78 | 0.05 | False | |||||
| Count - Duration of single invocation | 1.86 ms | 220.23 μs | 0.12 | 0.07 | False | |||||
| Count - Duration of single invocation | 179.90 μs | 169.58 μs | 0.94 | 0.03 | False | |||||
| Count - Duration of single invocation | 2.42 ms | 1.60 ms | 0.66 | 0.07 | False | |||||
| Count - Duration of single invocation | 67.34 μs | 54.40 μs | 0.81 | 0.03 | False | |||||
| Count - Duration of single invocation | 64.78 μs | 52.48 μs | 0.81 | 0.01 | False | |||||
| Count - Duration of single invocation | 399.69 μs | 323.14 μs | 0.81 | 0.00 | False | |||||
| Count - Duration of single invocation | 138.58 μs | 126.42 μs | 0.91 | 0.03 | False | |||||
| Count - Duration of single invocation | 66.04 μs | 57.20 μs | 0.87 | 0.04 | False | |||||
| Count - Duration of single invocation | 47.35 μs | 32.91 μs | 0.70 | 0.05 | False | |||||
| Count - Duration of single invocation | 89.34 μs | 74.99 μs | 0.84 | 0.00 | False | |||||
| Count - Duration of single invocation | 575.11 μs | 465.36 μs | 0.81 | 0.02 | False | |||||
| Count - Duration of single invocation | 84.93 μs | 72.17 μs | 0.85 | 0.00 | False | |||||
| Count - Duration of single invocation | 1.02 ms | 857.86 μs | 0.84 | 0.00 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Holmes.{0,25}Watson|Watson.{0,25}Holmes", Options: Compiled)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 69.4546364767699 < 78.18511987551511.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 52.21755092070954 (T) = (0 -68424.23611342587) / Math.Sqrt((2122004.736861152 / (227)) + (2203066.7441402692 / (32))) is greater than 1.9692374962334795 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (227) + (32) - 2, .975) and 0.1758689544234081 = (83025.91739588429 - 68424.23611342587) / 83025.91739588429 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Holmes", Options: None)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 456.53354035714284 < 543.3161773967635.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 170.1269101162156 (T) = (0 -458216.57553654286) / Math.Sqrt((111250339.27609079 / (299)) + (3178098.8392946026 / (31))) is greater than 1.9672228266464409 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (31) - 2, .975) and 0.20368096796704088 = (575418.3400172427 - 458216.57553654286) / 575418.3400172427 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock|Holmes|Watson|Irene|Adler|John|Baker", Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.6480423191964286 < 2.3252661010522964.
IsChangePoint: Marked as a change because one of 1/5/2023 12:24:13 PM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 388.10895291381973 (T) = (0 -1627150.2431855414) / Math.Sqrt((16133030.903327523 / (194)) + (79014152.89055267 / (18))) is greater than 1.9713247932431988 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (194) + (18) - 2, .975) and 0.33530491398849727 = (2447964.9051555996 - 1627150.2431855414) / 2447964.9051555996 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson", Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 93.12814421237245 < 105.17879086927928.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 111.84579619920457 (T) = (0 -92831.24534896045) / Math.Sqrt((3558378.1114651 / (299)) + (412044.59819663 / (31))) is greater than 1.9672228266464409 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (31) - 2, .975) and 0.16053364500451095 = (110583.64018586461 - 92831.24534896045) / 110583.64018586461 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: None)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 80.32982232603094 < 86.02229710273018.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 36.2551819110507 (T) = (0 -81121.34356527454) / Math.Sqrt((1134422.0930851388 / (299)) + (2283880.325049543 / (31))) is greater than 1.9672228266464409 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (31) - 2, .975) and 0.110631042897961 = (91212.24989638056 - 81121.34356527454) / 91212.24989638056 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)the", Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 652.433906923077 < 684.5338171725671.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 84.44077989247788 (T) = (0 -637154.707811375) / Math.Sqrt((84993641.7523256 / (181)) + (19814142.398031432 / (31))) is greater than 1.9713247932431988 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (181) + (31) - 2, .975) and 0.12245892366462546 = (726068.2434059306 - 637154.707811375) / 726068.2434059306 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 34.3027143558952 < 41.96515906234003.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 54.759148137968126 (T) = (0 -30884.180276013034) / Math.Sqrt((128685.89659150252 / (299)) + (1705464.0743477072 / (31))) is greater than 1.9672228266464409 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (31) - 2, .975) and 0.2945310584802015 = (43778.22815201325 - 30884.180276013034) / 43778.22815201325 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson|Irene|Adler|John|Baker", Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 220.23377243150685 < 1.6252592409267523.
IsChangePoint: Marked as a change because one of 1/5/2023 12:24:13 PM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 655.1719476078458 (T) = (0 -215300.92015796527) / Math.Sqrt((1140518118.6321115 / (268)) + (14949967.291597521 / (18))) is greater than 1.968352158354924 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (268) + (18) - 2, .975) and 0.8728202658649161 = (1692887.012401547 - 215300.92015796527) / 1692887.012401547 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson", Options: NonBacktracking)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 169.57625091205838 < 171.41242166025248.
IsChangePoint: Marked as a change because one of 10/27/2022 8:00:27 PM, 1/3/2023 6:29:42 AM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 22.903606562265555 (T) = (0 -169902.72356071149) / Math.Sqrt((7388128.494787677 / (299)) + (4703843.267529339 / (31))) is greater than 1.9672228266464409 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (31) - 2, .975) and 0.053590633431117904 = (179523.5017344321 - 169902.72356071149) / 179523.5017344321 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson|Irene|Adler|John|Baker", Options: NonBacktracking)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.5974748483333332 < 2.298189453161285.
IsChangePoint: Marked as a change because one of 1/5/2023 12:24:13 PM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 253.1840974366004 (T) = (0 -1596684.597441621) / Math.Sqrt((1983923438.4231358 / (194)) + (10149752.138270346 / (18))) is greater than 1.9713247932431988 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (194) + (18) - 2, .975) and 0.34248388394015494 = (2428358.1169230165 - 1596684.597441621) / 2428358.1169230165 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: NonBacktracking)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 54.402508540305014 < 62.76832162070525.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 50.497329801212075 (T) = (0 -54832.77701967103) / Math.Sqrt((948103.5878948271 / (299)) + (1643767.4059745478 / (31))) is greater than 1.9672228266464409 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (31) - 2, .975) and 0.17919286818262262 = (66803.48512355624 - 54832.77701967103) / 66803.48512355624 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: None)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 52.48304915408277 < 61.876336127682514.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 64.1807298996082 (T) = (0 -53198.95751221315) / Math.Sqrt((869749.9316476794 / (299)) + (1003923.0776194702 / (31))) is greater than 1.9672228266464409 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (31) - 2, .975) and 0.18476911679095073 = (65256.30788519989 - 53198.95751221315) / 65256.30788519989 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Holmes", Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 323.137130010933 < 379.7539247161459.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 201.2424258508494 (T) = (0 -324912.4578736429) / Math.Sqrt((29021370.710821457 / (299)) + (2388365.794401055 / (31))) is greater than 1.9672228266464409 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (31) - 2, .975) and 0.2053652570122576 = (408882.7738037309 - 324912.4578736429) / 408882.7738037309 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Holmes.{0,25}Watson|Watson.{0,25}Holmes", Options: NonBacktracking)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 126.41618011112864 < 131.06258487755036.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 29.078856083126833 (T) = (0 -126558.12910819733) / Math.Sqrt((4490649.821287423 / (262)) + (3588412.057397878 / (31))) is greater than 1.9681495542477299 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (262) + (31) - 2, .975) and 0.07728732282596576 = (137158.76267767698 - 126558.12910819733) / 137158.76267767698 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: NonBacktracking)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 57.20001123321124 < 63.59647474351331.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 45.17979435878823 (T) = (0 -54324.16067736814) / Math.Sqrt((1124952.266197945 / (299)) + (1863445.9477309664 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.17591102192267255 = (65920.26118843528 - 54324.16067736814) / 65920.26118843528 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 32.90953627819548 < 45.000142786458326.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 69.55594512592796 (T) = (0 -33559.2842682789) / Math.Sqrt((191654.12026152766 / (299)) + (1192856.7928639606 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.2875424115231803 = (47103.55368664976 - 33559.2842682789) / 47103.55368664976 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sher[a-z]+|Hol[a-z]+", Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 74.98629012419873 < 84.63349287026614.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 88.05403705139355 (T) = (0 -73715.7881702187) / Math.Sqrt((1032802.7629140748 / (227)) + (758210.2290978548 / (32))) is greater than 1.9692374962334795 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (227) + (32) - 2, .975) and 0.16718564920369677 = (88514.07051251538 - 73715.7881702187) / 88514.07051251538 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Holmes", Options: NonBacktracking)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 465.3591201923077 < 544.5959712193796.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 105.60801298585781 (T) = (0 -462233.22592465056) / Math.Sqrt((117059295.41380891 / (299)) + (23811604.98998775 / (31))) is greater than 1.9672228266464409 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (31) - 2, .975) and 0.19745303742031445 = (575957.8535302911 - 462233.22592465056) / 575957.8535302911 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes", Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 72.16800830853174 < 80.79191892286975.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 112.74002716998022 (T) = (0 -71063.16257923009) / Math.Sqrt((560479.3675508895 / (290)) + (392905.99693979416 / (31))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (290) + (31) - 2, .975) and 0.16089157626366812 = (84688.89188694386 - 71063.16257923009) / 84688.89188694386 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock|Holmes|Watson", Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 857.8627027138158 < 0.9657799067717633.
IsChangePoint: Marked as a change because one of 1/5/2023 12:24:13 PM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 111.05493917287626 (T) = (0 -832242.3728109167) / Math.Sqrt((1894170.42615868 / (299)) + (49580870.1292949 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.1814828183417596 = (1016768.3604696854 - 832242.3728109167) / 1016768.3604696854 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | ee6de921299d1b86a02d9049b9c8dd164cab1401 |
| Compare | 7fa0d5b5942f138c362f2753398b4a8d4f71eb73 |
| Diff | Diff |
Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Count - Duration of single invocation | 2.61 ms | 2.35 ms | 0.90 | 0.08 | False | |||||
| Count - Duration of single invocation | 13.10 ms | 11.14 ms | 0.85 | 0.01 | False | |||||
| Count - Duration of single invocation | 86.13 ms | 68.40 ms | 0.79 | 0.01 | False | |||||
| Count - Duration of single invocation | 2.25 ms | 1.98 ms | 0.88 | 0.09 | False | |||||
| Count - Duration of single invocation | 16.82 ms | 14.95 ms | 0.89 | 0.03 | False | |||||
| Count - Duration of single invocation | 51.11 ms | 47.39 ms | 0.93 | 0.02 | False | |||||
| Count - Duration of single invocation | 46.61 ms | 21.76 ms | 0.47 | 0.00 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "Huck[a-zA-Z]+|Saw[a-zA-Z]+", Options: NonBacktracking)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.347525996190476 < 2.4760924506830353.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 37.31525428153403 (T) = (0 -2356180.593633696) / Math.Sqrt((1563186373.0105762 / (299)) + (1354108227.0227723 / (31))) is greater than 1.9672228266464409 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (31) - 2, .975) and 0.09971325526695093 = (2617144.601337372 - 2356180.593633696) / 2617144.601337372 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "([A-Za-z]awyer|[A-Za-z]inn)\\s", Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 11.139905642857144 < 12.452405559999997.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 240.2144549685926 (T) = (0 -11124413.752931647) / Math.Sqrt((12968316205.440958 / (299)) + (1249658791.2543507 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.16390898231407847 = (13305266.433457302 - 11124413.752931647) / 13305266.433457302 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Tom|Sawyer|Huckleberry|Finn", Options: NonBacktracking)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 68.40330433333332 < 81.78846822297618.
IsChangePoint: Marked as a change because one of 10/26/2022 2:14:52 PM, 11/10/2022 9:13:35 AM, 1/5/2023 12:24:13 PM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 69.67392054925142 (T) = (0 -68089239.27981786) / Math.Sqrt((25149691836324.793 / (299)) + (29058653745.774532 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.23053610713670175 = (88489193.4648771 - 68089239.27981786) / 88489193.4648771 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "Huck[a-zA-Z]+|Saw[a-zA-Z]+", Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.9831670739583334 < 2.1377789451636904.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 44.253082706548675 (T) = (0 -1980402.0186024723) / Math.Sqrt((1270017463.979567 / (299)) + (1038244269.7218505 / (31))) is greater than 1.9672228266464409 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (31) - 2, .975) and 0.12070412912527054 = (2252258.976983885 - 1980402.0186024723) / 2252258.976983885 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "([A-Za-z]awyer|[A-Za-z]inn)\\s", Options: NonBacktracking)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 14.948125117647058 < 15.977996467256407.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 42.539657381478655 (T) = (0 -14932216.545441717) / Math.Sqrt((313097136000.8987 / (299)) + (54827883192.66146 / (30))) is greater than 1.9672451058622664 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (30) - 2, .975) and 0.13250605581317088 = (17213049.895626497 - 14932216.545441717) / 17213049.895626497 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "Tom.{10,25}river|river.{10,25}Tom", Options: None)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 47.38994173076923 < 48.650788064285706.
IsChangePoint: Marked as a change because one of 1/3/2023 9:28:36 AM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 34.66856044246651 (T) = (0 -46823754.893887) / Math.Sqrt((1197019888516.221 / (225)) + (274183550837.08972 / (31))) is greater than 1.9693475402190106 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (225) + (31) - 2, .975) and 0.08098355482207761 = (50949855.29320085 - 46823754.893887) / 50949855.29320085 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Tom|Sawyer|Huckleberry|Finn", Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 21.763995833333333 < 44.28311432493589.
IsChangePoint: Marked as a change because one of 1/5/2023 12:24:13 PM, 1/9/2023 6:56:09 PM falls between 1/1/2023 3:57:25 AM and 1/9/2023 6:56:09 PM.
IsImprovementStdDev: Marked as improvement because 1115.1627154747416 (T) = (0 -21749901.964711364) / Math.Sqrt((1815382589.3074377 / (299)) + (8852314797.317707 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.5335886722439233 = (46632447.94964779 - 21749901.964711364) / 46632447.94964779 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
