Closed
Description
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 01b57ccc4cc532034f717226f4c7cc07a1ee9518 |
Compare | fc65281072f9259f0b03a291b9c549bc9d792752 |
Diff | Diff |
Improvements in System.Collections.ContainsTrue<Int32>
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
Queue - Duration of single invocation | 444.68 μs | 104.26 μs | 0.23 | 0.00 | False | |||||
List - Duration of single invocation | 443.66 μs | 104.78 μs | 0.24 | 0.01 | False | |||||
Array - Duration of single invocation | 446.69 μs | 107.59 μs | 0.24 | 0.00 | False | |||||
ICollection - Duration of single invocation | 444.79 μs | 108.10 μs | 0.24 | 0.00 | False | |||||
ImmutableArray - Duration of single invocation | 449.38 μs | 112.64 μs | 0.25 | 0.00 | False | |||||
Stack - Duration of single invocation | 108.94 μs | 99.39 μs | 0.91 | 0.23 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrue<Int32>*'
Payloads
Histogram
System.Collections.ContainsTrue<Int32>.Queue(Size: 512)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 104.26052086111112 < 422.250828720238.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 328.6240682083871 (T) = (0 -105793.02979206349) / Math.Sqrt((66891902.981419995 / (299)) + (4335894.453468873 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.7643954051217758 = (449027.8716624529 - 105793.02979206349) / 449027.8716624529 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.ContainsTrue<Int32>.List(Size: 512)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 104.77755935402683 < 422.43166384678517.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 361.5877687355773 (T) = (0 -106798.8511810459) / Math.Sqrt((53441307.47268373 / (299)) + (3556043.951919308 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.7615612766946742 = (447909.0044627011 - 106798.8511810459) / 447909.0044627011 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.ContainsTrue<Int32>.Array(Size: 512)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 107.58901741965228 < 423.65018778183867.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 394.8632313638816 (T) = (0 -109163.35038968897) / Math.Sqrt((66504095.40880481 / (299)) + (2631680.7857848783 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.757867645711311 = (450841.65108945325 - 109163.35038968897) / 450841.65108945325 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.ContainsTrue<Int32>.ICollection(Size: 512)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 108.09500323275863 < 432.13736603330705.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 638.1097270239235 (T) = (0 -108603.44043069237) / Math.Sqrt((70737445.9285433 / (299)) + (247636.69070119353 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.7586171920796709 = (449922.0195770447 - 108603.44043069237) / 449922.0195770447 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.ContainsTrue<Int32>.ImmutableArray(Size: 512)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 112.64162455035971 < 426.21430332270415.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 409.85468598189885 (T) = (0 -112144.64115894896) / Math.Sqrt((80272807.95361523 / (299)) + (2137900.760609265 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.7530316524864402 = (454085.0772497948 - 112144.64115894896) / 454085.0772497948 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.ContainsTrue<Int32>.Stack(Size: 512)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 99.38738127260397 < 101.85794055026844.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 31.688871932244897 (T) = (0 -99607.4292075314) / Math.Sqrt((33786117.415182784 / (299)) + (70722.47504024084 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.1018808888038214 = (110906.70264756662 - 99607.4292075314) / 110906.70264756662 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 01b57ccc4cc532034f717226f4c7cc07a1ee9518 |
Compare | fc65281072f9259f0b03a291b9c549bc9d792752 |
Diff | Diff |
Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Common
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'
Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: IgnoreCase, Compiled)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.695158966341086 < 26.58956422401782.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 1260.884978301478 (T) = (0 -7654.821828412312) / Math.Sqrt((8305.56500443062 / (299)) + (1148.1139655893096 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.7254771153207767 = (27884.093660741193 - 7654.821828412312) / 27884.093660741193 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWords(Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 10.583303429080248 < 30.40967580965654.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 957.8297755007148 (T) = (0 -10648.432944962819) / Math.Sqrt((15081.098757657837 / (299)) + (2185.992996136678 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.6651406105840153 = (31799.71439216423 - 10648.432944962819) / 31799.71439216423 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 378.6082137886953 < 608.5569886057058.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 285.0827232544438 (T) = (0 -378.50408763873975) / Math.Sqrt((179.03687480245082 / (299)) + (0.8679427088914368 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.39829110133190837 = (629.0485124560643 - 378.50408763873975) / 629.0485124560643 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.034605106043544 < 20.269269955855208.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 1679.0173975679288 (T) = (0 -7038.472461702382) / Math.Sqrt((11541.998550226956 / (299)) + (161.3181581848657 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.6675689858171645 = (21172.731067238074 - 7038.472461702382) / 21172.731067238074 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchWord(Options: IgnoreCase, Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 535.6106722854622 < 1.6749654294490575.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 911.3942233723894 (T) = (0 -535.7345633356538) / Math.Sqrt((396.3972990629954 / (299)) + (2.283203490221871 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.6943027062778272 = (1752.5001834741329 - 535.7345633356538) / 1752.5001834741329 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: IgnoreCase, Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 359.2761327612593 < 445.2375514726022.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 56.065933814059086 (T) = (0 -361.4281238111122) / Math.Sqrt((9.128363702858524 / (299)) + (18.50660781126194 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.23057228445079614 = (469.7362942705166 - 361.4281238111122) / 469.7362942705166 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.SplitWords(Options: IgnoreCase, Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 17.02526166973018 < 53.00063627777718.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 796.6353123095807 (T) = (0 -17118.266414952912) / Math.Sqrt((213280.79415273722 / (299)) + (7823.855182263887 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.6895552298606369 = (55141.10096706823 - 17118.266414952912) / 55141.10096706823 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: None)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.426461125633715 < 20.714859115704858.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 1523.2832475020625 (T) = (0 -7426.883820881786) / Math.Sqrt((19102.712579732117 / (299)) + (113.82656401435347 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.6562716560181557 = (21606.841422638325 - 7426.883820881786) / 21606.841422638325 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: None)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 108.0569648994253 < 149.2160383169643.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 51.80814081399865 (T) = (0 -106432.57868575855) / Math.Sqrt((5816650.807709711 / (299)) + (4398317.303972394 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.3158012468419529 = (155557.98398418457 - 106432.57868575855) / 155557.98398418457 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: IgnoreCase, Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 17.621067618128382 < 52.28531185735593.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 923.2009447504194 (T) = (0 -17558.727670804336) / Math.Sqrt((95515.29220842785 / (299)) + (6757.942442068229 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.6824687572871498 = (55297.63786640372 - 17558.727670804336) / 55297.63786640372 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.OneNodeBacktracking(Options: None)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.6320520939247585 < 1.6735758543615513.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 7.46287602269439 (T) = (0 -1619.6255284506256) / Math.Sqrt((1107.4063037749233 / (299)) + (1366.0860232479074 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.07121701018144071 = (1743.814805186112 - 1619.6255284506256) / 1743.814805186112 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.OneNodeBacktracking(Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 244.15409305456413 < 414.58303379377395.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 255.94812253368784 (T) = (0 -246.04272711846215) / Math.Sqrt((97.05921721322174 / (299)) + (1.1149472262465745 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.4349621992297206 = (435.44472030552305 - 246.04272711846215) / 435.44472030552305 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchWord(Options: None)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.907875281240901 < 3.9814536934514204.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 73.58665780160247 (T) = (0 -2863.5941499626265) / Math.Sqrt((3143.853458400775 / (299)) + (1621.7282668287971 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.31983884667022344 = (4210.170098576934 - 2863.5941499626265) / 4210.170098576934 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.OneNodeBacktracking(Options: IgnoreCase, Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 270.6975900554702 < 629.9500397679375.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 296.63292449488426 (T) = (0 -267.57873829655244) / Math.Sqrt((302.834667197841 / (299)) + (3.695999670952534 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.5947091880403088 = (660.2141731334508 - 267.57873829655244) / 660.2141731334508 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 12.792525037232801 < 32.49481549799899.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 1687.5788470772761 (T) = (0 -12819.445017830218) / Math.Sqrt((17640.072296223283 / (299)) + (491.8746195231904 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.6228440380919028 = (33989.77163975993 - 12819.445017830218) / 33989.77163975993 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.SplitWords(Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 12.42607929830586 < 32.19896793031106.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 1539.6088714806071 (T) = (0 -12383.800066714195) / Math.Sqrt((18062.777461525264 / (299)) + (651.0691954326729 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.6318803197878113 = (33640.69005921124 - 12383.800066714195) / 33640.69005921124 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchWord(Options: Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 387.99308588520745 < 1.0347044481499994.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 659.3638107900856 (T) = (0 -390.74719352863747) / Math.Sqrt((24.810459782074837 / (299)) + (5.166557030771572 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.6406576781987121 = (1087.3954160754715 - 390.74719352863747) / 1087.3954160754715 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWords(Options: IgnoreCase, Compiled)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 15.414311957756777 < 50.37579796109587.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 555.1905383967282 (T) = (0 -15405.24676717974) / Math.Sqrt((621884.7362274482 / (299)) + (12936.2353298568 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.7111544757292696 = (53333.86004880812 - 15405.24676717974) / 53333.86004880812 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.SplitWords(Options: None)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 104.03598081563706 < 148.60954569133114.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 57.16955750982869 (T) = (0 -105420.60689364697) / Math.Sqrt((3696692.3077261457 / (299)) + (3778332.931814076 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.3221519637515491 = (155522.478868711 - 105420.60689364697) / 155522.478868711 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWords(Options: None)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 102.53009595315402 < 147.17480453787798.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 41.78055866243218 (T) = (0 -105819.26741762622) / Math.Sqrt((3729233.8366934047 / (299)) + (6553836.409358036 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.3123275865780277 = (153880.34382686947 - 105819.26741762622) / 153880.34382686947 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 01b57ccc4cc532034f717226f4c7cc07a1ee9518 |
Compare | fc65281072f9259f0b03a291b9c549bc9d792752 |
Diff | Diff |
Improvements in System.Tests.Perf_Uri
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
BuilderToString - Duration of single invocation | 497.74 ns | 467.34 ns | 0.94 | 0.00 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Uri*'
Payloads
Histogram
System.Tests.Perf_Uri.BuilderToString
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 467.3419534053783 < 471.96464914023113.
IsChangePoint: Marked as a change because one of 10/3/2022 8:45:37 PM, 11/30/2022 7:06:59 PM, 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 44.224744898008254 (T) = (0 -464.2347698990696) / Math.Sqrt((135.92162587872633 / (299)) + (3.7864671980860116 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.09491740188857388 = (512.9197830869321 - 464.2347698990696) / 512.9197830869321 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment