Skip to content

[Perf] Linux/x64: 258 Improvements on 12/18/2022 12:43:31 PM #10912

Closed

Description

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 01b57ccc4cc532034f717226f4c7cc07a1ee9518
Compare fc65281072f9259f0b03a291b9c549bc9d792752
Diff Diff

Improvements in System.Collections.ContainsTrue<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Queue - Duration of single invocation 444.68 μs 104.26 μs 0.23 0.00 False
List - Duration of single invocation 443.66 μs 104.78 μs 0.24 0.01 False
Array - Duration of single invocation 446.69 μs 107.59 μs 0.24 0.00 False
ICollection - Duration of single invocation 444.79 μs 108.10 μs 0.24 0.00 False
ImmutableArray - Duration of single invocation 449.38 μs 112.64 μs 0.25 0.00 False
Stack - Duration of single invocation 108.94 μs 99.39 μs 0.91 0.23 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrue&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsTrue<Int32>.Queue(Size: 512)


Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 104.26052086111112 < 422.250828720238.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 328.6240682083871 (T) = (0 -105793.02979206349) / Math.Sqrt((66891902.981419995 / (299)) + (4335894.453468873 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.7643954051217758 = (449027.8716624529 - 105793.02979206349) / 449027.8716624529 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsTrue&lt;Int32&gt;.List(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 104.77755935402683 < 422.43166384678517.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 361.5877687355773 (T) = (0 -106798.8511810459) / Math.Sqrt((53441307.47268373 / (299)) + (3556043.951919308 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.7615612766946742 = (447909.0044627011 - 106798.8511810459) / 447909.0044627011 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsTrue&lt;Int32&gt;.Array(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 107.58901741965228 < 423.65018778183867.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 394.8632313638816 (T) = (0 -109163.35038968897) / Math.Sqrt((66504095.40880481 / (299)) + (2631680.7857848783 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.757867645711311 = (450841.65108945325 - 109163.35038968897) / 450841.65108945325 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsTrue&lt;Int32&gt;.ICollection(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 108.09500323275863 < 432.13736603330705.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 638.1097270239235 (T) = (0 -108603.44043069237) / Math.Sqrt((70737445.9285433 / (299)) + (247636.69070119353 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.7586171920796709 = (449922.0195770447 - 108603.44043069237) / 449922.0195770447 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsTrue&lt;Int32&gt;.ImmutableArray(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 112.64162455035971 < 426.21430332270415.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 409.85468598189885 (T) = (0 -112144.64115894896) / Math.Sqrt((80272807.95361523 / (299)) + (2137900.760609265 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.7530316524864402 = (454085.0772497948 - 112144.64115894896) / 454085.0772497948 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsTrue&lt;Int32&gt;.Stack(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 99.38738127260397 < 101.85794055026844.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 31.688871932244897 (T) = (0 -99607.4292075314) / Math.Sqrt((33786117.415182784 / (299)) + (70722.47504024084 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.1018808888038214 = (110906.70264756662 - 99607.4292075314) / 110906.70264756662 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 01b57ccc4cc532034f717226f4c7cc07a1ee9518
Compare fc65281072f9259f0b03a291b9c549bc9d792752
Diff Diff

Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
MatchesWord - Duration of single invocation 27.80 μs 7.70 μs 0.28 0.00 False
MatchesWords - Duration of single invocation 31.66 μs 10.58 μs 0.33 0.00 False
Backtracking - Duration of single invocation 622.22 ns 378.61 ns 0.61 0.01 False
MatchesWord - Duration of single invocation 21.33 μs 7.03 μs 0.33 0.00 False
MatchWord - Duration of single invocation 1.78 μs 535.61 ns 0.30 0.00 False
Backtracking - Duration of single invocation 467.85 ns 359.28 ns 0.77 0.00 False
SplitWords - Duration of single invocation 54.78 μs 17.03 μs 0.31 0.01 False
MatchesWord - Duration of single invocation 21.80 μs 7.43 μs 0.34 0.01 False
ReplaceWords - Duration of single invocation 157.33 μs 108.06 μs 0.69 0.01 False
ReplaceWords - Duration of single invocation 55.04 μs 17.62 μs 0.32 0.00 False
OneNodeBacktracking - Duration of single invocation 1.77 μs 1.63 μs 0.92 0.03 False
OneNodeBacktracking - Duration of single invocation 442.91 ns 244.15 ns 0.55 0.02 False
MatchWord - Duration of single invocation 4.15 μs 2.91 μs 0.70 0.02 False
OneNodeBacktracking - Duration of single invocation 660.59 ns 270.70 ns 0.41 0.01 False
ReplaceWords - Duration of single invocation 33.95 μs 12.79 μs 0.38 0.00 False
SplitWords - Duration of single invocation 33.50 μs 12.43 μs 0.37 0.00 False
MatchWord - Duration of single invocation 1.09 μs 387.99 ns 0.36 0.00 False
MatchesWords - Duration of single invocation 54.05 μs 15.41 μs 0.29 0.01 False
SplitWords - Duration of single invocation 154.58 μs 104.04 μs 0.67 0.01 False
MatchesWords - Duration of single invocation 155.06 μs 102.53 μs 0.66 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: IgnoreCase, Compiled)


Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.695158966341086 < 26.58956422401782.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 1260.884978301478 (T) = (0 -7654.821828412312) / Math.Sqrt((8305.56500443062 / (299)) + (1148.1139655893096 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.7254771153207767 = (27884.093660741193 - 7654.821828412312) / 27884.093660741193 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWords(Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 10.583303429080248 < 30.40967580965654.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 957.8297755007148 (T) = (0 -10648.432944962819) / Math.Sqrt((15081.098757657837 / (299)) + (2185.992996136678 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.6651406105840153 = (31799.71439216423 - 10648.432944962819) / 31799.71439216423 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 378.6082137886953 < 608.5569886057058.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 285.0827232544438 (T) = (0 -378.50408763873975) / Math.Sqrt((179.03687480245082 / (299)) + (0.8679427088914368 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.39829110133190837 = (629.0485124560643 - 378.50408763873975) / 629.0485124560643 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.034605106043544 < 20.269269955855208.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 1679.0173975679288 (T) = (0 -7038.472461702382) / Math.Sqrt((11541.998550226956 / (299)) + (161.3181581848657 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.6675689858171645 = (21172.731067238074 - 7038.472461702382) / 21172.731067238074 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchWord(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 535.6106722854622 < 1.6749654294490575.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 911.3942233723894 (T) = (0 -535.7345633356538) / Math.Sqrt((396.3972990629954 / (299)) + (2.283203490221871 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.6943027062778272 = (1752.5001834741329 - 535.7345633356538) / 1752.5001834741329 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 359.2761327612593 < 445.2375514726022.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 56.065933814059086 (T) = (0 -361.4281238111122) / Math.Sqrt((9.128363702858524 / (299)) + (18.50660781126194 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.23057228445079614 = (469.7362942705166 - 361.4281238111122) / 469.7362942705166 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.SplitWords(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 17.02526166973018 < 53.00063627777718.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 796.6353123095807 (T) = (0 -17118.266414952912) / Math.Sqrt((213280.79415273722 / (299)) + (7823.855182263887 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.6895552298606369 = (55141.10096706823 - 17118.266414952912) / 55141.10096706823 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.426461125633715 < 20.714859115704858.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 1523.2832475020625 (T) = (0 -7426.883820881786) / Math.Sqrt((19102.712579732117 / (299)) + (113.82656401435347 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.6562716560181557 = (21606.841422638325 - 7426.883820881786) / 21606.841422638325 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 108.0569648994253 < 149.2160383169643.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 51.80814081399865 (T) = (0 -106432.57868575855) / Math.Sqrt((5816650.807709711 / (299)) + (4398317.303972394 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.3158012468419529 = (155557.98398418457 - 106432.57868575855) / 155557.98398418457 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 17.621067618128382 < 52.28531185735593.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 923.2009447504194 (T) = (0 -17558.727670804336) / Math.Sqrt((95515.29220842785 / (299)) + (6757.942442068229 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.6824687572871498 = (55297.63786640372 - 17558.727670804336) / 55297.63786640372 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.OneNodeBacktracking(Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.6320520939247585 < 1.6735758543615513.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 7.46287602269439 (T) = (0 -1619.6255284506256) / Math.Sqrt((1107.4063037749233 / (299)) + (1366.0860232479074 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.07121701018144071 = (1743.814805186112 - 1619.6255284506256) / 1743.814805186112 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.OneNodeBacktracking(Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 244.15409305456413 < 414.58303379377395.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 255.94812253368784 (T) = (0 -246.04272711846215) / Math.Sqrt((97.05921721322174 / (299)) + (1.1149472262465745 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.4349621992297206 = (435.44472030552305 - 246.04272711846215) / 435.44472030552305 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchWord(Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.907875281240901 < 3.9814536934514204.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 73.58665780160247 (T) = (0 -2863.5941499626265) / Math.Sqrt((3143.853458400775 / (299)) + (1621.7282668287971 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.31983884667022344 = (4210.170098576934 - 2863.5941499626265) / 4210.170098576934 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.OneNodeBacktracking(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 270.6975900554702 < 629.9500397679375.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 296.63292449488426 (T) = (0 -267.57873829655244) / Math.Sqrt((302.834667197841 / (299)) + (3.695999670952534 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.5947091880403088 = (660.2141731334508 - 267.57873829655244) / 660.2141731334508 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 12.792525037232801 < 32.49481549799899.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 1687.5788470772761 (T) = (0 -12819.445017830218) / Math.Sqrt((17640.072296223283 / (299)) + (491.8746195231904 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.6228440380919028 = (33989.77163975993 - 12819.445017830218) / 33989.77163975993 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.SplitWords(Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 12.42607929830586 < 32.19896793031106.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 1539.6088714806071 (T) = (0 -12383.800066714195) / Math.Sqrt((18062.777461525264 / (299)) + (651.0691954326729 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.6318803197878113 = (33640.69005921124 - 12383.800066714195) / 33640.69005921124 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchWord(Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 387.99308588520745 < 1.0347044481499994.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 659.3638107900856 (T) = (0 -390.74719352863747) / Math.Sqrt((24.810459782074837 / (299)) + (5.166557030771572 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.6406576781987121 = (1087.3954160754715 - 390.74719352863747) / 1087.3954160754715 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWords(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 15.414311957756777 < 50.37579796109587.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 555.1905383967282 (T) = (0 -15405.24676717974) / Math.Sqrt((621884.7362274482 / (299)) + (12936.2353298568 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.7111544757292696 = (53333.86004880812 - 15405.24676717974) / 53333.86004880812 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.SplitWords(Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 104.03598081563706 < 148.60954569133114.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 57.16955750982869 (T) = (0 -105420.60689364697) / Math.Sqrt((3696692.3077261457 / (299)) + (3778332.931814076 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.3221519637515491 = (155522.478868711 - 105420.60689364697) / 155522.478868711 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWords(Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 102.53009595315402 < 147.17480453787798.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 41.78055866243218 (T) = (0 -105819.26741762622) / Math.Sqrt((3729233.8366934047 / (299)) + (6553836.409358036 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.3123275865780277 = (153880.34382686947 - 105819.26741762622) / 153880.34382686947 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 01b57ccc4cc532034f717226f4c7cc07a1ee9518
Compare fc65281072f9259f0b03a291b9c549bc9d792752
Diff Diff

Improvements in System.Tests.Perf_Uri

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BuilderToString - Duration of single invocation 497.74 ns 467.34 ns 0.94 0.00 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Uri*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Uri.BuilderToString


Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 467.3419534053783 < 471.96464914023113.
IsChangePoint: Marked as a change because one of 10/3/2022 8:45:37 PM, 11/30/2022 7:06:59 PM, 12/17/2022 10:44:50 PM, 12/20/2022 4:30:45 AM falls between 12/11/2022 2:11:02 PM and 12/20/2022 4:30:45 AM.
IsImprovementStdDev: Marked as improvement because 44.224744898008254 (T) = (0 -464.2347698990696) / Math.Sqrt((135.92162587872633 / (299)) + (3.7864671980860116 / (5))) is greater than 1.9678502273660254 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (5) - 2, .975) and 0.09491740188857388 = (512.9197830869321 - 464.2347698990696) / 512.9197830869321 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions