Closed
Description
Run Information
| Architecture | arm64 |
|---|---|
| OS | ubuntu 20.04 |
| Baseline | 3205533cbb03414db5c05e307044606452ac963b |
| Compare | b869f40132d9f152bdc46b5a760953579843bae9 |
| Diff | Diff |
Improvements in Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| AddThenRemove_AbsoluteExpiration - Duration of single invocation | 53.00 μs | 48.03 μs | 0.91 | 0.72 | False | |||||
| CreateEntry - Duration of single invocation | 34.30 ns | 18.72 ns | 0.55 | 0.56 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests*'Payloads
Histogram
Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.AddThenRemove_AbsoluteExpiration
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 48.02848999999999 < 55.00169189708152.
IsChangePoint: Marked as a change because one of 6/23/2022 8:25:58 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsImprovementStdDev: Marked as improvement because 7.973424829761987 (T) = (0 -46795.27896735633) / Math.Sqrt((199073444.94754577 / (236)) + (68862840.99018094 / (23))) is greater than 1.9692374962334795 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (236) + (23) - 2, .975) and 0.2502557460231546 = (62414.98846992367 - 46795.27896735633) / 62414.98846992367 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.CreateEntry
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 18.723777216339357 < 28.642439512170224.
IsChangePoint: Marked as a change because one of 4/27/2022 2:48:42 PM, 6/22/2022 5:33:38 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsImprovementStdDev: Marked as improvement because 25.61438664567369 (T) = (0 -20.118887012834982) / Math.Sqrt((13.915151212552642 / (231)) + (5.897727923008161 / (28))) is greater than 1.9692374962334795 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (231) + (28) - 2, .975) and 0.398538240216248 = (33.44998528263621 - 20.118887012834982) / 33.44998528263621 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
| Architecture | arm64 |
|---|---|
| OS | ubuntu 20.04 |
| Baseline | 3205533cbb03414db5c05e307044606452ac963b |
| Compare | b869f40132d9f152bdc46b5a760953579843bae9 |
| Diff | Diff |
Improvements in System.Tests.Perf_Environment
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| ExpandEnvironmentVariables - Duration of single invocation | 567.82 ns | 508.01 ns | 0.89 | 0.02 | False | |||||
| GetEnvironmentVariables - Duration of single invocation | 20.95 μs | 16.89 μs | 0.81 | 0.01 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Environment*'Payloads
Histogram
System.Tests.Perf_Environment.ExpandEnvironmentVariables
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 508.0129619370397 < 539.4921906625838.
IsChangePoint: Marked as a change because one of 5/24/2022 11:43:29 AM, 6/23/2022 8:25:58 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsImprovementStdDev: Marked as improvement because 35.9848610677172 (T) = (0 -507.2743400422605) / Math.Sqrt((150.468474156507 / (237)) + (21.86452404620951 / (23))) is greater than 1.9692013862338849 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (237) + (23) - 2, .975) and 0.08199853818810146 = (552.5855471308636 - 507.2743400422605) / 552.5855471308636 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Environment.GetEnvironmentVariables
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 16.88809468656344 < 20.46950849845073.
IsChangePoint: Marked as a change because one of 4/27/2022 8:46:50 AM, 6/22/2022 5:33:38 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsImprovementStdDev: Marked as improvement because 81.74054763074922 (T) = (0 -17031.154370407217) / Math.Sqrt((355506.75576574146 / (232)) + (35857.89786764992 / (28))) is greater than 1.9692013862338849 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (232) + (28) - 2, .975) and 0.20290326614564452 = (21366.483698977405 - 17031.154370407217) / 21366.483698977405 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
