Regressions in ICU globalization microbenchmarks #77730
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | alpine 3.15 |
| Baseline | f4b715f1cfadcbf85486f369724528c051be5733 |
| Compare | a8ea8e67dc20689aa2e4c738a2f84ea3483ec872 |
| Diff | Diff |
Regressions in System.Globalization.Tests.StringSearch
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Globalization.Tests.StringSearch*'Related Issues
Regressions
- [Perf] Windows/x64: 3 Regressions on 10/25/2022 10:16:52 PM
- [Perf] Alpine/x64: 13 Regressions on 10/25/2022 10:16:52 PM
- [Perf] Linux/x64: 5 Regressions on 10/25/2022 10:16:52 PM
- [Perf] Windows/arm64: 11 Regressions on 10/25/2022 10:16:52 PM
- [Perf] Windows/arm64: 1 Regression on 10/25/2022 10:16:52 PM
Improvements
- [Perf] Linux/x64: 3 Improvements on 10/25/2022 10:16:52 PM
- [Perf] Linux/arm64: 3 Improvements on 10/25/2022 10:16:52 PM
- [Perf] Windows/arm64: 4 Improvements on 10/25/2022 10:16:52 PM
Payloads
Histogram
Edge Detector Info
System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (en-US, None, True))
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 3.4778568875381324 > 737.7096710539914.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -34.4434562500556 (T) = (0 -3509.878318063499) / Math.Sqrt((75.73519679229132 / (22)) + (192312.28640823922 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -3.9835693731756994 = (704.2900490069602 - 3509.878318063499) / 704.2900490069602 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, None, True))
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 5.463388254122405 > 2.1683168435435682.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -35.92694617570154 (T) = (0 -4866.389786775149) / Math.Sqrt((1081.1735463180112 / (22)) + (177446.77020579352 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -1.3798745308588118 = (2044.8093896021664 - 4866.389786775149) / 2044.8093896021664 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (, IgnoreCase, True))
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 781.4682706839588 > 741.5567027451641.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -8.87722642300704 (T) = (0 -774.2461878940562) / Math.Sqrt((368.9991682802959 / (22)) + (748.8526707185031 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.08088299272502637 = (716.3089743341186 - 774.2461878940562) / 716.3089743341186 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (, IgnoreCase, True))
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 533.1238339243107 > 529.0981993486763.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -25.158729650032445 (T) = (0 -541.8583535099809) / Math.Sqrt((41.9013980568892 / (22)) + (19.89834355128826 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.08076992418216662 = (501.3632794417485 - 541.8583535099809) / 501.3632794417485 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (, None, True))
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 6.714073450271101 > 3.9794821239877067.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -31.92475821744105 (T) = (0 -7190.940700508054) / Math.Sqrt((702.3535633842807 / (22)) + (327517.3024920029 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.8956219622786153 = (3793.446606761323 - 7190.940700508054) / 3793.446606761323 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, None, True))
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 4.704891103454969 > 2.176003494929239.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -34.34595255971069 (T) = (0 -4948.7381858052795) / Math.Sqrt((609.0208225895935 / (22)) + (205537.70176964914 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -1.4121328292796187 = (2051.6026836230308 - 4948.7381858052795) / 2051.6026836230308 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (, None, True))
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 4.604345788545753 > 685.9526031478639.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -32.97429061611585 (T) = (0 -3560.528560361033) / Math.Sqrt((84.24666816328991 / (22)) + (225224.56867890482 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -4.445169009547024 = (653.8876119581141 - 3560.528560361033) / 653.8876119581141 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, None, True))
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 7.697149280985077 > 3.986216307087259.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -28.92035721425405 (T) = (0 -7102.175401152876) / Math.Sqrt((983.2222021904659 / (22)) + (374872.3601622246 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.8648780206002814 = (3808.38602991673 - 7102.175401152876) / 3808.38602991673 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (pl-PL, None, False))
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 738.3453194242094 > 686.7923447017769.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -16.644301811770973 (T) = (0 -730.5555736984468) / Math.Sqrt((133.09015889314318 / (22)) + (286.94769643140063 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.10007798112031158 = (664.0943517062801 - 730.5555736984468) / 664.0943517062801 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, IgnoreCase, True))
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 551.5152330302398 > 523.1353566564028.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -17.48624130281412 (T) = (0 -548.8303287199552) / Math.Sqrt((8.524478883162635 / (22)) + (226.46572909951897 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.10037282183450977 = (498.7676156931622 - 548.8303287199552) / 498.7676156931622 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, None, True))
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 2.582708715125821 > 525.0820958060093.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -36.484324551519855 (T) = (0 -2963.0866452484565) / Math.Sqrt((28.342791575091777 / (22)) + (132096.2924956813 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -4.9217634702666535 = (500.3723401189866 - 2963.0866452484565) / 500.3723401189866 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (, None, True))
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 2.576881128929599 > 523.2868817979111.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -26.485367207732722 (T) = (0 -2956.1671188974465) / Math.Sqrt((12.437656857148067 / (22)) + (249680.1531075944 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -4.929465634472556 = (498.5554013014205 - 2956.1671188974465) / 498.5554013014205 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (en-US, IgnoreCase, True))
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 762.3864551566398 > 752.768160745577.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -12.074856176912661 (T) = (0 -776.4127024266257) / Math.Sqrt((246.3782718320968 / (22)) + (378.96674779275503 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.08296818049824481 = (716.9302998998724 - 776.4127024266257) / 716.9302998998724 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (pl-PL, None, False))
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 557.3742612611512 > 532.3676234159848.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -10.102683564556107 (T) = (0 -551.3130269614716) / Math.Sqrt((147.2814912162584 / (22)) + (294.85844611173354 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.08137099633791905 = (509.82782858843296 - 551.3130269614716) / 509.82782858843296 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository