Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

[performanceautofiler]

Run Information

Architecture	x64
OS	alpine 3.12
Baseline	0c5e7daf58e5d79cb7a2a8767d1f0e6ec4de4dd4
Compare	22870c2e31ae3c028cece52babd3cae57b560713
Diff	Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Count - Duration of single invocation	75.65 ms	80.81 ms	1.07	0.01	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Tom|Sawyer|Huckleberry|Finn", Options: NonBacktracking)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 80.8123890576923 > 79.49659552624999.
IsChangePoint: Marked as a change because one of 3/6/2022 9:32:10 PM, 4/6/2022 12:51:22 AM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -45.46775125534151 (T) = (0 -81103148.30648707) / Math.Sqrt((203461295893.64603 / (26)) + (152571896797.61606 / (31))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (31) - 2, .025) and -0.06757233312416588 = (75969698.53007068 - 81103148.30648707) / 75969698.53007068 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run Information

Architecture	x64
OS	alpine 3.12
Baseline	0c5e7daf58e5d79cb7a2a8767d1f0e6ec4de4dd4
Compare	22870c2e31ae3c028cece52babd3cae57b560713
Diff	Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Count - Duration of single invocation	75.65 ms	80.81 ms	1.07	0.01	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Tom|Sawyer|Huckleberry|Finn", Options: NonBacktracking)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 80.8123890576923 > 79.49659552624999.
IsChangePoint: Marked as a change because one of 3/6/2022 9:32:10 PM, 4/6/2022 12:51:22 AM, 4/12/2022 4:54:37 AM falls between 4/3/2022 1:52:36 PM and 4/12/2022 4:54:37 AM.
IsRegressionStdDev: Marked as regression because -45.46775125534151 (T) = (0 -81103148.30648707) / Math.Sqrt((203461295893.64603 / (26)) + (152571896797.61606 / (31))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (31) - 2, .025) and -0.06757233312416588 = (75969698.53007068 - 81103148.30648707) / 75969698.53007068 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	performanceautofiler[bot]
Assignees:	-
Labels:	area-System.Text.RegularExpressions, untriaged, refs/heads/main, RunKind=micro, Regression, CoreClr, x64, alpine 3.12
Milestone:	-

[EgorBo]

#67184 cc @joperezr

[danmoseley]

@DrewScoggins is it possible to have the script include not just the test name, but also the parameters? In this case, it runs with many inputs.

Assuming the report is listing them all, there was only one that regressed: [System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Tom|Sawyer|Huckleberry|Finn", Options: NonBacktracking)]