Skip to content

[Perf] Linux/x64: 77 Regressions on 12/21/2022 11:29:15 AM #11104

Closed

Description

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<SimpleStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromString - Duration of single invocation 635.53 ns 671.84 ns 1.06 0.01 False
DeserializeFromString - Duration of single invocation 666.67 ns 704.45 ns 1.06 0.00 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;SimpleStructWithProperties&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<SimpleStructWithProperties>.DeserializeFromString(Mode: Reflection)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 671.8423643592581 > 668.5774539053499.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -28.392906477601986 (T) = (0 -676.0908990397227) / Math.Sqrt((17.45796218906637 / (32)) + (23.127198436526296 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.06023965598256064 = (637.6774300270498 - 676.0908990397227) / 637.6774300270498 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;SimpleStructWithProperties&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 704.4519084995876 > 695.1266650050139.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -29.148608111136923 (T) = (0 -705.8950804384264) / Math.Sqrt((26.340632578085096 / (32)) + (19.91482631595129 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.06084942888159289 = (665.4055337359428 - 705.8950804384264) / 665.4055337359428 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.WriteJson<HashSet<String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToUtf8Bytes - Duration of single invocation 22.17 μs 23.72 μs 1.07 0.01 False
SerializeToUtf8Bytes - Duration of single invocation 20.48 μs 22.23 μs 1.09 0.01 False
SerializeToWriter - Duration of single invocation 21.41 μs 23.15 μs 1.08 0.01 False
SerializeToWriter - Duration of single invocation 20.29 μs 21.47 μs 1.06 0.01 False
SerializeToStream - Duration of single invocation 21.87 μs 23.67 μs 1.08 0.01 False
SerializeToStream - Duration of single invocation 20.55 μs 22.11 μs 1.08 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;HashSet&lt;String&gt;&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<HashSet<String>>.SerializeToUtf8Bytes(Mode: Reflection)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.722381299608966 > 23.05005926126403.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -60.010794037349804 (T) = (0 -23594.071658681518) / Math.Sqrt((7798.724719306144 / (32)) + (8909.918142917486 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.07426305570329186 = (21963.029942638303 - 23594.071658681518) / 21963.029942638303 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;HashSet&lt;String&gt;&gt;.SerializeToUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.23029101266572 > 21.567770277638274.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -77.70312536904908 (T) = (0 -22201.29154244032) / Math.Sqrt((10623.136991051262 / (32)) + (1089.5976068772843 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.07450620686236256 = (20661.855092740443 - 22201.29154244032) / 20661.855092740443 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;HashSet&lt;String&gt;&gt;.SerializeToWriter(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.147714685487863 > 22.514176152707257.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -37.300941721569444 (T) = (0 -23103.85526295302) / Math.Sqrt((13912.968937705908 / (32)) + (22200.238925190493 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.07059520021902937 = (21580.383751231355 - 23103.85526295302) / 21580.383751231355 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;HashSet&lt;String&gt;&gt;.SerializeToWriter(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.466303422619045 > 21.300966394992304.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -33.810774540228486 (T) = (0 -21659.682208639835) / Math.Sqrt((14823.52814222481 / (32)) + (19193.498917316956 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.06501910129613642 = (20337.36501277755 - 21659.682208639835) / 20337.36501277755 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;HashSet&lt;String&gt;&gt;.SerializeToStream(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.670422570088043 > 23.021876799246705.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -58.51983784382132 (T) = (0 -23608.57830955342) / Math.Sqrt((7870.487074676849 / (32)) + (9812.933299862832 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.07494441676457213 = (21962.603778725454 - 23608.57830955342) / 21962.603778725454 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;HashSet&lt;String&gt;&gt;.SerializeToStream(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.11029728107345 > 21.582137599070013.
IsChangePoint: Marked as a change because one of 12/5/2022 3:14:07 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -52.614393281316985 (T) = (0 -22047.00121386997) / Math.Sqrt((8985.163497200292 / (32)) + (7962.303734669277 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.0685757416160883 = (20632.137110399508 - 22047.00121386997) / 20632.137110399508 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.IO.Compression.Deflate

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Compress - Duration of single invocation 243.43 μs 301.68 μs 1.24 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Compression.Deflate*'

Payloads

Baseline
Compare

Histogram

System.IO.Compression.Deflate.Compress(level: Fastest, file: "sum")


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 301.6836044871795 > 255.74273451923077.
IsChangePoint: Marked as a change because one of 10/3/2022 11:00:05 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -57.257931939497546 (T) = (0 -301009.2306571983) / Math.Sqrt((5091060.134006686 / (32)) + (15364924.313884027 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.23674239806537403 = (243388.78583613253 - 301009.2306571983) / 243388.78583613253 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.WriteJson<Hashtable>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToStream - Duration of single invocation 46.48 μs 49.70 μs 1.07 0.01 False
SerializeToUtf8Bytes - Duration of single invocation 46.93 μs 50.39 μs 1.07 0.01 False
SerializeToWriter - Duration of single invocation 46.51 μs 48.97 μs 1.05 0.01 False
SerializeToStream - Duration of single invocation 46.82 μs 49.97 μs 1.07 0.00 False
SerializeToWriter - Duration of single invocation 46.33 μs 49.19 μs 1.06 0.01 False
SerializeToUtf8Bytes - Duration of single invocation 46.75 μs 49.50 μs 1.06 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;Hashtable&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<Hashtable>.SerializeToStream(Mode: Reflection)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 49.70172786777315 > 48.92493422159432.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -50.79675549963927 (T) = (0 -49720.31266448799) / Math.Sqrt((58569.410913883155 / (32)) + (29692.2718224499 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.06413263239997215 = (46723.79283431257 - 49720.31266448799) / 46723.79283431257 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;Hashtable&gt;.SerializeToUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 50.3910282824223 > 49.43088350036836.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -42.78220948810945 (T) = (0 -49995.85867413949) / Math.Sqrt((21895.38625192418 / (32)) + (75874.68201330835 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.06371316688899059 = (47001.25957861446 - 49995.85867413949) / 47001.25957861446 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;Hashtable&gt;.SerializeToWriter(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 48.97167736789073 > 48.37214084290158.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -47.75304606116451 (T) = (0 -49163.30867887841) / Math.Sqrt((89583.52499200152 / (32)) + (17167.5280794242 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.0632713446405469 = (46237.78203624845 - 49163.30867887841) / 46237.78203624845 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;Hashtable&gt;.SerializeToStream(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 49.967329130534004 > 49.048492135583345.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -48.06702490721386 (T) = (0 -49992.820067675784) / Math.Sqrt((124543.38195887156 / (32)) + (11345.692334281033 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.06912712449322103 = (46760.40755337956 - 49992.820067675784) / 46760.40755337956 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;Hashtable&gt;.SerializeToWriter(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 49.192429556878295 > 48.548468041335134.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -29.78794517602734 (T) = (0 -49497.785081372705) / Math.Sqrt((214331.97494417 / (32)) + (72336.76017085396 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.06643832791347169 = (46414.10926988817 - 49497.785081372705) / 46414.10926988817 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;Hashtable&gt;.SerializeToUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 49.50157211234177 > 49.05684516832343.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -43.72621000953317 (T) = (0 -49681.30022004978) / Math.Sqrt((46308.28939034832 / (32)) + (55032.709059133136 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.06277916525590523 = (46746.58842045241 - 49681.30022004978) / 46746.58842045241 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.WriteJson<ArrayList>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToUtf8Bytes - Duration of single invocation 25.32 μs 26.88 μs 1.06 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;ArrayList&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<ArrayList>.SerializeToUtf8Bytes(Mode: Reflection)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 26.884430372995418 > 26.57197619440857.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -7.815749094596565 (T) = (0 -26923.270713271013) / Math.Sqrt((1043733.3107062455 / (32)) + (12003.89355571311 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.05592291751614322 = (25497.38268452669 - 26923.270713271013) / 25497.38268452669 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.WriteJson<SimpleStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToString - Duration of single invocation 594.85 ns 638.81 ns 1.07 0.01 False
SerializeToString - Duration of single invocation 481.18 ns 525.69 ns 1.09 0.01 False
SerializeToWriter - Duration of single invocation 245.51 ns 258.10 ns 1.05 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;SimpleStructWithProperties&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<SimpleStructWithProperties>.SerializeToString(Mode: Reflection)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 638.8124421296295 > 625.5727349610523.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -19.485085294322392 (T) = (0 -639.386563033632) / Math.Sqrt((23.097168316469585 / (32)) + (72.74152008806084 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.07124723125484733 = (596.8618115210077 - 639.386563033632) / 596.8618115210077 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;SimpleStructWithProperties&gt;.SerializeToString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 525.6901811655168 > 509.35095050074.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -54.802906309549385 (T) = (0 -522.6348063115538) / Math.Sqrt((4.512418401362847 / (32)) + (6.843588374040357 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.08190323635039103 = (483.0698243167936 - 522.6348063115538) / 483.0698243167936 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;SimpleStructWithProperties&gt;.SerializeToWriter(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 258.0963235661159 > 256.42636459992553.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -27.695692904090166 (T) = (0 -260.01154845654264) / Math.Sqrt((2.348402895200715 / (32)) + (3.935486877067521 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.06107690678111887 = (245.04496026146987 - 260.01154845654264) / 245.04496026146987 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.WriteJson<ImmutableDictionary<String, String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToUtf8Bytes - Duration of single invocation 48.38 μs 51.27 μs 1.06 0.01 False
SerializeToStream - Duration of single invocation 52.79 μs 55.48 μs 1.05 0.00 False
SerializeToWriter - Duration of single invocation 47.07 μs 50.83 μs 1.08 0.01 False
SerializeToStream - Duration of single invocation 47.83 μs 51.07 μs 1.07 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<ImmutableDictionary<String, String>>.SerializeToUtf8Bytes(Mode: SourceGen)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 51.27312737704918 > 50.79655101739112.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -53.77218559419003 (T) = (0 -51241.20517576477) / Math.Sqrt((44452.25035075539 / (32)) + (26333.79525594033 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.05937043182658445 = (48369.487797968664 - 51241.20517576477) / 48369.487797968664 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.SerializeToStream(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 55.47882582344517 > 55.27044142834596.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -48.66520902630139 (T) = (0 -55776.092793517404) / Math.Sqrt((52383.57858714056 / (32)) + (50113.323803391606 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.061587040677713804 = (52540.2917107109 - 55776.092793517404) / 52540.2917107109 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.SerializeToWriter(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 50.83218599369381 > 49.56745644427314.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -42.25131086867199 (T) = (0 -50679.429206170695) / Math.Sqrt((68873.75961925043 / (32)) + (65544.94196715008 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.06775773008270503 = (47463.41588390583 - 50679.429206170695) / 47463.41588390583 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.SerializeToStream(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 51.07251237785016 > 50.21897424347747.
IsChangePoint: Marked as a change because one of 12/5/2022 3:14:07 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -51.13245469923749 (T) = (0 -51077.05288699844) / Math.Sqrt((66777.62851061621 / (32)) + (22003.157312873886 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.06110706550937817 = (48135.626033626686 - 51077.05288699844) / 48135.626033626686 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions