.NET 7.0 Preview 4 Microbenchmarks Performance Study Report

[dakersnar]

Data

This time we have covered following configs:

• architectures: x64, x86, arm64
• Unix: Ubuntu 18.04, Ubuntu 20.04, pop 20.04, Alpine 3.13, macOS Monterey
• Windows: 10 & 11

Operating System	Bit	Processor Name
ubuntu 20.04	Arm64	Unknown processor
Windows 10	Arm64	Microsoft SQ1 3.0 GHz
Windows 11	Arm64	Microsoft SQ1 3.0 GHz
Windows 11	Arm64	Unknown processor
macOS Monterey 12.3	Arm64	Apple M1 Max
Windows 10	X64	Intel Xeon Platinum 8272CL CPU 2.60GHz
Windows 10	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
Windows 10	X64	Intel Core i7-6700 CPU 3.40GHz (Skylake)
Windows 11	X64	AMD Ryzen Threadripper PRO 3945WX 12-Cores
Windows 11	X64	AMD Ryzen 9 5950X
Windows 11	X64	Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Windows 11	X64	Intel Core i9-9900T CPU 2.10GHz
alpine 3.13	X64	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
pop 22.04	X64	Intel Core i7-6600U CPU 2.60GHz (Skylake)
ubuntu 18.04	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
ubuntu 18.04	X64	Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
ubuntu 20.04	X64	AMD Ryzen 9 5900X
ubuntu 20.04	X64	Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Windows 10	X86	Intel Xeon CPU E5-1650 v4 3.60GHz
Windows 10	X86	Intel Core i7-6700 CPU 3.40GHz (Skylake)
Windows 11	X86	AMD Ryzen Threadripper PRO 3945WX 12-Cores
macOS Monterey 12.2.1	X64	Intel Core i7-5557U CPU 3.10GHz (Broadwell)
macOS Monterey 12.3.1	X64	Intel Core i7-4870HQ CPU 2.50GHz (Haswell)

Most of the benchmarks were run on bare-metal machines, some were executed via WSL.

This would not be possible without the help from: @adamsitnik @AndyAyersMS @carlossanlop @danmoseley @jeffhandley @kunalspathak @mrsharm and @tannergooding who contributed their results and time.

The full report generated by the tool is available here. The full report contains improvements, so if you read it from the end you can see the biggest perf improvements. There are plenty of them!

Again, the full historical data turned out to be extremely useful. For details about methodology please read #41871. Preview3 report can be found here.

Notable Improvements

• System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, OrdinalIgnoreCase, False)), System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "x", comparisonType: OrdinalIgnoreCase), System.Memory.ReadOnlySpan.IndexOfString(input: "Hello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello Worldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylong!xyz", value: "w", comparisonType: OrdinalIgnoreCase), etc
  • caused by: Vectorize IndexOf for OrdinalIgnoreCase #67758
  • added comment: Vectorize IndexOf for OrdinalIgnoreCase #67758 (comment)
• System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "[a-z]shing", Options: None), and others
  • caused by: Add RegexFindOptimization for embedded strings #67907
  • added comment: Add RegexFindOptimization for embedded strings #67907 (comment)
• Exceptions.Handling.ThrowAndCatch*
  • caused by: Reflection Invoke refactoring to support future byref-like and IL emit features #66357
  • added comment: Reflection Invoke refactoring to support future byref-like and IL emit features #66357 (comment)
• System.Text.RegularExpressions.Tests.Perf_Regex_Common.OneNodeBacktracking(Options: IgnoreCase, Compiled)
  • caused by: Changing the logic for how we deal with RegexOptions.IgnoreCase matching. #67184
  • added comment: Changing the logic for how we deal with RegexOptions.IgnoreCase matching. #67184 (comment)

Regressions

By design

• System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Ctor(*, options: NonBacktracking)
  • already reported by the bot: Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas #67915
  • explained: Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas #67915 (comment)

Investigation in progress

• FractalPerf.Launch.Test
  • detected in [Perf] Changes at 3/30/2022 8:03:41 PM perf-autofiling-issues#4414, but not solved
  • the group of benchmark regressions in the above were addressed in Regressions due to QJFL and OSR  #67594
  • comment explanation: Regressions due to QJFL and OSR  #67594 (comment)
• System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldUsingBitConverterBE, System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldUsingBitConverterLE
  • being investigated here: Regressions in System.Buffers.Binary.Tests.BinaryReadAndWriteTests #68338
  • comment added: Regressions in System.Buffers.Binary.Tests.BinaryReadAndWriteTests #68338 (comment)
• Benchstone.BenchI.Array2.Test, SciMark2.kernel.benchSparseMult, System.Collections.CopyTo<Int32>.ReadOnlySpan(Size: 2048), LinqBenchmarks.Count00ForX, System.Collections.CopyTo<Int32>.Memory(Size: 2048)
  • likely caused by OSR: Enable QJFL and OSR by default for x64 and arm64 #65675
  • added a comment: Regressions due to QJFL and OSR  #67594 (comment)
• System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Tom|Sawyer|Huckleberry|Finn", Options: NonBacktracking), System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "([A-Za-z]awyer|[A-Za-z]inn)\s", Options: NonBacktracking), System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Twain", Options: NonBacktracking), System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "Tom.{10,25}river|river.{10,25}Tom", Options: NonBacktracking)
  • regressions detected: Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig #67913
  • added a comment: Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig #67913 (comment)
• System.Collections.ContainsTrue<Int32>.Span(Size: 512)
  • caused by this (PGO update): [main] Update dependencies from dnceng/internal/dotnet-optimization #68109
  • added a comment: [main] Update dependencies from dnceng/internal/dotnet-optimization #68109 (comment)
• System.Collections.IterateForEach<Int32>.List(Size: 512)
  • detected here: [Perf] Changes at 4/16/2022 10:12:41 PM perf-autofiling-issues#4874
  • added comment: [Perf] Changes at 4/16/2022 10:12:41 PM perf-autofiling-issues#4874 (comment)
  • Probably caused by a PGO update
• System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>.NewCustomConverter
  • possibly caused by: Split Reflection and SourceGen TypeInfos #67526
  • added comment: Split Reflection and SourceGen TypeInfos #67526 (comment)

Already solved

• System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(*, Options: NonBacktracking)
  • already solved: Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock #67912 (comment)
• System.Net.Security.Tests.SslStreamTests.HandshakeRSA2048CertAsync(protocol: Tls12), System.Net.Security.Tests.SslStreamTests.HandshakeECDSA256CertAsync(protocol: Tls12), System.Net.Security.Tests.SslStreamTests.HandshakeRSA4096CertAsync(protocol: Tls12), System.Net.Security.Tests.SslStreamTests.HandshakeECDSA256CertAsync(protocol: Tls13), System.Net.Security.Tests.SslStreamTests.HandshakeRSA2048CertAsync(protocol: Tls13)
  • already detected: Regressions in System.Net.Security.Tests.SslStreamTests #68408
  • fixed by Update expired certificates for HTTPS/TLS benchmarks performance#2405
  • added comment: Regressions in System.Net.Security.Tests.SslStreamTests #68408 (comment)
• System.Text.Json.Tests.Perf_Get.GetBoolean
  • added a comment: Add JSON type hierarchies #67961 (comment)
  • first comment was slightly inaccurate, added a second clarifying comment: Add JSON type hierarchies #67961 (comment)
  • regression seems to be solved looking at historical data
• System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 2, Options: None)

Noise, flaky or multimodal

The following benchmarks showed up in the report generated by the tool, but were not actual regressions:

• System.Numerics.Tests.Perf_VectorOf<*>.*
• System.Numerics.Tests.Perf_Matrix4x4.IdentityBenchmark
• System.Numerics.Tests.Perf_Matrix4x4.TranslationBenchmark
• System.Numerics.Tests.Perf_Vector3.EqualityOperatorBenchmark
• System.Numerics.Tests.Perf_Vector4.AddOperatorBenchmark
• System.Numerics.Tests.Perf_Vector4.AddFunctionBenchmark
• System.Numerics.Tests.Perf_Vector4.ClampBenchmark
• System.Numerics.Tests.Perf_Vector4.InequalityOperatorBenchmark
• Noisy results on one mac machine:
  • System.Numerics.Tests.Constructor.SpanCastBenchmark_Single
  • System.Numerics.Tests.Constructor.ConstructorBenchmark_Single

Noisy benchmarks from previous months:

• System.Collections.Concurrent.Count<String>.Dictionary(Size: 512)
• System.Collections.TryAddDefaultSize<String>.ConcurrentDictionary(Count: 512)
  • whether it is slower or faster correlates with the alloc delta. Very noisy.
• PerfLabTests.CastingPerf.CheckArrayIsInterfaceNo

Statistics

• Statistical Test threshold: 10%, the noise filter: 2 ns
• Result is conclusion: Slower|Faster|Same|Noise|Unknown. Noise means that the difference was larger than 10% but not 2 ns.

Total: 104158
Same: 82.88 %
Slower: 3.43 %
Faster: 4.97 %
Noise: 8.67 %
Unknown: 0.05 %

Statistics per Architecture

Architecture	Same	Slower	Faster	Noise	Unknown
Arm64	82.56 %	3.76 %	4.53 %	9.06 %	0.08 %
X64	82.45 %	3.47 %	5.02 %	9.01 %	0.05 %
X86	85.56 %	2.65 %	5.42 %	6.33 %	0.04 %

Statistics per Operating System

Operating System	Same	Slower	Faster	Noise	Unknown
macOS Monterey 12.3	87.53 %	4.08 %	3.90 %	4.35 %	0.13 %
ubuntu 20.04	81.56 %	3.66 %	5.38 %	9.34 %	0.05 %
Windows 10	83.43 %	2.84 %	5.28 %	8.42 %	0.04 %
Windows 11	82.52 %	3.18 %	4.84 %	9.41 %	0.06 %
alpine 3.13	84.57 %	3.55 %	4.86 %	7.00 %	0.02 %
macOS Monterey 12.2.1	83.08 %	3.38 %	4.16 %	9.31 %	0.07 %
macOS Monterey 12.3.1	83.36 %	2.37 %	4.20 %	10.04 %	0.02 %
pop 22.04	79.47 %	2.52 %	6.95 %	11.06 %	0.00 %
ubuntu 18.04	82.67 %	6.36 %	4.25 %	6.62 %	0.10 %

Statistics per Namespace

Namespace	Same	Slower	Faster	Noise	Unknown
BenchmarksGame	89.65 %	1.45 %	8.90 %	0.00 %	0.00 %
Benchstone.BenchF	94.57 %	4.57 %	0.87 %	0.00 %	0.00 %
Benchstone.BenchI	92.57 %	5.80 %	1.63 %	0.00 %	0.00 %
Benchstone.MDBenchF	99.13 %	0.00 %	0.87 %	0.00 %	0.00 %
Benchstone.MDBenchI	96.52 %	0.00 %	3.48 %	0.00 %	0.00 %
Devirtualization	88.70 %	0.00 %	0.00 %	11.30 %	0.00 %
Exceptions	82.49 %	4.71 %	12.80 %	0.00 %	0.00 %
FractalPerf	13.04 %	86.96 %	0.00 %	0.00 %	0.00 %
GuardedDevirtualization	91.51 %	0.21 %	0.00 %	8.28 %	0.00 %
Inlining	71.01 %	0.00 %	0.00 %	28.99 %	0.00 %
Interop	92.75 %	7.25 %	0.00 %	0.00 %	0.00 %
JetStream	54.35 %	0.00 %	45.65 %	0.00 %	0.00 %
Layout	100.00 %	0.00 %	0.00 %	0.00 %	0.00 %
Lowering	100.00 %	0.00 %	0.00 %	0.00 %	0.00 %
MicroBenchmarks.Serializers	98.61 %	0.80 %	0.60 %	0.00 %	0.00 %
Microsoft.AspNetCore.Server.Kestrel.Performance	91.85 %	4.35 %	3.80 %	0.00 %	0.00 %
Microsoft.Extensions.Caching.Memory.Tests	81.82 %	9.49 %	8.70 %	0.00 %	0.00 %
Microsoft.Extensions.Configuration.Xml	97.83 %	0.00 %	2.17 %	0.00 %	0.00 %
Microsoft.Extensions.DependencyInjection	92.15 %	2.66 %	4.35 %	0.84 %	0.00 %
Microsoft.Extensions.Logging	94.29 %	2.48 %	2.36 %	0.87 %	0.00 %
Microsoft.Extensions.Primitives.Performance	61.96 %	0.54 %	0.00 %	37.50 %	0.00 %
Microsoft.Extensions.Primitives	80.92 %	1.21 %	0.00 %	17.87 %	0.00 %
PerfLabTests	69.29 %	8.48 %	17.72 %	4.51 %	0.00 %
PerfLabTests.CastingPerf2	84.64 %	11.88 %	3.48 %	0.00 %	0.00 %
SciMark2	89.57 %	9.57 %	0.87 %	0.00 %	0.00 %
SIMD	99.52 %	0.48 %	0.00 %	0.00 %	0.00 %
Span	96.52 %	0.70 %	2.78 %	0.00 %	0.00 %
StoreBlock	80.29 %	0.43 %	1.30 %	17.97 %	0.00 %
System.Buffers.Binary.Tests	60.87 %	30.43 %	3.26 %	5.43 %	0.00 %
System.Buffers.Tests	91.42 %	2.40 %	5.58 %	0.60 %	0.00 %
System.Buffers.Text.Tests	87.47 %	0.96 %	2.56 %	9.02 %	0.00 %
System.Collections	92.06 %	4.09 %	3.47 %	0.39 %	0.00 %
System.Collections.Concurrent	89.12 %	1.97 %	2.31 %	6.60 %	0.00 %
System.Collections.Tests	90.82 %	3.82 %	2.48 %	2.89 %	0.00 %
System.ComponentModel.Tests	95.65 %	1.19 %	3.16 %	0.00 %	0.00 %
System.ConsoleTests	95.65 %	2.90 %	1.45 %	0.00 %	0.00 %
System.Diagnostics	65.22 %	6.09 %	28.70 %	0.00 %	0.00 %
System.Diagnostics.Tracing	97.83 %	2.17 %	0.00 %	0.00 %	0.00 %
System.Formats.Cbor.Tests	98.66 %	0.33 %	1.00 %	0.00 %	0.00 %
System.Globalization.Tests	92.43 %	2.09 %	4.02 %	1.46 %	0.00 %
System	95.41 %	0.00 %	2.66 %	1.93 %	0.00 %
System.IO.Compression	99.09 %	0.48 %	0.42 %	0.00 %	0.00 %
System.IO.MemoryMappedFiles.Tests	87.50 %	5.15 %	7.35 %	0.00 %	0.00 %
System.IO.Pipelines.Tests	96.74 %	1.09 %	2.17 %	0.00 %	0.00 %
System.IO.Pipes.Tests	94.20 %	2.90 %	2.90 %	0.00 %	0.00 %
System.IO.Tests	89.08 %	4.10 %	5.57 %	1.26 %	0.00 %
System.Linq.Tests	93.54 %	3.66 %	2.45 %	0.34 %	0.00 %
System.MathBenchmarks	96.21 %	0.91 %	2.10 %	0.77 %	0.00 %
System.Memory	80.42 %	1.90 %	5.77 %	11.90 %	0.00 %
System.Net.Http.Tests	88.59 %	3.08 %	8.33 %	0.00 %	0.00 %
System.Net.NetworkInformation.Tests	66.30 %	7.61 %	26.09 %	0.00 %	0.00 %
System.Net.Primitives.Tests	87.55 %	2.57 %	2.96 %	6.92 %	0.00 %
System.Net.Security.Tests	61.90 %	27.17 %	10.92 %	0.00 %	0.00 %
System.Net.Sockets.Tests	89.29 %	4.46 %	6.25 %	0.00 %	0.00 %
System.Net.Tests	97.08 %	1.46 %	1.46 %	0.00 %	0.00 %
System.Net.WebSockets.Tests	76.09 %	8.70 %	15.22 %	0.00 %	0.00 %
System.Numerics.Tests	59.65 %	2.17 %	2.33 %	35.85 %	0.00 %
System.Reflection	91.01 %	2.03 %	5.36 %	1.59 %	0.00 %
System.Runtime.InteropServices.Tests	93.48 %	0.00 %	4.35 %	2.17 %	0.00 %
System.Runtime.Serialization.Formatters.Tests	100.00 %	0.00 %	0.00 %	0.00 %	0.00 %
System.Security.Cryptography.Primitives.Tests.Performance	91.30 %	3.80 %	4.89 %	0.00 %	0.00 %
System.Security.Cryptography.Tests	93.24 %	0.00 %	6.76 %	0.00 %	0.00 %
System.Security.Cryptography.X509Certificates.Tests	91.30 %	0.00 %	8.70 %	0.00 %	0.00 %
System.Tests	87.76 %	2.68 %	2.94 %	6.63 %	0.00 %
System.Text.Encodings.Web.Tests	90.04 %	5.62 %	4.35 %	0.00 %	0.00 %
System.Text.Json.Document.Tests	98.17 %	0.17 %	1.65 %	0.00 %	0.00 %
System.Text.Json.Reader.Tests	79.35 %	7.61 %	13.04 %	0.00 %	0.00 %
System.Text.Json.Serialization.Tests	97.03 %	1.45 %	1.53 %	0.00 %	0.00 %
System.Text.Json.Tests	96.73 %	1.42 %	1.73 %	0.11 %	0.00 %
System.Text	86.38 %	12.46 %	1.16 %	0.00 %	0.00 %
System.Text.RegularExpressions.Tests	62.32 %	10.96 %	25.87 %	0.00 %	0.86 %
System.Text.Tests	91.20 %	4.66 %	2.80 %	1.35 %	0.00 %
System.Threading.Channels.Tests	92.75 %	3.99 %	2.90 %	0.36 %	0.00 %
System.Threading.Tasks.Tests	86.96 %	10.14 %	1.45 %	1.45 %	0.00 %
System.Threading.Tasks	88.51 %	4.35 %	6.52 %	0.62 %	0.00 %
System.Threading.Tests	89.64 %	2.31 %	2.59 %	5.46 %	0.00 %
System.Xml.Linq	80.32 %	4.58 %	10.07 %	5.03 %	0.00 %
System.Xml.Tests	99.13 %	0.00 %	0.87 %	0.00 %	0.00 %
V8.Crypto	95.65 %	4.35 %	0.00 %	0.00 %	0.00 %
V8.Richards	100.00 %	0.00 %	0.00 %	0.00 %	0.00 %
XmlDocumentTests.XmlDocumentTests	88.41 %	4.35 %	1.45 %	5.80 %	0.00 %
XmlDocumentTests.XmlNodeListTests	86.96 %	6.52 %	0.00 %	6.52 %	0.00 %
XmlDocumentTests.XmlNodeTests	47.83 %	0.00 %	0.00 %	52.17 %	0.00 %
System.Drawing.Tests	91.15 %	2.69 %	1.15 %	5.00 %	0.00 %
HardwareIntrinsics.RayTracer	83.33 %	5.56 %	5.56 %	5.56 %	0.00 %

Big thanks to everyone involved!

Tagging subscribers to this area: @dotnet/area-meta
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Data

This time we have covered following configs:

• architectures: x64, x86, arm64
• Unix: Ubuntu 18.04, Ubuntu 20.04, pop 20.04, Alpine 3.13, macOS Monterey
• Windows: 10 & 11

Operating System	Bit	Processor Name
ubuntu 20.04	Arm64	Unknown processor
Windows 10	Arm64	Microsoft SQ1 3.0 GHz
Windows 11	Arm64	Microsoft SQ1 3.0 GHz
Windows 11	Arm64	Unknown processor
macOS Monterey 12.3	Arm64	Apple M1 Max
Windows 10	X64	Intel Xeon Platinum 8272CL CPU 2.60GHz
Windows 10	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
Windows 10	X64	Intel Core i7-6700 CPU 3.40GHz (Skylake)
Windows 11	X64	AMD Ryzen Threadripper PRO 3945WX 12-Cores
Windows 11	X64	AMD Ryzen 9 5950X
Windows 11	X64	Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Windows 11	X64	Intel Core i9-9900T CPU 2.10GHz
alpine 3.13	X64	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
pop 22.04	X64	Intel Core i7-6600U CPU 2.60GHz (Skylake)
ubuntu 18.04	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
ubuntu 18.04	X64	Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
ubuntu 20.04	X64	AMD Ryzen 9 5900X
ubuntu 20.04	X64	Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Windows 10	X86	Intel Xeon CPU E5-1650 v4 3.60GHz
Windows 10	X86	Intel Core i7-6700 CPU 3.40GHz (Skylake)
Windows 11	X86	AMD Ryzen Threadripper PRO 3945WX 12-Cores
macOS Monterey 12.2.1	X64	Intel Core i7-5557U CPU 3.10GHz (Broadwell)
macOS Monterey 12.3.1	X64	Intel Core i7-4870HQ CPU 2.50GHz (Haswell)

Most of the benchmarks were run on bare-metal machines, some were executed via WSL.

This would not be possible without the help from: @adsitnik @AndyAyersMS @carlossanlop @danmoseley @jeffhandley @kunalspathak @mrsharm and @tannergooding who contributed their results and time.

The full report generated by the tool is available here. The full report contains improvements, so if you read it from the end you can see the biggest perf improvements. There are plenty of them!

Again, the full historical data turned out to be extremely useful. For details about methodology please read #41871. Preview3 report can be found here.

Notable Improvements

• System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, OrdinalIgnoreCase, False)), System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "x", comparisonType: OrdinalIgnoreCase), System.Memory.ReadOnlySpan.IndexOfString(input: "Hello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello Worldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylong!xyz", value: "w", comparisonType: OrdinalIgnoreCase), etc
  • caused by: Vectorize IndexOf for OrdinalIgnoreCase #67758
  • added comment: Vectorize IndexOf for OrdinalIgnoreCase #67758 (comment)
• System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "[a-z]shing", Options: None), and others
  • caused by: Add RegexFindOptimization for embedded strings #67907
  • added comment: Add RegexFindOptimization for embedded strings #67907 (comment)
• Exceptions.Handling.ThrowAndCatch*
  • caused by: Reflection Invoke refactoring to support future byref-like and IL emit features #66357
  • added comment: Reflection Invoke refactoring to support future byref-like and IL emit features #66357 (comment)
• System.Text.RegularExpressions.Tests.Perf_Regex_Common.OneNodeBacktracking(Options: IgnoreCase, Compiled)
  • caused by: Changing the logic for how we deal with RegexOptions.IgnoreCase matching. #67184
  • added comment: Changing the logic for how we deal with RegexOptions.IgnoreCase matching. #67184 (comment)

Regressions

By design

• System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Ctor(*, options: NonBacktracking)
  • already reported by the bot: Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas #67915
  • explained: Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas #67915 (comment)

Investigation in progress

• FractalPerf.Launch.Test
  • detected in [Perf] Changes at 3/30/2022 8:03:41 PM perf-autofiling-issues#4414, but not solved
  • the group of benchmark regressions in the above were addressed in Regressions due to QJFL and OSR  #67594
  • comment explanation: Regressions due to QJFL and OSR  #67594 (comment)
• System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldUsingBitConverterBE, System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldUsingBitConverterLE
  • being investigated here: Regressions in System.Buffers.Binary.Tests.BinaryReadAndWriteTests #68338
  • comment added: Regressions in System.Buffers.Binary.Tests.BinaryReadAndWriteTests #68338 (comment)
• Benchstone.BenchI.Array2.Test, SciMark2.kernel.benchSparseMult, System.Collections.CopyTo<Int32>.ReadOnlySpan(Size: 2048), LinqBenchmarks.Count00ForX, System.Collections.CopyTo<Int32>.Memory(Size: 2048)
  • likely caused by OSR: Enable QJFL and OSR by default for x64 and arm64 #65675
  • added a comment: Regressions due to QJFL and OSR  #67594 (comment)
• System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Tom|Sawyer|Huckleberry|Finn", Options: NonBacktracking), System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "([A-Za-z]awyer|[A-Za-z]inn)\s", Options: NonBacktracking), System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Twain", Options: NonBacktracking), System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "Tom.{10,25}river|river.{10,25}Tom", Options: NonBacktracking)
  • regressions detected: Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig #67913
  • added a comment: Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig #67913 (comment)
• System.Collections.ContainsTrue<Int32>.Span(Size: 512)
  • caused by this (PGO update): [main] Update dependencies from dnceng/internal/dotnet-optimization #68109
  • added a comment: [main] Update dependencies from dnceng/internal/dotnet-optimization #68109 (comment)
• System.Collections.IterateForEach<Int32>.List(Size: 512)
  • detected here: [Perf] Changes at 4/16/2022 10:12:41 PM perf-autofiling-issues#4874
  • added comment: [Perf] Changes at 4/16/2022 10:12:41 PM perf-autofiling-issues#4874 (comment)
  • Probably caused by a PGO update
• System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>.NewCustomConverter
  • possibly caused by: Split Reflection and SourceGen TypeInfos #67526
  • added comment: Split Reflection and SourceGen TypeInfos #67526 (comment)

Already solved

• System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(*, Options: NonBacktracking)
  • already solved: Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock #67912 (comment)
• System.Net.Security.Tests.SslStreamTests.HandshakeRSA2048CertAsync(protocol: Tls12), System.Net.Security.Tests.SslStreamTests.HandshakeECDSA256CertAsync(protocol: Tls12), System.Net.Security.Tests.SslStreamTests.HandshakeRSA4096CertAsync(protocol: Tls12), System.Net.Security.Tests.SslStreamTests.HandshakeECDSA256CertAsync(protocol: Tls13), System.Net.Security.Tests.SslStreamTests.HandshakeRSA2048CertAsync(protocol: Tls13)
  • already detected: Regressions in System.Net.Security.Tests.SslStreamTests #68408
  • fixed by Update expired certificates for HTTPS/TLS benchmarks performance#2405
  • added comment: Regressions in System.Net.Security.Tests.SslStreamTests #68408 (comment)
• System.Text.Json.Tests.Perf_Get.GetBoolean
  • added a comment: Add JSON type hierarchies #67961 (comment)
  • first comment was slightly inaccurate, added a second clarifying comment: Add JSON type hierarchies #67961 (comment)
  • regression seems to be solved looking at historical data
• System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 2, Options: None)

Noise, flaky or multimodal

The following benchmarks showed up in the report generated by the tool, but were not actual regressions:

• System.Numerics.Tests.Perf_VectorOf<*>.*
• System.Numerics.Tests.Perf_Matrix4x4.IdentityBenchmark
• System.Numerics.Tests.Perf_Matrix4x4.TranslationBenchmark
• System.Numerics.Tests.Perf_Vector3.EqualityOperatorBenchmark
• System.Numerics.Tests.Perf_Vector4.AddOperatorBenchmark
• System.Numerics.Tests.Perf_Vector4.AddFunctionBenchmark
• System.Numerics.Tests.Perf_Vector4.ClampBenchmark
• System.Numerics.Tests.Perf_Vector4.InequalityOperatorBenchmark
• Noisy results on one mac machine:
  • System.Numerics.Tests.Constructor.SpanCastBenchmark_Single
  • System.Numerics.Tests.Constructor.ConstructorBenchmark_Single

Noisy benchmarks from previous months:

• System.Collections.Concurrent.Count<String>.Dictionary(Size: 512)
• System.Collections.TryAddDefaultSize<String>.ConcurrentDictionary(Count: 512)
  • whether it is slower or faster correlates with the alloc delta. Very noisy.
• PerfLabTests.CastingPerf.CheckArrayIsInterfaceNo

Statistics

• Statistical Test threshold: 10%, the noise filter: 2 ns
• Result is conclusion: Slower|Faster|Same|Noise|Unknown. Noise means that the difference was larger than 10% but not 2 ns.

Total: 104158
Same: 82.88 %
Slower: 3.43 %
Faster: 4.97 %
Noise: 8.67 %
Unknown: 0.05 %

Statistics per Architecture

Architecture	Same	Slower	Faster	Noise	Unknown
Arm64	82.56 %	3.76 %	4.53 %	9.06 %	0.08 %
X64	82.45 %	3.47 %	5.02 %	9.01 %	0.05 %
X86	85.56 %	2.65 %	5.42 %	6.33 %	0.04 %

Statistics per Operating System

Operating System	Same	Slower	Faster	Noise	Unknown
macOS Monterey 12.3	87.53 %	4.08 %	3.90 %	4.35 %	0.13 %
ubuntu 20.04	81.56 %	3.66 %	5.38 %	9.34 %	0.05 %
Windows 10	83.43 %	2.84 %	5.28 %	8.42 %	0.04 %
Windows 11	82.52 %	3.18 %	4.84 %	9.41 %	0.06 %
alpine 3.13	84.57 %	3.55 %	4.86 %	7.00 %	0.02 %
macOS Monterey 12.2.1	83.08 %	3.38 %	4.16 %	9.31 %	0.07 %
macOS Monterey 12.3.1	83.36 %	2.37 %	4.20 %	10.04 %	0.02 %
pop 22.04	79.47 %	2.52 %	6.95 %	11.06 %	0.00 %
ubuntu 18.04	82.67 %	6.36 %	4.25 %	6.62 %	0.10 %

Statistics per Namespace

Namespace	Same	Slower	Faster	Noise	Unknown
BenchmarksGame	89.65 %	1.45 %	8.90 %	0.00 %	0.00 %
Benchstone.BenchF	94.57 %	4.57 %	0.87 %	0.00 %	0.00 %
Benchstone.BenchI	92.57 %	5.80 %	1.63 %	0.00 %	0.00 %
Benchstone.MDBenchF	99.13 %	0.00 %	0.87 %	0.00 %	0.00 %
Benchstone.MDBenchI	96.52 %	0.00 %	3.48 %	0.00 %	0.00 %
Devirtualization	88.70 %	0.00 %	0.00 %	11.30 %	0.00 %
Exceptions	82.49 %	4.71 %	12.80 %	0.00 %	0.00 %
FractalPerf	13.04 %	86.96 %	0.00 %	0.00 %	0.00 %
GuardedDevirtualization	91.51 %	0.21 %	0.00 %	8.28 %	0.00 %
Inlining	71.01 %	0.00 %	0.00 %	28.99 %	0.00 %
Interop	92.75 %	7.25 %	0.00 %	0.00 %	0.00 %
JetStream	54.35 %	0.00 %	45.65 %	0.00 %	0.00 %
Layout	100.00 %	0.00 %	0.00 %	0.00 %	0.00 %
Lowering	100.00 %	0.00 %	0.00 %	0.00 %	0.00 %
MicroBenchmarks.Serializers	98.61 %	0.80 %	0.60 %	0.00 %	0.00 %
Microsoft.AspNetCore.Server.Kestrel.Performance	91.85 %	4.35 %	3.80 %	0.00 %	0.00 %
Microsoft.Extensions.Caching.Memory.Tests	81.82 %	9.49 %	8.70 %	0.00 %	0.00 %
Microsoft.Extensions.Configuration.Xml	97.83 %	0.00 %	2.17 %	0.00 %	0.00 %
Microsoft.Extensions.DependencyInjection	92.15 %	2.66 %	4.35 %	0.84 %	0.00 %
Microsoft.Extensions.Logging	94.29 %	2.48 %	2.36 %	0.87 %	0.00 %
Microsoft.Extensions.Primitives.Performance	61.96 %	0.54 %	0.00 %	37.50 %	0.00 %
Microsoft.Extensions.Primitives	80.92 %	1.21 %	0.00 %	17.87 %	0.00 %
PerfLabTests	69.29 %	8.48 %	17.72 %	4.51 %	0.00 %
PerfLabTests.CastingPerf2	84.64 %	11.88 %	3.48 %	0.00 %	0.00 %
SciMark2	89.57 %	9.57 %	0.87 %	0.00 %	0.00 %
SIMD	99.52 %	0.48 %	0.00 %	0.00 %	0.00 %
Span	96.52 %	0.70 %	2.78 %	0.00 %	0.00 %
StoreBlock	80.29 %	0.43 %	1.30 %	17.97 %	0.00 %
System.Buffers.Binary.Tests	60.87 %	30.43 %	3.26 %	5.43 %	0.00 %
System.Buffers.Tests	91.42 %	2.40 %	5.58 %	0.60 %	0.00 %
System.Buffers.Text.Tests	87.47 %	0.96 %	2.56 %	9.02 %	0.00 %
System.Collections	92.06 %	4.09 %	3.47 %	0.39 %	0.00 %
System.Collections.Concurrent	89.12 %	1.97 %	2.31 %	6.60 %	0.00 %
System.Collections.Tests	90.82 %	3.82 %	2.48 %	2.89 %	0.00 %
System.ComponentModel.Tests	95.65 %	1.19 %	3.16 %	0.00 %	0.00 %
System.ConsoleTests	95.65 %	2.90 %	1.45 %	0.00 %	0.00 %
System.Diagnostics	65.22 %	6.09 %	28.70 %	0.00 %	0.00 %
System.Diagnostics.Tracing	97.83 %	2.17 %	0.00 %	0.00 %	0.00 %
System.Formats.Cbor.Tests	98.66 %	0.33 %	1.00 %	0.00 %	0.00 %
System.Globalization.Tests	92.43 %	2.09 %	4.02 %	1.46 %	0.00 %
System	95.41 %	0.00 %	2.66 %	1.93 %	0.00 %
System.IO.Compression	99.09 %	0.48 %	0.42 %	0.00 %	0.00 %
System.IO.MemoryMappedFiles.Tests	87.50 %	5.15 %	7.35 %	0.00 %	0.00 %
System.IO.Pipelines.Tests	96.74 %	1.09 %	2.17 %	0.00 %	0.00 %
System.IO.Pipes.Tests	94.20 %	2.90 %	2.90 %	0.00 %	0.00 %
System.IO.Tests	89.08 %	4.10 %	5.57 %	1.26 %	0.00 %
System.Linq.Tests	93.54 %	3.66 %	2.45 %	0.34 %	0.00 %
System.MathBenchmarks	96.21 %	0.91 %	2.10 %	0.77 %	0.00 %
System.Memory	80.42 %	1.90 %	5.77 %	11.90 %	0.00 %
System.Net.Http.Tests	88.59 %	3.08 %	8.33 %	0.00 %	0.00 %
System.Net.NetworkInformation.Tests	66.30 %	7.61 %	26.09 %	0.00 %	0.00 %
System.Net.Primitives.Tests	87.55 %	2.57 %	2.96 %	6.92 %	0.00 %
System.Net.Security.Tests	61.90 %	27.17 %	10.92 %	0.00 %	0.00 %
System.Net.Sockets.Tests	89.29 %	4.46 %	6.25 %	0.00 %	0.00 %
System.Net.Tests	97.08 %	1.46 %	1.46 %	0.00 %	0.00 %
System.Net.WebSockets.Tests	76.09 %	8.70 %	15.22 %	0.00 %	0.00 %
System.Numerics.Tests	59.65 %	2.17 %	2.33 %	35.85 %	0.00 %
System.Reflection	91.01 %	2.03 %	5.36 %	1.59 %	0.00 %
System.Runtime.InteropServices.Tests	93.48 %	0.00 %	4.35 %	2.17 %	0.00 %
System.Runtime.Serialization.Formatters.Tests	100.00 %	0.00 %	0.00 %	0.00 %	0.00 %
System.Security.Cryptography.Primitives.Tests.Performance	91.30 %	3.80 %	4.89 %	0.00 %	0.00 %
System.Security.Cryptography.Tests	93.24 %	0.00 %	6.76 %	0.00 %	0.00 %
System.Security.Cryptography.X509Certificates.Tests	91.30 %	0.00 %	8.70 %	0.00 %	0.00 %
System.Tests	87.76 %	2.68 %	2.94 %	6.63 %	0.00 %
System.Text.Encodings.Web.Tests	90.04 %	5.62 %	4.35 %	0.00 %	0.00 %
System.Text.Json.Document.Tests	98.17 %	0.17 %	1.65 %	0.00 %	0.00 %
System.Text.Json.Reader.Tests	79.35 %	7.61 %	13.04 %	0.00 %	0.00 %
System.Text.Json.Serialization.Tests	97.03 %	1.45 %	1.53 %	0.00 %	0.00 %
System.Text.Json.Tests	96.73 %	1.42 %	1.73 %	0.11 %	0.00 %
System.Text	86.38 %	12.46 %	1.16 %	0.00 %	0.00 %
System.Text.RegularExpressions.Tests	62.32 %	10.96 %	25.87 %	0.00 %	0.86 %
System.Text.Tests	91.20 %	4.66 %	2.80 %	1.35 %	0.00 %
System.Threading.Channels.Tests	92.75 %	3.99 %	2.90 %	0.36 %	0.00 %
System.Threading.Tasks.Tests	86.96 %	10.14 %	1.45 %	1.45 %	0.00 %
System.Threading.Tasks	88.51 %	4.35 %	6.52 %	0.62 %	0.00 %
System.Threading.Tests	89.64 %	2.31 %	2.59 %	5.46 %	0.00 %
System.Xml.Linq	80.32 %	4.58 %	10.07 %	5.03 %	0.00 %
System.Xml.Tests	99.13 %	0.00 %	0.87 %	0.00 %	0.00 %
V8.Crypto	95.65 %	4.35 %	0.00 %	0.00 %	0.00 %
V8.Richards	100.00 %	0.00 %	0.00 %	0.00 %	0.00 %
XmlDocumentTests.XmlDocumentTests	88.41 %	4.35 %	1.45 %	5.80 %	0.00 %
XmlDocumentTests.XmlNodeListTests	86.96 %	6.52 %	0.00 %	6.52 %	0.00 %
XmlDocumentTests.XmlNodeTests	47.83 %	0.00 %	0.00 %	52.17 %	0.00 %
System.Drawing.Tests	91.15 %	2.69 %	1.15 %	5.00 %	0.00 %
HardwareIntrinsics.RayTracer	83.33 %	5.56 %	5.56 %	5.56 %	0.00 %

Big thanks to everyone involved!

Author:	dakersnar
Assignees:	-
Labels:	area-Meta, tenet-performance, tracking
Milestone:	-

[dakersnar]

TODO: ensure all the "in investigation" regressions get addressed/solved. If they do not, we should make standalone issues for instances where I only left a comment.