.NET 7.0 Preview 3 Microbenchmarks Performance Study Report #67976
Description
Data
This time we have covered following configs:
- architectures: x64, x86, arm64, arm
- Unix: Ubuntu 18.04, Ubuntu 20.04, pop 20.04, Alpine 3.13, CentOS 7, Debian 11, macOS Monterey
- Windows: 7, 10 & 11
| Operating System | Bit | Processor Name |
|---|---|---|
| alpine 3.12 | Arm64 | Unknown processor |
| ubuntu 18.04 | Arm64 | Unknown processor |
| Windows 10 | Arm64 | Microsoft SQ1 3.0 GHz |
| Windows 11 | Arm64 | Unknown processor |
| Windows 11 | Arm64 | Microsoft SQ1 3.0 GHz |
| macOS Monterey 12.3 | Arm64 | Apple M1 Max |
| Windows 10 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz |
| Windows 11 | X64 | AMD Ryzen Threadripper PRO 3945WX 12-Cores |
| Windows 11 | X64 | Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) |
| Windows 11 | X64 | Intel Core i9-9900T CPU 2.10GHz |
| alpine 3.13 | X64 | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) |
| centos 7 | X64 | Intel Xeon CPU E5530 2.40GHz |
| debian 11 | X64 | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) |
| opensuse-leap 15.3 | X64 | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) |
| ubuntu 18.04 | X64 | Intel Xeon CPU E5-1650 v4 3.60GHz |
| ubuntu 18.04 | X64 | Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge) |
| Windows 10 | Arm | Microsoft SQ1 3.0 GHz |
| Windows 11 | Arm | Unknown processor |
| ubuntu 18.04 | Arm | ARMv7 Processor rev 3 (v7l) |
| Windows 10 | X86 | Intel Xeon CPU E5-1650 v4 3.60GHz |
| Windows 11 | X86 | AMD Ryzen Threadripper PRO 3945WX 12-Cores |
| Windows 11 | X86 | Intel Core i7-10510U CPU 1.80GHz |
| Windows 7 SP1 | X86 | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) |
| macOS Monterey 12.2.1 | X64 | Intel Core i7-5557U CPU 3.10GHz (Broadwell) |
| macOS Monterey 12.3.1 | X64 | Intel Core i7-4870HQ CPU 2.50GHz (Haswell) |
Most of the benchmarks were run on bare-metal machines, some were executed via WSL.
This would not be possible without the help from: @AndyAyersMS @carlossanlop @danmoseley @dakersnar @janvorli @jeffhandley @kunalspathak and @sblom who contributed their results and time.
The full report generated by the tool is available here. The full report contains also improvements, so if you read it from the end you can see the biggest perf improvements. There are plenty of them!
Again, the full historical data turned out to be extremely useful. For details about methodology please read #41871. Preview2 report can be found here.
Stress bugs
Regressions
By design
-
System.Diagnostics.Perf_Process.GetProcessById -
System.Text.Json.Serialization.Tests.ReadJson<ArrayList>*
Investigation in progress
- System.ConsoleTests.Perf_Console.ResetColor, System.ConsoleTests.Perf_Console.ForegroundColor, System.ConsoleTests.Perf_Console.BackgroundColor
- it has already been reported in Regressions in System.ConsoleTests.Perf_Console and Perf_BitOperations #67598
-
BenchmarksGame.RegexRedux_1.RunBench,System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: ".{0,2}(Tom|Sawyer|Huckleberry|Finn)", Options: Compiled)- detected in Use StartsWith in regex compiler / source gen for shorter strings #65222 (comment), but not solved
- added a comment: Use StartsWith in regex compiler / source gen for shorter strings #65222 (comment)
- another benchmark: Use StartsWith in regex compiler / source gen for shorter strings #65222 (comment)
-
BenchmarksGame.BinaryTrees*,ByteMark.BenchBitOps,System.Tests.Perf_GC<Byte>.NewOperator_Array(length: 1000)- added a comment: regression on binary-trees regression in 5.0 #60659 (comment)
- opened BenchmarksGame.BinaryTrees benchmarks have regressed #67958
- most likely caused by GC regions (or OSR)
-
System.Text.Tests.Perf_StringBuilder.Append_Char_Capacity(length: 100000)- opened System.Text.Tests.Perf_StringBuilder.Append_Char_Capacity has regressed on x86 #67854
- specific to x86
-
System.IO.Tests.Perf_RandomAccess.ReadScatterAsyncand other async file IO benchmarks have regressed on Windows- opened Async File IO has regressed on Windows #67965
- specific to Windows
-
System.Tests.Perf_Enum.EnumToString(value: Yellow) -
System.Tests.Perf_DateTime.GetNow -
PerfLabTests.LowLevelPerf.StaticDelegate -
LinqBenchmarks.Where00ForX- opened LinqBenchmarks.Where00ForX has regressed on x86 #67968
- specific to x86
-
PerfLabTests.StackWalk.Walk- opened PerfLabTests.StackWalk.Walk has regressed on arm64 #67971
- specific to arm64
-
System.Threading.Tests.Perf_Timer.ScheduleManyThenDisposeMany- opened System.Threading.Tests.Perf_Timer.ScheduleManyThenDisposeMany has regressed on 64 bit Windows #67972
- specific to 64 bit Windows
-
System.Tests.Perf_Int32.Parse,System.Tests.Perf_Int16.TryParse,System.Tests.Perf_SByte.TryParse- opened Parsing integers has regressed on arm32 #67974
- specific to arm
Already solved
-
System.Tests.Perf_GC<Char>.AllocateUninitializedArray(length: 10000, pinned: True)- alread solved, added a comment: Change
CrossLoaderAllocatorHashto not use the GC #67160 (comment)
- alread solved, added a comment: Change
-
System.Buffers.Tests.ReadOnlySequenceTests<Byte>.IterateGetPositionTenSegments- regressed on x64 Unix, but has been already solved
-
System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Validated*,System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get*- a lot of socket benchmarks have regressed on Windows, but have been already solved
Noise, flaky or multimodal
The following benchmarks showed up in the report generated by the tool, but were not actual regressions:
System.Collections.Concurrent.Count<String>.Dictionary(Size: 512)System.Numerics.Tests.Perf_VectorOf<*>.*System.Collections.TryAddDefaultSize<String>.ConcurrentDictionary(Count: 512)- whether it is slower or faster correlates with the alloc delta. Very noisy.
PerfLabTests.CastingPerf.CheckArrayIsInterfaceNo
Statistics
- Statistical Test threshold: 10%, the noise filter: 2 ns
- Result is conclusion: Slower|Faster|Same|Noise|Unknown. Noise means that the difference was larger than 10% but not 2 ns.
Total: 105070
Same: 74.24 %
Slower: 4.64 %
Faster: 10.13 %
Noise: 10.95 %
Unknown: 0.04 %
Statistics per Architecture
| Architecture | Same | Slower | Faster | Noise | Unknown |
|---|---|---|---|---|---|
| Arm64 | 79.78 % | 3.72 % | 6.13 % | 10.35 % | 0.02 % |
| Arm | 73.79 % | 5.46 % | 14.75 % | 5.96 % | 0.04 % |
| X64 | 71.73 % | 4.47 % | 11.05 % | 12.71 % | 0.04 % |
| X86 | 74.81 % | 6.22 % | 9.64 % | 9.29 % | 0.04 % |
Statistics per Operating System
| Operating System | Same | Slower | Faster | Noise | Unknown |
|---|---|---|---|---|---|
| alpine 3.12 | 87.67 % | 2.02 % | 2.38 % | 7.93 % | 0.00 % |
| debian 11 | 75.36 % | 2.93 % | 8.66 % | 13.04 % | 0.00 % |
| macOS Monterey 12.3 | 75.89 % | 0.64 % | 12.30 % | 11.17 % | 0.00 % |
| ubuntu 18.04 | 66.00 % | 3.53 % | 21.29 % | 9.07 % | 0.11 % |
| ubuntu 20.04 | 81.05 % | 6.23 % | 6.73 % | 5.99 % | 0.00 % |
| Windows 10 | 74.50 % | 5.62 % | 8.36 % | 11.51 % | 0.01 % |
| Windows 11 | 74.32 % | 6.18 % | 8.88 % | 10.59 % | 0.02 % |
| alpine 3.13 | 67.48 % | 11.22 % | 4.63 % | 16.68 % | 0.00 % |
| centos 7 | 72.34 % | 3.79 % | 12.99 % | 10.79 % | 0.09 % |
| macOS Monterey 12.2.1 | 80.04 % | 2.13 % | 4.71 % | 13.03 % | 0.09 % |
| macOS Monterey 12.3.1 | 80.02 % | 1.24 % | 3.43 % | 15.31 % | 0.00 % |
| opensuse-leap 15.3 | 80.02 % | 2.61 % | 4.09 % | 13.28 % | 0.00 % |
| pop 20.04 | 84.93 % | 4.87 % | 6.50 % | 3.69 % | 0.00 % |
| Windows 7 SP1 | 85.61 % | 2.86 % | 3.80 % | 7.73 % | 0.00 % |
Statistics per Namespace
| Namespace | Same | Slower | Faster | Noise | Unknown |
|---|---|---|---|---|---|
| BenchmarksGame | 79.91 % | 15.77 % | 4.33 % | 0.00 % | 0.00 % |
| Benchstone.BenchF | 85.54 % | 8.77 % | 5.69 % | 0.00 % | 0.00 % |
| Benchstone.BenchI | 81.93 % | 5.94 % | 12.14 % | 0.00 % | 0.00 % |
| Benchstone.MDBenchF | 92.26 % | 5.16 % | 2.58 % | 0.00 % | 0.00 % |
| Benchstone.MDBenchI | 88.85 % | 4.18 % | 6.97 % | 0.00 % | 0.00 % |
| Devirtualization | 73.79 % | 5.52 % | 3.45 % | 17.24 % | 0.00 % |
| Exceptions | 89.38 % | 7.43 % | 3.19 % | 0.00 % | 0.00 % |
| FractalPerf | 91.18 % | 5.88 % | 2.94 % | 0.00 % | 0.00 % |
| GuardedDevirtualization | 89.54 % | 0.19 % | 0.09 % | 10.18 % | 0.00 % |
| Inlining | 36.08 % | 0.00 % | 5.15 % | 58.76 % | 0.00 % |
| Interop | 89.90 % | 6.06 % | 4.04 % | 0.00 % | 0.00 % |
| JetStream | 75.00 % | 8.33 % | 16.67 % | 0.00 % | 0.00 % |
| Layout | 84.85 % | 9.09 % | 6.06 % | 0.00 % | 0.00 % |
| Lowering | 92.59 % | 0.00 % | 0.00 % | 7.41 % | 0.00 % |
| MicroBenchmarks.Serializers | 78.51 % | 0.54 % | 20.95 % | 0.00 % | 0.00 % |
| Microsoft.Extensions.Logging | 75.19 % | 1.84 % | 22.21 % | 0.77 % | 0.00 % |
| PerfLabTests | 64.33 % | 9.46 % | 20.95 % | 5.27 % | 0.00 % |
| PerfLabTests.CastingPerf2 | 75.06 % | 4.20 % | 20.74 % | 0.00 % | 0.00 % |
| SciMark2 | 69.09 % | 2.42 % | 28.48 % | 0.00 % | 0.00 % |
| SIMD | 86.20 % | 2.36 % | 11.45 % | 0.00 % | 0.00 % |
| Span | 91.52 % | 2.55 % | 5.94 % | 0.00 % | 0.00 % |
| StoreBlock | 81.23 % | 0.62 % | 0.62 % | 17.54 % | 0.00 % |
| System.Buffers.Tests | 81.16 % | 2.57 % | 12.78 % | 3.49 % | 0.00 % |
| System.Buffers.Text.Tests | 77.85 % | 1.72 % | 7.66 % | 12.77 % | 0.00 % |
| System.Collections | 84.57 % | 3.78 % | 11.08 % | 0.57 % | 0.00 % |
| System.Collections.Concurrent | 79.23 % | 3.08 % | 7.56 % | 10.13 % | 0.00 % |
| System.Collections.Tests | 79.08 % | 4.06 % | 10.41 % | 6.45 % | 0.00 % |
| System.ConsoleTests | 80.25 % | 12.96 % | 6.79 % | 0.00 % | 0.00 % |
| System.Diagnostics | 78.33 % | 11.30 % | 10.37 % | 0.00 % | 0.00 % |
| System.Diagnostics.Tracing | 90.00 % | 5.00 % | 5.00 % | 0.00 % | 0.00 % |
| System.Formats.Cbor.Tests | 84.92 % | 0.00 % | 15.08 % | 0.00 % | 0.00 % |
| System.Globalization.Tests | 84.99 % | 4.37 % | 8.86 % | 1.78 % | 0.00 % |
| System | 84.01 % | 4.71 % | 9.26 % | 2.02 % | 0.00 % |
| System.IO.Compression | 91.62 % | 0.41 % | 7.97 % | 0.00 % | 0.00 % |
| System.IO.Pipelines.Tests | 85.19 % | 1.85 % | 12.96 % | 0.00 % | 0.00 % |
| System.IO.Pipes.Tests | 90.00 % | 5.33 % | 4.67 % | 0.00 % | 0.00 % |
| System.IO.Tests | 79.41 % | 8.05 % | 11.10 % | 1.44 % | 0.00 % |
| System.Linq.Tests | 80.26 % | 8.11 % | 10.96 % | 0.67 % | 0.00 % |
| System.MathBenchmarks | 84.68 % | 4.74 % | 9.77 % | 0.81 % | 0.00 % |
| System.Memory | 66.17 % | 2.99 % | 8.40 % | 22.44 % | 0.00 % |
| System.Net.Http.Tests | 60.33 % | 34.96 % | 4.71 % | 0.00 % | 0.00 % |
| System.Net.Primitives.Tests | 82.60 % | 3.04 % | 8.56 % | 5.80 % | 0.00 % |
| System.Net.Sockets.Tests | 87.50 % | 9.82 % | 2.68 % | 0.00 % | 0.00 % |
| System.Net.Tests | 81.48 % | 10.49 % | 8.02 % | 0.00 % | 0.00 % |
| System.Numerics.Tests | 50.81 % | 3.87 % | 5.38 % | 39.94 % | 0.00 % |
| System.Reflection | 84.50 % | 1.11 % | 12.55 % | 1.85 % | 0.00 % |
| System.Runtime.InteropServices.Tests | 79.41 % | 5.88 % | 11.76 % | 2.94 % | 0.00 % |
| System.Security.Cryptography.Tests | 92.52 % | 0.68 % | 6.80 % | 0.00 % | 0.00 % |
| System.Tests | 75.29 % | 5.32 % | 10.64 % | 8.76 % | 0.00 % |
| System.Text.Encodings.Web.Tests | 87.50 % | 3.26 % | 8.70 % | 0.54 % | 0.00 % |
| System.Text.Json.Document.Tests | 88.95 % | 2.70 % | 8.36 % | 0.00 % | 0.00 % |
| System.Text.Json.Reader.Tests | 83.70 % | 3.26 % | 13.04 % | 0.00 % | 0.00 % |
| System.Text.Json.Tests | 84.97 % | 1.35 % | 13.67 % | 0.00 % | 0.00 % |
| System.Text | 87.41 % | 2.72 % | 9.88 % | 0.00 % | 0.00 % |
| System.Text.Tests | 83.93 % | 6.05 % | 8.04 % | 1.98 % | 0.00 % |
| System.Threading.Tasks.Tests | 81.82 % | 4.55 % | 13.64 % | 0.00 % | 0.00 % |
| System.Threading.Tasks | 75.00 % | 6.17 % | 17.86 % | 0.97 % | 0.00 % |
| System.Threading.Tests | 82.13 % | 3.80 % | 8.43 % | 5.65 % | 0.00 % |
| System.Xml.Linq | 79.07 % | 0.60 % | 12.27 % | 8.05 % | 0.00 % |
| System.Xml.Tests | 14.17 % | 0.00 % | 85.83 % | 0.00 % | 0.00 % |
| V8.Crypto | 80.00 % | 6.67 % | 13.33 % | 0.00 % | 0.00 % |
| V8.Richards | 80.00 % | 3.33 % | 16.67 % | 0.00 % | 0.00 % |
| XmlDocumentTests.XmlDocumentTests | 86.36 % | 6.06 % | 4.55 % | 3.03 % | 0.00 % |
| XmlDocumentTests.XmlNodeListTests | 84.09 % | 2.27 % | 13.64 % | 0.00 % | 0.00 % |
| XmlDocumentTests.XmlNodeTests | 50.00 % | 4.55 % | 2.27 % | 43.18 % | 0.00 % |
| System.Net.Security.Tests | 86.13 % | 9.68 % | 4.19 % | 0.00 % | 0.00 % |
| Microsoft.AspNetCore.Server.Kestrel.Performance | 84.09 % | 6.82 % | 9.09 % | 0.00 % | 0.00 % |
| Microsoft.Extensions.Caching.Memory.Tests | 74.38 % | 5.79 % | 19.83 % | 0.00 % | 0.00 % |
| Microsoft.Extensions.Configuration.Xml | 90.91 % | 0.00 % | 9.09 % | 0.00 % | 0.00 % |
| Microsoft.Extensions.DependencyInjection | 72.73 % | 0.88 % | 24.63 % | 1.76 % | 0.00 % |
| Microsoft.Extensions.Primitives.Performance | 50.00 % | 0.00 % | 0.00 % | 50.00 % | 0.00 % |
| Microsoft.Extensions.Primitives | 70.20 % | 2.02 % | 14.14 % | 13.64 % | 0.00 % |
| System.Buffers.Binary.Tests | 67.05 % | 10.23 % | 13.64 % | 9.09 % | 0.00 % |
| System.ComponentModel.Tests | 90.91 % | 1.65 % | 7.44 % | 0.00 % | 0.00 % |
| System.IO.MemoryMappedFiles.Tests | 84.85 % | 11.36 % | 3.79 % | 0.00 % | 0.00 % |
| System.Net.NetworkInformation.Tests | 100.00 % | 0.00 % | 0.00 % | 0.00 % | 0.00 % |
| System.Net.WebSockets.Tests | 81.82 % | 9.09 % | 9.09 % | 0.00 % | 0.00 % |
| System.Runtime.Serialization.Formatters.Tests | 54.55 % | 45.45 % | 0.00 % | 0.00 % | 0.00 % |
| System.Security.Cryptography.Primitives.Tests.Performance | 77.27 % | 10.23 % | 12.50 % | 0.00 % | 0.00 % |
| System.Security.Cryptography.X509Certificates.Tests | 84.85 % | 0.00 % | 15.15 % | 0.00 % | 0.00 % |
| System.Text.Json.Serialization.Tests | 80.54 % | 5.02 % | 14.43 % | 0.00 % | 0.00 % |
| System.Text.RegularExpressions.Tests | 67.28 % | 6.24 % | 25.16 % | 0.00 % | 1.33 % |
| System.Threading.Channels.Tests | 89.17 % | 0.00 % | 10.83 % | 0.00 % | 0.00 % |
| System.Drawing.Tests | 83.86 % | 2.76 % | 8.66 % | 4.72 % | 0.00 % |
| HardwareIntrinsics.RayTracer | 88.24 % | 0.00 % | 0.00 % | 11.76 % | 0.00 % |
Big thanks to everyone involved!