Reduce heap allocations for asynchronously-completing methods

[divega]

.NET Core includes some helpers to implement nearly allocation-free IAsyncEnumerable<T> iterators.

One way to take advantage of this feature is to use async interators support in C# in the implementation. Another one is to use ManualResetValueTaskSourceCore<T> manually.

See https://github.com/dotnet/corefx/issues/32664 for more details.

EDIT BY @roji: investigate opportunities for using the new AsyncMethodBuilderOverrideAttribute and PoolingAsyncValueTaskMethodBuilders, dotnet/runtime#49903

[benaadams]

This may address it also dotnet/coreclr#26310

[roji]

I've done another test of DOTNET_SYSTEM_THREADING_POOLASYNCVALUETASKS (dotnet/coreclr#26310) on the latest EF+Npgsql TechEmpower Fortunes non-platform benchmark. on Citrine. tl;dr I can see a reduction of ~50% total MB allocated in both DotMemory and BenchmarkDotNet, but I can't see a throughput difference in either BenchmarkDotNet or the heavily-concurrent Fortunes web scenario.

Confirmed running locally with DotMemory, that when DOTNET_SYSTEM_THREADING_POOLASYNCVALUETASKS is set to true, almost all AsyncStateMachineBoxes disappear; allocated bytes (for 10000 iterations) goes down from 32.40MB to 16.10MB, and allocated number of objects from 379.9K to 300K.

Commits used, replacing Task by ValueTask in the Fortunes query hot path (perf/ValueTaskEverywhere): EFCore, EFCore.PG, Npgsql.

Non-concurrent (BenchmarkDotNet)

Non-concurrent BenchmarkDotNet tests run locally on my laptop, with the Fortunes scenario:

Without DOTNET_SYSTEM_THREADING_POOLASYNCVALUETASKS

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19042
Intel Core i7-9750H CPU 2.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=6.0.100-preview.4.21220.4
  [Host]     : .NET Core 5.0.5 (CoreCLR 5.0.521.16609, CoreFX 5.0.521.16609), X64 RyuJIT
  DefaultJob : .NET Core 5.0.5 (CoreCLR 5.0.521.16609, CoreFX 5.0.521.16609), X64 RyuJIT

Method	Mean	Error	StdDev	Gen 0	Gen 1	Gen 2	Allocated
LoadFortunesRows	175.2 us	3.25 us	3.20 us	0.4883	-	-	3.4 KB

With DOTNET_SYSTEM_THREADING_POOLASYNCVALUETASKS

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19042
Intel Core i7-9750H CPU 2.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=6.0.100-preview.4.21220.4
[Host] : .NET Core 5.0.5 (CoreCLR 5.0.521.16609, CoreFX 5.0.521.16609), X64 RyuJIT
DefaultJob : .NET Core 5.0.5 (CoreCLR 5.0.521.16609, CoreFX 5.0.521.16609), X64 RyuJIT

Method	Mean	Error	StdDev	Gen 0	Gen 1	Gen 2	Allocated
LoadFortunesRows	179.7 us	3.48 us	4.65 us	0.2441	-	-	1.73 KB

Concurrent (web scenario)

Ran the standard TechEmpower Fortunes non-platform EF benchmark.

Without DOTNET_SYSTEM_THREADING_POOLASYNCVALUETASKS

RPS: 250,976

Complete run results

Average results:

| db               |       |
| ---------------- | ----- |
| CPU Usage (%)    | 24    |
| Cores usage (%)  | 686   |
| Working Set (MB) | 43    |
| Build Time (ms)  | 1,710 |
| Start Time (ms)  | 351   |

| application                             |                                  |
| --------------------------------------- | -------------------------------- |
| CPU Usage (%)                           | 95                               |
| Cores usage (%)                         | 2,654                            |
| Working Set (MB)                        | 459                              |
| Private Memory (MB)                     | 1,586                            |
| Build Time (ms)                         | 2,825                            |
| Start Time (ms)                         | 225                              |
| Published Size (KB)                     | 104,913                          |
| .NET Core SDK Version                   | 6.0.100-preview.5.21256.2        |
| ASP.NET Core Version                    | 6.0.0-preview.5.21255.10+1db20af |
| .NET Runtime Version                    | 6.0.0-preview.5.21256.2+8f3ac9b  |
| Max CPU Usage (%)                       | 94                               |
| Max Working Set (MB)                    | 480                              |
| Max GC Heap Size (MB)                   | 303                              |
| Size of committed memory by the GC (MB) | 382                              |
| Max Number of Gen 0 GCs / sec           | 8.00                             |
| Max Number of Gen 1 GCs / sec           | 2.40                             |
| Max Number of Gen 2 GCs / sec           | 1.00                             |
| Max Time in GC (%)                      | 1.20                             |
| Max Gen 0 Size (B)                      | 45,820,802                       |
| Max Gen 1 Size (B)                      | 14,670,227                       |
| Max Gen 2 Size (B)                      | 15,134,054                       |
| Max LOH Size (B)                        | 125,282                          |
| Max Allocation Rate (B/sec)             | 2,149,198,864                    |
| Max GC Heap Fragmentation               | 79                               |
| # of Assemblies Loaded                  | 115                              |
| Max Exceptions (#/s)                    | 648                              |
| Max Lock Contention (#/s)               | 623                              |
| Max ThreadPool Threads Count            | 46                               |
| Max ThreadPool Queue Length             | 48                               |
| Max ThreadPool Items (#/s)              | 746,046                          |
| Max Active Timers                       | 10                               |
| IL Jitted (B)                           | 651,104                          |
| Methods Jitted                          | 8,800                            |

| load                   |           |
| ---------------------- | --------- |
| CPU Usage (%)          | 15        |
| Cores usage (%)        | 417       |
| Working Set (MB)       | 37        |
| Private Memory (MB)    | 358       |
| Start Time (ms)        | 0         |
| First Request (ms)     | 234       |
| Requests/sec           | 250,976   |
| Requests               | 3,789,650 |
| Mean latency (ms)      | 1.06      |
| Max latency (ms)       | 20.36     |
| Bad responses          | 0         |
| Socket errors          | 0         |
| Read throughput (MB/s) | 327.19    |
| Latency 50th (ms)      | 1.00      |
| Latency 75th (ms)      | 1.27      |
| Latency 90th (ms)      | 1.47      |
| Latency 99th (ms)      | 3.26      |

Crank commandline:

dotnet run -- --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/database.benchmarks.yml --scenario fortunes_ef --profile aspnet-citrine-lin --application.framework net6.0 --application.options.outputFiles Z:\projects\test\bin\Release\net5.0\publish\*.dll --application.options.collectCounters true -i 5

With DOTNET_SYSTEM_THREADING_POOLASYNCVALUETASKS

RPS: 251,336

Complete run results

Average results:

| db               |       |
| ---------------- | ----- |
| CPU Usage (%)    | 24    |
| Cores usage (%)  | 680   |
| Working Set (MB) | 44    |
| Build Time (ms)  | 1,754 |
| Start Time (ms)  | 371   |

| application                             |                                  |
| --------------------------------------- | -------------------------------- |
| CPU Usage (%)                           | 95                               |
| Cores usage (%)                         | 2,660                            |
| Working Set (MB)                        | 459                              |
| Private Memory (MB)                     | 1,577                            |
| Build Time (ms)                         | 2,762                            |
| Start Time (ms)                         | 222                              |
| Published Size (KB)                     | 104,913                          |
| .NET Core SDK Version                   | 6.0.100-preview.5.21256.2        |
| ASP.NET Core Version                    | 6.0.0-preview.5.21255.10+1db20af |
| .NET Runtime Version                    | 6.0.0-preview.5.21256.2+8f3ac9b  |
| Max CPU Usage (%)                       | 94                               |
| Max Working Set (MB)                    | 480                              |
| Max GC Heap Size (MB)                   | 309                              |
| Size of committed memory by the GC (MB) | 381                              |
| Max Number of Gen 0 GCs / sec           | 8.00                             |
| Max Number of Gen 1 GCs / sec           | 2.20                             |
| Max Number of Gen 2 GCs / sec           | 1.00                             |
| Max Time in GC (%)                      | 3.00                             |
| Max Gen 0 Size (B)                      | 48,048,307                       |
| Max Gen 1 Size (B)                      | 14,112,371                       |
| Max Gen 2 Size (B)                      | 13,888,410                       |
| Max LOH Size (B)                        | 217,077                          |
| Max Allocation Rate (B/sec)             | 2,166,158,403                    |
| Max GC Heap Fragmentation               | 79                               |
| # of Assemblies Loaded                  | 115                              |
| Max Exceptions (#/s)                    | 664                              |
| Max Lock Contention (#/s)               | 964                              |
| Max ThreadPool Threads Count            | 45                               |
| Max ThreadPool Queue Length             | 23                               |
| Max ThreadPool Items (#/s)              | 746,741                          |
| Max Active Timers                       | 11                               |
| IL Jitted (B)                           | 650,822                          |
| Methods Jitted                          | 8,788                            |

| load                   |           |
| ---------------------- | --------- |
| CPU Usage (%)          | 15        |
| Cores usage (%)        | 419       |
| Working Set (MB)       | 37        |
| Private Memory (MB)    | 358       |
| Start Time (ms)        | 0         |
| First Request (ms)     | 234       |
| Requests/sec           | 251,336   |
| Requests               | 3,795,093 |
| Mean latency (ms)      | 1.06      |
| Max latency (ms)       | 18.59     |
| Bad responses          | 0         |
| Socket errors          | 0         |
| Read throughput (MB/s) | 327.66    |
| Latency 50th (ms)      | 0.99      |
| Latency 75th (ms)      | 1.27      |
| Latency 90th (ms)      | 1.47      |
| Latency 99th (ms)      | 3.33      |

Crank commandline:

dotnet run -- --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/database.benchmarks.yml --scenario fortunes_ef --profile aspnet-citrine-lin --application.framework net6.0 --application.options.outputFiles Z:\projects\test\bin\Release\net5.0\publish\*.dll --application.environmentvariables DOTNET_SYSTEM_THREADING_POOLASYNCVALUETASKS=true --application.options.collectCounters true -i 5

/cc @stephentoub @sebastienros @ajcvickers