[Performance] Startup regression

[sebastienros]

Smallest commits set:
fe98008...10222f9

Crank command lines:

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/json.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/azure.profile.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario mapaction --profile intel-lin-app --profile intel-load2-load --application.framework net8.0 --application.collectDependencies true --application.options.collectCounters true --application.aspNetCoreVersion 8.0.0-preview.6.23302.1 --application.runtimeVersion 8.0.0-preview.6.23305.2 --application.sdkVersion 8.0.100-preview.6.23305.4

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/json.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/azure.profile.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario mapaction --profile intel-lin-app --profile intel-load2-load --application.framework net8.0 --application.collectDependencies true --application.options.collectCounters true --application.aspNetCoreVersion 8.0.0-preview.6.23302.1 --application.runtimeVersion 8.0.0-preview.6.23306.5 --application.sdkVersion 8.0.100-preview.6.23305.4

[EgorBo]

Seems like arm64 is not affected

[sebastienros]

@EgorBo you are correct, was checking and the last run is past the culprit build on these machines too.

[EgorBo]

Methods jitted 3,644 --> 11,398 so something went wrong with R2R, perhaps, @jkotas can spot an offender
My guess is #85551 (cc @tannergooding)

| application                             | before                    | after                     |          |
| --------------------------------------- | ------------------------- | ------------------------- | -------- |
| CPU Usage (%)                           |                       100 |                        99 |   -1.00% |
| Cores usage (%)                         |                     2,794 |                     2,780 |   -0.50% |
| Working Set (MB)                        |                       464 |                       471 |   +1.51% |
| Private Memory (MB)                     |                     1,239 |                     1,248 |   +0.73% |
| Build Time (ms)                         |                     3,094 |                     3,083 |   -0.36% |
| Start Time (ms)                         |                       188 |                       403 | +114.36% |
| Published Size (KB)                     |                    90,772 |                    90,905 |   +0.15% |
| Symbols Size (KB)                       |                        21 |                        21 |    0.00% |
| .NET Core SDK Version                   | 8.0.100-preview.6.23305.4 | 8.0.100-preview.6.23305.4 |          |
| Max CPU Usage (%)                       |                       100 |                        99 |   -0.39% |
| Max Working Set (MB)                    |                       483 |                       498 |   +3.08% |
| Max GC Heap Size (MB)                   |                       341 |                       331 |   -3.03% |
| Size of committed memory by the GC (MB) |                       418 |                       431 |   +3.16% |
| Max Number of Gen 0 GCs / sec           |                      1.00 |                      1.00 |    0.00% |
| Max Number of Gen 1 GCs / sec           |                      1.00 |                      1.00 |    0.00% |
| Max Number of Gen 2 GCs / sec           |                      1.00 |                      1.00 |    0.00% |
| Max Time in GC (%)                      |                      1.00 |                      1.00 |    0.00% |
| Max Gen 0 Size (B)                      |                       584 |                       584 |    0.00% |
| Max Gen 1 Size (B)                      |                 4,045,344 |                 4,053,456 |   +0.20% |
| Max Gen 2 Size (B)                      |                 4,074,984 |                 4,022,064 |   -1.30% |
| Max LOH Size (B)                        |                   185,080 |                   381,744 | +106.26% |
| Max POH Size (B)                        |                 1,187,064 |                 1,187,064 |    0.00% |
| Max Allocation Rate (B/sec)             |               174,741,712 |               175,985,216 |   +0.71% |
| Max GC Heap Fragmentation               |                         1 |                         1 |   +1.23% |
| # of Assemblies Loaded                  |                        88 |                        88 |    0.00% |
| Max Exceptions (#/s)                    |                       452 |                       448 |   -0.88% |
| Max Lock Contention (#/s)               |                       110 |                       118 |   +7.27% |
| Max ThreadPool Threads Count            |                        48 |                        48 |    0.00% |
| Max ThreadPool Queue Length             |                       134 |                       139 |   +3.73% |
| Max ThreadPool Items (#/s)              |                 1,111,404 |                 1,121,252 |   +0.89% |
| Max Active Timers                       |                         1 |                         1 |    0.00% |
| IL Jitted (B)                           |                   366,349 |                   773,346 | +111.10% |
| Methods Jitted                          |                     3,634 |                    11,398 | +213.65% |


| load                   | before     | after      |         |
| ---------------------- | ---------- | ---------- | ------- |
| CPU Usage (%)          |         67 |         68 |  +1.49% |
| Cores usage (%)        |      1,877 |      1,903 |  +1.39% |
| Working Set (MB)       |         48 |         48 |   0.00% |
| Private Memory (MB)    |        358 |        358 |   0.00% |
| Start Time (ms)        |          0 |          0 |         |
| First Request (ms)     |        191 |        328 | +71.73% |
| Requests/sec           |    965,738 |    985,739 |  +2.07% |
| Requests               | 14,582,098 | 14,884,089 |  +2.07% |
| Mean latency (ms)      |       0.51 |       0.50 |  -1.12% |
| Max latency (ms)       |      50.18 |      36.19 | -27.88% |
| Bad responses          |          0 |          0 |         |
| Socket errors          |          0 |          0 |         |
| Read throughput (MB/s) |     199.86 |     204.00 |  +2.07% |
| Latency 50th (ms)      |       0.24 |       0.23 |  -2.08% |
| Latency 75th (ms)      |       0.28 |       0.27 |  -2.52% |
| Latency 90th (ms)      |       0.33 |       0.32 |  -2.45% |
| Latency 99th (ms)      |       9.41 |       9.51 |  +1.06% |

[tannergooding]

Is there an easy way to get a list of functions that are now jitted that weren't before?

[tannergooding]

I had done the r2rdump check for corelib as part of the PR and hadn't seen any differences; but maybe I had done something wrong here...

[EgorBo]

Is there an easy way to get a list of functions that are now jitted that weren't before?

--application.environmentVariables DOTNET_JitDisasmSummary=1 --application.options.displayOutput true

should do the trick

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Smallest commits set:
fe98008...10222f9

Crank command lines:

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/json.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/azure.profile.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario mapaction --profile intel-lin-app --profile intel-load2-load --application.framework net8.0 --application.collectDependencies true --application.options.collectCounters true --application.aspNetCoreVersion 8.0.0-preview.6.23302.1 --application.runtimeVersion 8.0.0-preview.6.23305.2 --application.sdkVersion 8.0.100-preview.6.23305.4

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/json.benchmarks.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/azure.profile.yml --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml --scenario mapaction --profile intel-lin-app --profile intel-load2-load --application.framework net8.0 --application.collectDependencies true --application.options.collectCounters true --application.aspNetCoreVersion 8.0.0-preview.6.23302.1 --application.runtimeVersion 8.0.0-preview.6.23306.5 --application.sdkVersion 8.0.100-preview.6.23305.4

Author:	sebastienros
Assignees:	-
Labels:	tenet-performance, area-CodeGen-coreclr, untriaged, needs-area-label
Milestone:	-