CPU time vs code size tradeoffs

[adamsitnik]

While reviewing some of the stale PRs I’ve noticed that we are frequently discussing the tradeoff between improving CPU time at the cost of increased code size. Example: #90459 (comment)

We have the tools to measure the former, but I am not sure about the latter.

I believe that we should at least provide a clear definition of what we mean by the code size (size of the managed assembly? size of precompiled native code? both?). If possible, BenchmarkDotNet should also include the metrics we care about, by default when benchmarking local dotnet runtime builds. Of course, this should be documented as well.

Ideally, we would also describe our decision process behind making or rejecting such tradeoffs, with some examples.

@jkotas @stephentoub thoughts?

Tagging subscribers to this area: @dotnet/area-meta
See info in area-owners.md if you want to be subscribed.

Issue Details

---

While reviewing some of the stale PRs I’ve noticed that we are frequently discussing the tradeoff between improving CPU time at the cost of increased code size. Example: #90459 (comment)

We have the tools to measure the former, but I am not sure about the latter.

I believe that we should at least provide a clear definition of what we mean by the code size (size of the managed assembly? size of precompiled native code? both?). If possible, BenchmarkDotNet should also include the metrics we care about, by default when benchmarking local dotnet runtime builds.

Ideally, we would also describe our decision process behind making or rejecting such tradeoffs, with some examples.

@jkotas @stephentoub thoughts?

Author:	adamsitnik
Assignees:	-
Labels:	area-Meta, tenet-performance
Milestone:	-

[stephentoub]

I believe that we should at least provide a clear definition of what we mean by the code size

Native assembly code plus all the runtime data structures required to keep track of everything. IL is generally less interesting.

[adamsitnik]

all the runtime data structures required to keep track of everything

Could you provide some exact names? It would be easier for me to search for it in the ClrMD APIs.

[jkotas]

We have macro benchmarks to measure the published binary sizes and startup time for different application types that will detect this type of regressions. #93072 is an example of a regression detected by these benchmarks.

ClrMD APIs.

ClrMD API does not have APIs for this. It is hard to account for all contributing costs reliably.

For micro benchmarking purposes, I typically create a test that has thousand instances of the construct over different types and then measure the working set or startup time for one instance vs. thousand instances. Here is an example of such test from years ago: https://gist.github.com/jkotas/102dc708cca8d2c85002cb47bdd49870

[ericstj]

There are also linker tests that track code size for common application types. @eerhardt

[eerhardt]

We have a set of ASP.NET benchmark apps where we track the native AOT size. Links to the app code:

• https://github.com/aspnet/Benchmarks/tree/main/src/BenchmarksApps/BasicMinimalApi
• https://github.com/aspnet/Benchmarks/tree/main/src/BenchmarksApps/TodosApi
• https://github.com/aspnet/Benchmarks/tree/main/src/BenchmarksApps/Grpc/BasicGrpc

These are run a couple times a day on the latest bits and any size changes larger than 2% get automatic issues logged. For example:

Published size regression: todosapipublishaot, todosapipublishaot, todosapipublishaot (dotnet/aspnetcore#56139)

We also have a "dotnet new console" template benchmark that is run in the perf lab and it measures the size of the default "Hello World" app. I'm not sure where that code is and can't find the link to the results right now. cc @LoopedBard3 @MichalStrehovsky

[MichalStrehovsky]

We also have a "dotnet new console" template benchmark that is run in the perf lab and it measures the size of the default "Hello World" app. I'm not sure where that code is and can't find the link to the results right now. cc @LoopedBard3 @MichalStrehovsky

I think it's this one: https://github.com/dotnet/performance/tree/main/src/scenarios/emptyconsolenativeaot
Some dashboard is at: https://dataexplorer.azure.com/dashboards/c9dc2cc3-d492-4137-ab16-93db390da316?p-_startTime=30days&p-_endTime=now&p-_queue=v-Ubuntu.2204.Amd64.Tiger.Perf#c8e6c269-7ed1-457f-9225-97e9611e4a8a