Library and Tools Subset build times regression (50+% increase)

[LoopedBard3]

Description

Runtime builds for "-subset tools+libs" have started taking much more time as seen in timeouts in the performance pipelines and the official runtime build pipelines. The build time seems to have increased by 50+% (~40 min to 1+ hr in the perf pipeline).

Regression?

Data is below with diffs from before and after, although both performance pipeline runs used the same Dotnet SDK version: 8.0.100-alpha.1.23061.8.

Data

Specific runs that should properly capture the closest normal and regressed build:

• Performance pipeline (Look at "Libraries Build..." jobs):
  • Normal: https://dev.azure.com/dnceng/internal/_build/results?buildId=2119219&view=results
  • Regressed: https://dev.azure.com/dnceng/internal/_build/results?buildId=2119313&view=results
  • Compare: https://github.com/dotnet/runtime/compare/75eac964..f6bed512 (Smaller than below compare)
• official builds pipeline (Look at "Libraries Build... " jobs in the Build stage:
  • Normal: https://dev.azure.com/dnceng/internal/_build/results?buildId=2119244&view=results
  • Regressed: https://dev.azure.com/dnceng/internal/_build/results?buildId=2119371&view=results
  • Compare: https://github.com/dotnet/runtime/compare/75eac964..463954db

[jkoritzinsky]

We updated the Roslyn compiler version in this commit range. Maybe there's a regression in the compiler?

[hoyosjs]

This is related to #76454

[hoyosjs]

Looking at the time taken to build SPC it becomes pretty obvious window:

2023-02-21T21:45:00Z	12
2023-02-21T10:30:00Z	11.666666666666666
2023-02-21T16:30:00Z	11.666666666666666
2023-02-22T01:30:00Z	11.666666666666666
2023-02-21T15:45:00Z	11.6
2023-02-21T19:30:00Z	11.6
2023-02-21T22:30:00Z	11.333333333333334
2023-02-21T12:45:00Z	11.111111111111111
2023-02-21T09:00:00Z	11
2023-02-21T12:00:00Z	11
2023-02-21T18:00:00Z	11
2023-02-20T15:45:00Z	9.6666666666666661
2023-02-21T23:15:00Z	9.4
2023-02-22T02:15:00Z	9.2
2023-02-21T11:15:00Z	9.1428571428571423
2023-02-21T17:15:00Z	8.6666666666666661
2023-02-22T00:00:00Z	8.25
2023-02-21T04:30:00Z	7
2023-02-20T13:30:00Z	6.75
2023-02-20T12:45:00Z	6.666666666666667
2023-02-20T06:45:00Z	6.5
2023-02-20T11:15:00Z	6.5
2023-02-21T08:15:00Z	6.333333333333333
2023-02-20T22:30:00Z	6.2
2023-02-20T10:30:00Z	6.166666666666667
2023-02-20T07:30:00Z	6
2023-02-20T08:15:00Z	6
2023-02-20T15:00:00Z	6
2023-02-20T17:15:00Z	6
2023-02-21T00:00:00Z	6
2023-02-21T02:15:00Z	6
2023-02-21T03:00:00Z	5.8571428571428568
2023-02-20T20:15:00Z	5.8
2023-02-20T23:15:00Z	5.8

After 8 UTC - which matches the merge time of #81164

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Runtime builds for "-subset tools+libs" have started taking much more time as seen in timeouts in the performance pipelines and the official runtime build pipelines. The build time seems to have increased by 50+% (~40 min to 1+ hr in the perf pipeline).

Regression?

Data is below with diffs from before and after, although both performance pipeline runs used the same Dotnet SDK version: 8.0.100-alpha.1.23061.8.

Data

Specific runs that should properly capture the closest normal and regressed build:

• Performance pipeline (Look at "Libraries Build..." jobs):
  • Normal: https://dev.azure.com/dnceng/internal/_build/results?buildId=2119219&view=results
  • Regressed: https://dev.azure.com/dnceng/internal/_build/results?buildId=2119313&view=results
  • Compare: https://github.com/dotnet/runtime/compare/75eac964..f6bed512 (Smaller than below compare)
• official builds pipeline (Look at "Libraries Build... " jobs in the Build stage:
  • Normal: https://dev.azure.com/dnceng/internal/_build/results?buildId=2119244&view=results
  • Regressed: https://dev.azure.com/dnceng/internal/_build/results?buildId=2119371&view=results
  • Compare: https://github.com/dotnet/runtime/compare/75eac964..463954db

Author:	LoopedBard3
Assignees:	-
Labels:	area-Infrastructure-libraries, tenet-performance, untriaged
Milestone:	-

[jcouv]

@EgorBo To validate whether this build time increase was caused by the new length-based switch dispatch analysis, there is a command-line flag that will disable the analysis: /features:disable-length-based-switch. I believe <Features>disable-length-based-switch</Features> from MSBuild will also work, but I have not confirmed the exact syntax.

[EgorBo]

Can confirm that build.cmd Clr.CoreLib -c Release is regressed from 30s to 40s on my machine. <Features>disable-length-based-switch</Features> didn't help so must be something else then

[agocke]

Moving to Roslyn since this is definitely a compiler perf regression.