Split IISFunctionals startup/shadow copy tests

[HaoK]

Fixes #37915

Start splitting up tests to try and reduce the long pole of IIS.Functional tests

[HaoK]

Splitting the these tests gets IIS.Functionals down to 25 minutes from 33 minutes, I'm not sure its worth moving the HTTP2/3 tests out as it only shaves off about 2 minutes from the running time, but IIS functionals are still the long pole, with the next closest around 18 minutes, so we could shave another 7 minutes from the total time for asnpetcore-ci PR checks by splitting things up more

It gets a bit uglier from here, we'd potentially have to split Startup/Shutdown tests out, which would create 4-5 more test projects since that would require splitting the base tests into 2 groups but would almost certainly get the time down to where we want...

Thoughts @adityamandaleeka @Tratcher @dougbu @sebastienros @wtgodbe

[dougbu]

@HaoK if I'm reading this correctly, you're currently splitting IIS.FunctionalTests into 3 parts -- tests from Common.FunctionalTests, new IIS.Http.FunctionalTests, and new IIS.ShadowCopy.Tests.

• I think you're suggesting IIS.Http.FunctionalTests doesn't help much. Is that correct❔
• I also am pretty sure work items for IIS.FunctionalTests, IIS.Http.FunctionalTests, and IIS.ShadowCopy.Tests don't overlap because the Common.FunctionalTests tests execute only in IIS.FunctionalTests (though the other two reuse the same Infrastructure files). Right❔
• How long do the work items for each of those take on the supported Helix queues❔
  • Need more than one run to get a solid sense here.
  • The specific Helix queue probably matters. Suggest temporarily updating helix-matrix.yml to limit Helix runs to src/Servers/IIS/IIS/test/**/*.csproj (if that works -- I hope it does) and Helix.Common.props to use only queues to where those tests can run, then doing a few manual aspnetcore-helix-matrix builds.

[HaoK]

Not quite, I just moved ShadowTests into a separate work item, as well as the handful of Http3/Http2 tests as well. But to get IIS.Functionals all the way down to 17 minutes we will need to split some of the Common.FunctionalTests into a separate, Common.StartupTests or something.

The only queue that matters it he windows 11 queue since that's the only queue these tests run on for the pr checks, there hasn't been basically much variance from what I've seen over 5-6 runs, as they split out wait/delay/queue time from actual machine time, for example

https://helix.dot.net/api/jobs/bf1d596a-36e0-44c0-b19c-19b1bf483897/workitems/IIS.FunctionalTests--net7.0?api-version=2019-06-17

I'm really only asking if we think its worth the additional split up of the shared functional tests into 4 more test projects to cut 7 minutes off our PR runs.

Today we have:
IIS.FunctionalTests
IIS.NewHandler.FunctionalTests
IIS.NewShim.FunctionalTests
IISExpress.FunctionalTests

We'd end up with something like

IIS.FunctionalTests
IIS.NewHandler.FunctionalTests
IIS.NewShim.FunctionalTests
IISExpress.FunctionalTests
IIS.Startup.FunctionalTests
IIS.NewHandler.Startup.FunctionalTests
IIS.NewShim.Startup.FunctionalTests
IISExpress.Startup.FunctionalTests
IIS.Http.FunctionalTests
IIS.ShadowCopy.Tests

[dougbu]

At the moment, the $(HelixTimeout) for all these projects (from FunctionalTest.props) is 01:15:00. If we're already down below that we're ahead 😺

The only queue that matters it he windows 11 queue since that's the only queue these tests run on for the pr checks

That's not the case if we're considering lowering the above timeout.

[HaoK]

Last time I looked at the kusto data we didn't have any jobs that took longer than 17 minutes in total other than IIS Functionals, so we can probably start with just 30 minutes timeout everywhere, and just remove the special timeout. Kusto is down right now so I can't check...

Actually it looks like most of the other test projects only include the startup/shutdown tests i was going to split out already, so it looks like we can probably get what we want just by splitting two more test files into IIS.LongTests, I'll give that a shot and see if that gets us below 20 minutes

[Tratcher]

You should be able to clean out some unneeded flags and dependencies that don't apply to the tests in these split projects.

[HaoK]

Okay I think this is the ideal split, since the startup/shutdown tests take 17 minutes, now we have 3 permutations of them which are the long pole, I think can get rid of the Http tests and put those back so we'll end up with:

IIS.FunctionalTests (-ShadowCopy, startup/shutdown), should be < 10 minutes
IIS.ShadowCopy.Tests ~9 min
IIS.NewHandler.FunctionalTests (unchanged, only Startup/Shutdown tests) ~17 min
IIS.NewShim.FunctionalTests (unchanged Startup/Shutdown tests) ~17 min
IIS.LongTests (only startup/shutdown tests) ~17 min

I'll go ahead and change the global helix job timeout to a consistent 30 min as part of this iteration before trying a few runs to see how things look, we should shave around 15 minutes from our average PRs I think

[HaoK]

Current results:
IIS.Functionals: 6m34s
IIS.LongTests: 17m53s
IIS.NewHandler: 16m
IIS.NewShim: 15m
IIS.ShadowCopy: 9m 44s
IIS.Express: 8m 51s

[Tratcher]

Nice. What were the times before?

[HaoK]

Nice. What were the times before?

IIS.Functionals 34m
IIS.NewHandlers, NewShim ~17m (these should be unchanged)

So we should gain around 17 minutes since we queue all the helix workitems at once

[HaoK]

Windows jobs on helix matrix look green https://dev.azure.com/dnceng/public/_build/results?buildId=1484808&view=logs&j=8c9dd6ae-5017-5a70-997c-c2386fed4e85&t=67839355-6a03-55d6-fbd7-5d56c2fd9ea1