IIS functional tests sometimes corrupt IIS config

[HaoK]

Most IIS functional tests that are quarantined, are a result of some form of:

System.AggregateException : Operation did not succeed after 12 retries (Sequence contains more than one element) (Sequence contains more than one element) (Sequence contains more than one element) (Sequence contains more than one element) (Sequence contains more than one element) (Sequence contains more than one element) (Sequence contains more than one element) (Sequence contains more than one element) (Sequence contains more than one element) (Sequence contains more than one element) (Sequence contains more than one element) (Sequence contains more than one element)
---- System.InvalidOperationException : Sequence contains more than one element
---- System.InvalidOperationException : Sequence contains more than one element
---- System.InvalidOperationException : Sequence contains more than one element
---- System.InvalidOperationException : Sequence contains more than one element
---- System.InvalidOperationException : Sequence contains more than one element
---- System.InvalidOperationException : Sequence contains more than one element
---- System.InvalidOperationException : Sequence contains more than one element
---- System.InvalidOperationException : Sequence contains more than one element
---- System.InvalidOperationException : Sequence contains more than one element
---- System.InvalidOperationException : Sequence contains more than one element
---- System.InvalidOperationException : Sequence contains more than one element
---- System.InvalidOperationException : Sequence contains more than one element

Which looks like a result of apphost config getting corrupted in a bad state, local retries will not help this situation.

This is the main tracking issue for finding the root cause

[HaoK]

From @dougbu in #42658

I looked into this a bit and suspect IISDeployer isn't doing enough to reliably configure the IIS site. The code in its WaitUntilSiteStarted(...) method confirms IIS only has a single site and app pool. And the site only has a single virtual directory. But the VD seems to be "stuck" w/ its default physical path.

On one hand, many IIS tests use the same ApplicationDeployer implementation. On the other, this is the only test that sets a custom initializationPage configuration.

Just noticed 07abf1f went in on the 7th. Could that have restarted the problems❔ We didn't see this failure between 27 April and 8 July.

See also the Kusto data:

let start = 90d;                   // Timespan value
let end = 0m;                      // Timespan value
let repo = "dotnet/aspnetcore";    // Optional: set to empty string if you want results from all repositories
let testName="";                   // Optional: set to empty string if you want results about all tests in the repo
let message="Wrong physical path"; // Controls use of has_all()
AzureDevOpsTests
| where iff(repo == "", Repository == Repository, Repository == repo)
    and iff(testName == "", TestName == TestName, TestName == testName)
    and iff(message == "", Message == Message, Message has_all("Wrong", "physical", "path"))
    and Branch == 'main'
    and Outcome == "Failed" and RunCompleted >= ago(start) and RunCompleted <= ago(end)
| project Repository, BuildId, RunCompleted, TestName, Message
| order by RunCompleted asc