Increase timeout of Windows test job

[dougbu]

• no consistency in failures, maybe we have more tests now?
  • both of the runs below involved test failures but results were not uploaded
• https://dev.azure.com/dnceng/public/_build/results?buildId=768850
• https://dev.azure.com/dnceng/public/_build/results?buildId=768703

[dougbu]

Surprised to see the job that had been timing out succeeded in only 49 minutes because the two failing builds this morning logged test failures as well as the timeout. Will look at the *.dmp files from the failures before my final decision on this PR.

[dougbu]

I'm not seeing any test assemblies in the *.dmp files. The smaller dump for https://dev.azure.com/dnceng/public/_build/results?buildId=768850 shows code in VBCSCompiler.dll was running while that for https://dev.azure.com/dnceng/public/_build/results?buildId=768703 was running Microsoft.Build.dll code (with the compiler nowhere in sight).

@HaoK @BrennanConroy and @davidfowl will dumps be collected just because a process happens to be running when a build is cancelled❔ Alternatively is there a way to determine what the dotnet process is really waiting for❔

[dougbu]

Note the most recent rolling build was successful but didn't include any changes except to doc comments: fa2d86e...9f0ae10

[HaoK]

The new stuff for hangs and dumps was helix specific, so its not going to help for windows test jobs unfortunately

[dougbu]

🆗 Since the job usually finishes well before its current 180 minute timeout and just uploading a *.dmp file can take 15 minutes, I'm going to increase (double) the cancelTimeoutInMinutes instead. Hope is that will give job enough time to upload test results as well as any dumps.

[dougbu]

I meant artifacts/logs/

[dougbu]

@MattGal @ilyas1974 is something odd happening w/ https://dev.azure.com/dnceng/public/_build/results?buildId=769815❔ The job w/ the increased cancel timeout has been sitting in Queued state for a couple of hours now. Later builds seem to be getting Windows.Server.Amd64.VS2019.Open machines ahead of it.

[dougbu]

More to the point, is my best option cancelling that build and starting the pipeline again❔

[ilyas1974]

it appears that the build is currently in progress so I do not believe any additional action needs to be taken.

[dougbu]

Weird. I'm glad the job finally started running❕

[MattGal]

More to the point, is my best option cancelling that build and starting the pipeline again❔

If it's been > 90 minutes, definitely so. In those scenarios you're hitting https://github.com/dotnet/core-eng/issues/10136; in this situation we just don't have enough machines and AzDO has a hard 90 minute timeout they've offered to eventually extend, but who knows when...