slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1391099	dotnet/maui	Controls CollectionView	Log
2957619	dotnet-runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	#59556
1391092	dotnet/maui	Controls Layout	Log	dotnet/maui#35072
2957531	dotnet-maui	RunOniOS_BlazorDebug x64	Log
1391160	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#54040
1391078	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126955
2957474	dotnet-maui	RunOniOS_MauiReleaseTrimFull x64	Log
1390990	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34637
1391026	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66383
1390971	dotnet/maui	Controls Entry	Log	dotnet/maui#34632
1390872	dotnet/maui	Controls Shell	Log
1390890	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#61667
1390611	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35011
1390623	dotnet/maui	Controls Layout	Log	dotnet/maui#34997
1390620	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33312
1390617	dotnet/maui	Controls CollectionView	Log	dotnet/maui#32674
1390776	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127268
1390512	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1390745	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127074
1390727	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127264
1390531	dotnet/maui	Controls Shell	Log
1390662	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126838
1390615	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34614
2957291	dotnet-maui	RunOniOS_MauiDebug x64	Log
2957283	dotnet-maui	RunOniOS_BlazorRelease x64	Log
1390570	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1390552	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#126947
1390549	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66409
1390445	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log	dotnet/runtime#127192
1390015	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35061
1390399	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1390330	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127252
1390377	dotnet/aspnetcore	Tests: macOS	Log
2957154	dotnet-performance	Performance powershell ubuntu 2204 x64 Viper main	Log
1390246	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log	dotnet/runtime#127250
1390193	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126853
1390069	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127043
1389981	dotnet/maui	Controls Shell	Log
2956989	dotnet-maui	Controls CollectionView	Log
2956988	dotnet-maui	RunOniOS_MauiNativeAOT ARM64	Log
1389985	dotnet/aspnetcore	Tests: macOS	Log
1389943	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127194
1389865	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#53947
1389723	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127188
1389725	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127188
1389281	dotnet/maui	Controls Layout	Log	dotnet/maui#35056
1389678	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127228
2956892	dotnet-performance	Performance scenarios windows Win11 x64 Viper main	Log
1389405	dotnet/runtime	Libraries Test Run release coreclr osx arm64 Debug	Log	dotnet/runtime#127058
1389602	dotnet/maui	Controls Layout	Log	dotnet/maui#34600
2956713	dotnet-maui	Controls Layout	Log
1389618	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127229
1389616	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127229
1389550	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#126385
1389632	dotnet/sdk	TestBuild: macOS (arm64)	Log
1389538	dotnet/maui	Controls Shell	Log	dotnet/maui#35061
2956833	dotnet-performance	Performance maui_scenarios_ios_innerloop Default Mono_InnerLoop_Simulator osx 15 x64 iPhone13 main	Log
1388819	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log	dotnet/runtime#126838
2956724	dotnet-maui	RunOniOS_MauiDebug_CoreCLR ARM64	Log
1389354	dotnet/maui	Controls Shell	Log
2956725	dotnet-maui	Controls Layout	Log
1389371	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#125559
2956704	dotnet-maui	RunOniOS_MauiNativeAOT ARM64	Log
1389315	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127188
1389305	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127193
2956726	dotnet-maui	Build Device Tests (Mono)	Log
1389293	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126093
1389285	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127210
1389270	dotnet/maui	Controls ViewBaseTests,VisualStateManager,Window	Log
2956585	dotnet-maui	Controls CollectionView	Log
1389302	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log
2956670	dotnet-maui	RunOniOS_MauiDebug x64	Log
1388870	dotnet/maui	Controls Layout	Log
1389193	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127221
1388730	dotnet/maui	Controls Layout	Log	dotnet/maui#31672
1389118	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127216
1388912	dotnet/runtime	Libraries Test Run checked coreclr osx arm64 Debug	Log	dotnet/runtime#127081
2956584	dotnet-maui	RunOniOS_MauiRelease x64	Log
1388951	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#53947
1388618	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34615
1388619	dotnet/maui	Controls Shell	Log	dotnet/maui#34637
1388732	dotnet/maui	Controls Shell	Log	dotnet/maui#34737
1388792	dotnet/roslyn	Build_macOS_Debug	Log
1388736	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127060
1388718	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#127206
1388617	dotnet/maui	Controls Layout	Log	dotnet/maui#34632
2956458	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1388586	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66200
2956384	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1388552	dotnet/aspnetcore	Tests: macOS	Log
1388519	dotnet/aspnetcore	Tests: macOS	Log
1388505	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
2956355	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2956301	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1387757	dotnet/maui	Controls CollectionView	Log
1388248	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log	dotnet/runtime#127192
2956248	dotnet-performance	Performance powershell ubuntu 2204 x64 Viper main	Log
2956203	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1388054	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127178
1388032	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127033
Displaying 100 of 1363 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
43	325	1363

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.