slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1430420	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#128457
1430381	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#128455
1430378	dotnet/runtime	osx-x64 Debug PALTests	Log
1430361	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#128447
2981062	dotnet-efcore	macOS	Log
1429978	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35476
1430230	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35566
2981021	dotnet-runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	#59556
2980960	dotnet-maui	RunOniOS_MauiRelease x64	Log
1429761	dotnet/maui	Controls Shell	Log	dotnet/maui#35385
2980949	dotnet-maui	RunOniOS_MauiNativeAOT ARM64	Log
1430176	dotnet/aspnetcore	Tests: macOS	Log
1430091	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127989
1429695	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35484
1430039	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127889
2980908	dotnet-maui	RunOniOS_BlazorRelease ARM64
1429988	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#128356
1429744	dotnet/runtime	Libraries Test Run checked coreclr osx arm64 Debug	Log	dotnet/runtime#128442
1429806	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#125437
1429707	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#128441
1429687	dotnet/runtime	Libraries Test Run checked coreclr osx arm64 Debug	Log	dotnet/runtime#128438
1429887	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#128360
1429892	dotnet/aspnetcore	Tests: macOS	Log
1429871	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log	dotnet/runtime#125437
1429647	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#53950
1429493	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35362
2980825	dotnet-maui	RunOniOS_BlazorRelease x64	Log
2980808	dotnet-maui	RunOniOS_MauiReleaseTrimFull x64	Log
1429722	dotnet/maui	Controls ScrollView	Log	dotnet/maui#35393
2980840	dotnet-runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	#60853
1429468	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#54355
1429758	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66753
1429700	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR	Log	dotnet/runtime#128409
1429699	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log	dotnet/runtime#128408
1429579	dotnet/maui	Controls Shell	Log	dotnet/maui#35544
1429607	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#128436
1429602	dotnet/runtime	osx-arm64 checked	Log
1429584	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#128434
1429305	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#54371
1429201	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#54388
1429511	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#128433
1429526	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log
1429460	dotnet/runtime	osx-x64 Debug Installer_Build_And_Test	Log	dotnet/runtime#128351
1429503	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#127894
1429510	dotnet/runtime	osx-x64 Debug PALTests	Log	dotnet/runtime#127894
1427870	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#127762
1429423	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#128430
1429413	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128428
1429402	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128403
1429420	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128429
1429447	dotnet/xharness	AOT Helix Tests Build_Debug	Log	dotnet/xharness#1597
2980661	dotnet-runtime	osx-arm64 release bluebird	Log
1429386	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128105
1429397	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128369
1427741	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#54381
1429375	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128281
1429371	dotnet/runtime	osx-x64 Debug Installer_Build_And_Test	Log	dotnet/runtime#128394
1429324	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128422
1429367	dotnet/runtime	osx-x64 Release NativeAOT	Log	dotnet/runtime#128427
1429311	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128091
1429273	dotnet/runtime	osx-x64 Debug Installer_Build_And_Test	Log	dotnet/runtime#128426
1429257	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128419
1429245	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128365
1429228	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#127894
1429323	dotnet/runtime	osx-x64 Release Libraries_Release_CoreCLR	Log
1429210	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128313
1429187	dotnet/runtime	coreclr Pri1 Runtime Tests Run R2R_CG2 osx x64 checked	Log
1429166	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128229
1429127	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128423
1429182	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128382
1429269	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66748
1429123	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128415
1428998	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#54394
1429070	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128399
1429178	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#65638
1429020	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128306
1428985	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128388
1428969	dotnet/runtime	osx-x64 Debug Installer_Build_And_Test	Log	dotnet/runtime#128383
1429116	dotnet/aspnetcore	Tests: macOS	Log
1428396	dotnet/runtime	coreclr Pri1 Runtime Tests Run R2R_CG2 osx x64 checked	Log
1428955	dotnet/runtime	osx-x64 Debug Installer_Build_And_Test	Log	dotnet/runtime#124173
1428907	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128411
1429080	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#54384
1428393	dotnet/runtime	osx-x64 Release Libraries_Release_CoreCLR	Log	dotnet/runtime#128409
1428895	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128235
1428882	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#128224
1429056	dotnet/sdk	TestBuild: macOS (x64)	Log
1428825	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#117261
1428224	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#54380
1428781	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128225
1428776	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128402
1428795	dotnet/runtime	osx-x64 Release Installer_Build_And_Test	Log
1428980	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1428977	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1428438	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#54386
1428952	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#54393
1428453	dotnet/maui	Controls Cells,CheckBox,ContextActions,CustomRenderers,DatePicker,Dispatcher,DisplayAlert,DisplayPrompt,DragAndDrop	Log
1428654	dotnet/runtime	coreclr Pri1 Runtime Tests Run R2R_CG2 osx x64 checked	Log
1428704	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#127818
1428227	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log
Displaying 100 of 1655 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
105	503	1655

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.