slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Repository	Step Name	Console log	Pull Request
3001567	dotnet-sdk	TestBuild: macOS (arm64)	Log
1466394	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#66487
1466303	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#129447
1466309	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#54773
1466263	dotnet/maui	Controls Shell	Log	dotnet/maui#35933
1466277	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#67093
1466210	dotnet/maui	Controls CollectionView	Log
1466191	dotnet/maui	Controls ScrollView	Log	dotnet/maui#35878
1466212	dotnet/aspnetcore	Build: macOS arm64	Log
1466103	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#67098
3001383	dotnet-runtime	Upload Build Artifact BuildArtifacts_windows_arm64_Release_coreclr	Log
1466011	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#66778
1466004	dotnet/runtime	osx-arm64 Release CoreCLR_Libraries	Log
1465374	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#67183
1465937	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#67146
1465914	dotnet/runtime	osx-x64 release Runtime_Release	Log
1465875	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35924
1465889	dotnet/runtime	iossimulator-arm64 Release AllSubsets_NativeAOT_Smoke	Log	dotnet/runtime#128970
1465868	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34600
1465846	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#128768
1465808	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1465781	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
1465722	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#129442
1465711	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#67232
1465618	dotnet/maui	Controls CollectionView	Log
3001130	dotnet-performance	Performance maui_scenarios_ios Default Mono osx 26 x64 iPhone17 main	Log
3001115	dotnet-efcore	macOS	Log
1465346	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1465253	dotnet/maui	Controls Shell	Log	dotnet/maui#35932
1465238	dotnet/maui	Controls Shell	Log
3001023	dotnet-diagnostics	MacOS_arm64_Release	Log
3000959	dotnet-maui	Controls Shell	Log
1465197	dotnet/maui	Controls Layout	Log
1465188	dotnet/maui	Controls Shell	Log	dotnet/maui#35933
3000939	dotnet-maui	Controls Shell	Log
3000944	dotnet-efcore	macOS	Log
1465092	dotnet/sdk	TestBuild: macOS (x64)	Log
1464606	dotnet/maui	Controls Shell	Log	dotnet/maui#35898
1465071	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#128970
1465066	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#129389
1465015	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129434
1462260	dotnet/aspnetcore	Test: macOS	Log	dotnet/aspnetcore#67190
1464557	dotnet/maui	Controls Shell	Log	dotnet/maui#34869
1464930	dotnet/aspnetcore	Build: macOS x64	Log
1464898	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#67098
1457491	dotnet/aspnetcore	Test: macOS	Log	dotnet/aspnetcore#67119
1464427	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34532
1464792	dotnet/aspnetcore	Test: macOS	Log	dotnet/aspnetcore#65772
1464784	dotnet/aspnetcore	Test: macOS	Log	dotnet/aspnetcore#66487
1464772	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#67224
1464767	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#67214
3000762	dotnet-efcore	macOS	Log
1464372	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35912
1464366	dotnet/maui	Controls CollectionView	Log
1464680	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127750
1464640	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#129250
3000734	dotnet-sdk	TestBuild: macOS (arm64)	Log
3000537	dotnet-maui	Controls CollectionView	Log
1464658	dotnet/aspnetcore	Test: macOS	Log
3000719	dotnet-runtime	Performance ios_scenarios iOSCoreCLR R2R iOSStripSymbols osx x64 perfiphone17 net11.0	Log
1464615	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#128409
1464582	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#129334
1464603	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#67093
3000684	dotnet-sdk	TestBuild: macOS (arm64)	Log
1464426	dotnet/maui	Controls Shell	Log	dotnet/maui#35893
1464491	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#129043
1461141	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#66692
3000655	dotnet-sdk	TestBuild: macOS (arm64)	Log
1464482	dotnet/runtime	osx-x64 Release NativeAOT_Libs	Log
3000536	dotnet-maui	RunOniOS_MauiNativeAOT x64	Log
1464480	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
3000642	dotnet-runtime	iossimulator-x64 release NativeAOT	Log
1464433	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#67035
1464356	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#67074
1464353	dotnet/aspnetcore	Test: macOS	Log	dotnet/aspnetcore#66487
1464321	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_NativeAOT_Smoke	Log	dotnet/runtime#129416
3000516	dotnet-diagnostics	MacOS_x64_Release	Log
1464292	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#66832
1464268	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#128148
1464193	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129380
1464177	dotnet/aspnetcore	Test: macOS	Log
1463910	dotnet/maui	Controls CollectionView	Log
1463935	dotnet/maui	Controls Shell	Log
1464113	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#67006
1464109	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#66680
1463930	dotnet/runtime	osx-x64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129407
1463905	dotnet/aspnetcore	Test: macOS	Log	dotnet/aspnetcore#67207
1463842	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#67178
1463838	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#67059
1463829	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#67072
1463828	dotnet/runtime	maccatalyst-x64 Release AllSubsets_CoreCLR	Log
1463152	dotnet/aspnetcore	Test: macOS	Log	dotnet/aspnetcore#67194
1463578	dotnet/maui	Controls Shell	Log	dotnet/maui#35597
1463636	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#67006
1463630	dotnet/aspnetcore	Build: macOS x64	Log	dotnet/aspnetcore#66796
1463365	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33007
1463586	dotnet/aspnetcore	Build: macOS arm64	Log	dotnet/aspnetcore#66901
1463240	dotnet/maui	Controls CarouselView	Log
1463373	dotnet/maui	Controls CollectionView	Log	dotnet/maui#31820
1463443	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log	dotnet/runtime#128409
Displaying 100 of 2163 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
61	368	2163

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.