slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2758245	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
1106058	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#118037
1106105	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#49932
1105151	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#118029
2757072	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1104486	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono_RuntimeTests	Log
2756808	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2756412	dotnet-runtime	Performance ios_scenarios iOSNativeAOT JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2756261	dotnet-performance	Performance roslyn ubuntu 2204 arm64 Ampere 8.0	Log
2755893	dotnet-performance	Performance powershell ubuntu 2204 x64 Tiger 8.0	Log
1102514	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#117700
1102553	dotnet/arcade	Windows_NT Build_Release	Log	dotnet/arcade#15989
1102559	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#117950
2755736	dotnet-sdk	TestBuild: macOS (x64)	Log
1102475	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log
2754777	dotnet-runtime	Performance ios_scenarios iOSNativeAOT JIT osx x64 perfiphone12mini net10.0	Log
2754972	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2754548	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2755626	dotnet-sdk	AoT: macOS (x64)	Log
1101389	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
1100229	dotnet/runtime	Libraries Test Run release coreclr osx x64 Debug	Log	dotnet/runtime#117817
1100214	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#117858
1100220	dotnet/runtime	osx-x64 Release CoreCLR_Release	Log
1099458	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
2752908	dotnet-performance	Performance powershell ubuntu 2204 x64 Tiger main	Log
2752555	dotnet-performance	Performance powershell ubuntu 2204 x64 Tiger 9.0	Log
2752547	dotnet-performance	Performance nativeaot_scenarios ubuntu 2204 x64 Tiger 8.0	Log
2752440	dotnet-performance	Performance nativeaot_scenarios ubuntu 2204 x64 Tiger 8.0	Log
2752114	dotnet-performance	Performance roslyn ubuntu 2204 x64 Tiger 9.0	Log
1097408	dotnet/performance	Performance akadeindexedset windows 24H2 x64 Open main	Log	dotnet/performance#4862
1097295	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#49829
1097241	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#117621
2751595	dotnet-performance	Performance nativeaot_scenarios ubuntu 2204 x64 Tiger 8.0	Log
2751723	dotnet-performance	Performance powershell ubuntu 2204 x64 Tiger 9.0	Log
1097174	dotnet/performance	Performance sdk_scenarios windows 24H2 x64 Open main	Log
1097089	dotnet/arcade	Linux Build_Debug	Log	dotnet/arcade#15967
1097028	dotnet/runtime	Libraries Test Run release coreclr osx x64 Debug	Log	dotnet/runtime#117707
1097052	dotnet/arcade	Linux Build_Debug	Log	dotnet/arcade#15974
1097007	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#117583
1096963	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#116882
1097033	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#49822
1096980	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1096885	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#49818
1096937	dotnet/performance	Performance scenarios ubuntu 2204 x64 Open 8.0	Log
2751564	dotnet-sdk	Darwin	Log	#51661
2751563	dotnet-sdk	Darwin	Log	#51575
2751562	dotnet-sdk	Darwin_AoT_Tests	Log	#51662
2751495	dotnet-performance	Performance nativeaot_scenarios ubuntu 2204 x64 Tiger 9.0	Log
2751275	dotnet-performance	Performance powershell ubuntu 2204 x64 Tiger main	Log
1096198	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log	dotnet/aspnetcore#62735
2751158	dotnet-runtime	ios-arm64 release iOSMono	Log
2750895	dotnet-performance	Performance roslyn ubuntu 2204 x64 Tiger 8.0	Log
2750845	dotnet-sdk	Darwin	Log	#51575
2750655	dotnet-performance	Performance nativeaot_scenarios ubuntu 2204 x64 Tiger 8.0	Log
2750577	dotnet-runtime	tvossimulator-x64 release Mono	Log
1095043	dotnet/xharness	Helix Tests Build_Debug	Log
1095041	dotnet/xharness	Helix Tests Build_Debug	Log
2750387	dotnet-performance	Performance powershell ubuntu 2204 x64 Tiger 9.0	Log
1094869	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1439
2750131	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2750130	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2750099	dotnet-sdk	Darwin	Log	#51527
2750018	dotnet-performance	Performance powershell ubuntu 2204 x64 Tiger 8.0	Log
2749976	dotnet-performance	Performance nativeaot_scenarios ubuntu 2204 x64 Tiger main	Log
1093567	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
2748584	dotnet-performance	Performance bepuphysics ubuntu 2204 x64 Viper main	Log
2748353	dotnet-performance	Performance bepuphysics ubuntu 2204 x64 Viper main	Log
2748318	dotnet-sdk	Darwin	Log	#51480
2748319	dotnet-sdk	Darwin_AoT_Tests	Log	#51482
2747976	dotnet-sdk	Darwin	Log	#51450
2747972	dotnet-performance	Performance bepuphysics ubuntu 2204 x64 Viper main	Log
2747706	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2747700	dotnet-sdk	Darwin	Log	#51450
2747698	dotnet-sdk	Darwin	Log	#51451
2747576	dotnet-sdk	TestBuild: macOS (x64)	Log	#51332
2747597	dotnet-performance	Performance bepuphysics ubuntu 2204 x64 Viper main	Log
1091545	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
2747145	dotnet-performance	Performance bepuphysics ubuntu 2204 x64 Viper main	Log
1090931	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#117498
1090889	dotnet/runtime	iossimulator-arm64 Release AllSubsets_Mono	Log
2746696	dotnet-performance	Performance bepuphysics ubuntu 2204 x64 Viper main	Log
1090290	dotnet/runtime	iossimulator-arm64 Release AllSubsets_Mono	Log
2746552	dotnet-sdk	Darwin	Log	#51331
2746550	dotnet-sdk	Darwin_AoT_Tests	Log	#51329
1089691	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#117023
2746182	dotnet-performance	Performance mlnet ubuntu 2204 arm64 Ampere 9.0	Log
1089706	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#116411
1087609	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#117330
1089548	dotnet/runtime	tvossimulator-x64 Release AllSubsets_Mono	Log
2745977	dotnet-sdk	AoT: macOS (x64)	Log	#51327
2745952	dotnet-sdk	Darwin_AoT_Tests	Log	#51330
2745951	dotnet-sdk	Darwin	Log	#51331
2745950	dotnet-sdk	Darwin	Log	#51329
2745897	dotnet-sdk	AoT: macOS (x64)	Log	#51332
2745853	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2745678	dotnet-performance	Performance bepuphysics ubuntu 2204 x64 Viper main	Log
1088858	dotnet/runtime	iossimulator-x64 Release AllSubsets_Mono	Log
1088670	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#117374
2745437	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
1088664	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#117420
Displaying 100 of 397 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
3	23	397

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.