slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1118071	dotnet/runtime	iossimulator-x64 Release AllSubsets_Mono_RuntimeTests	Log	dotnet/runtime#118507
2767816	dotnet-runtime	Performance ios_scenarios iOSNativeAOT JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1117683	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log	dotnet/runtime#118525
1117647	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#118523
1117517	dotnet/runtime	ios-arm64 Release AllSubsets_Mono	Log	dotnet/runtime#116419
1117484	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log	dotnet/runtime#118507
2766861	dotnet-runtime	Performance ios_scenarios iOSNativeAOT JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2766692	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2766678	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2766661	dotnet-sdk	Darwin	Log	#52072
1116108	dotnet/runtime	osx-x64 release Installer_Build_And_Test		dotnet/runtime#118464
2766474	dotnet-sdk	Darwin	Log	#52071
2766473	dotnet-sdk	Darwin_AoT_Tests	Log	#52072
2766398	dotnet-sdk	TestBuild: macOS (x64)	Log	#52073
2766397	dotnet-sdk	AoT: macOS (x64)	Log	#51985
2766356	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1115332	dotnet/runtime	iossimulator-arm64 Release AllSubsets_Mono	Log	dotnet/runtime#117593
2766008	dotnet-sdk	Darwin_AoT_Tests	Log	#52072
2766006	dotnet-sdk	Darwin	Log	#52070
1114835	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#118415
2765656	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
1114865	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#118338
1114800	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log	dotnet/aspnetcore#63135
2765191	dotnet-performance	Performance roslyn ubuntu 2204 x64 Tiger 8.0	Log
1114128	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#49724
1113958	dotnet/sdk	Darwin Build_Release	Log	dotnet/sdk#49852
1113922	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#49724
1113921	dotnet/sdk	Darwin Build_Release	Log	dotnet/sdk#50041
1113918	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#49822
2765019	dotnet-runtime	Performance ios_scenarios iOSNativeAOT JIT osx x64 perfiphone12mini net10.0	Log
2764962	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2764943	dotnet-sdk	Darwin	Log	#52071
1113804	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#49828
1113791	dotnet/sdk	Darwin Build_Release	Log	dotnet/sdk#50044
1113783	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#50041
1113780	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#49822
1113777	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#49724
1113771	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#49852
1113770	dotnet/sdk	Darwin Build_Release	Log	dotnet/sdk#49853
1113768	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#49843
1113733	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#49889
2764854	dotnet-sdk	AoT: macOS (x64)	Log	#52073
2764729	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2764717	dotnet-sdk	TestBuild: macOS (x64)	Log	#52073
1113358	dotnet/runtime	Libraries Test Run release coreclr osx x64 Debug	Log	dotnet/runtime#118306
2763939	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2763908	dotnet-performance	Performance roslyn windows 22H2 arm64 Ampere main	Log
1112288	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#117845
1112277	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#117997
1112282	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#118241
2763703	dotnet-performance	Performance roslyn windows 22H2 arm64 Ampere main	Log
2763566	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2763483	dotnet-performance	Performance roslyn windows 22H2 arm64 Ampere 8.0	Log
2763398	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2763349	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
1112022	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#118165
1112018	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#118241
1112015	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#118302
2763287	dotnet-performance	Performance mlnet windows 22H2 arm64 Ampere 9.0	Log
2763092	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2763028	dotnet-performance	Performance roslyn windows 22H2 arm64 Ampere main	Log
1111698	dotnet/runtime	Libraries Test Run release coreclr osx x64 Debug	Log	dotnet/runtime#118256
2762999	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2762894	dotnet-performance	Performance scenarios Affinity_85 windows 22H2 arm64 Ampere main	Log
2762285	dotnet-sdk	TestBuild: macOS (x64)	Log
1110732	dotnet/runtime	ios-arm64 Release AllSubsets_Mono	Log	dotnet/runtime#117398
1110442	dotnet/runtime	ios-arm64 Release AllSubsets_Mono	Log	dotnet/runtime#118207
1110710	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log
2762259	dotnet-sdk	TestBuild: macOS (x64)	Log
2762239	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
1110544	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log	dotnet/runtime#118232
1110673	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_NativeAOT_RuntimeTests_AppSandbox	Log	dotnet/runtime#118037
1110660	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#118245
1110655	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log	dotnet/runtime#118228
1110637	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log	dotnet/runtime#118037
1110642	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log
1110631	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#118180
2762182	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2762183	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1110602	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#118246
2762132	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1110573	dotnet/runtime	osx-x64 Debug PALTests	Log
1110560	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#50000
1110481	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#116157
1110523	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#118190
1110520	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log	dotnet/runtime#117788
1110510	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#50016
2762065	dotnet-sdk	TestBuild: macOS (x64)	Log
1110466	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#118241
1110460	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#50015
1110102	dotnet/runtime	Libraries Test Run release coreclr osx x64 Debug	Log	dotnet/runtime#118227
1110434	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#118050
1110361	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#118239
2762001	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1110388	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#49843
1110264	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#50000
1110344	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#118027
1110325	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#118007
1110034	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log	dotnet/runtime#118221
2761955	dotnet-performance	Performance roslyn windows 22H2 arm64 Ampere 8.0	Log
Displaying 100 of 179 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	45	179

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.