slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Repository	Step Name	Console log	Pull Request
1535813	dotnet/runtime	osx-x64 Debug PALTests	Log
1535800	dotnet/runtime	osx-x64 Release Installer_Build_And_Test	Log
1535798	dotnet/aspnetcore	Tests: macOS	Log
1535763	dotnet/runtime	coreclr Pri0 Runtime Tests Run R2R_CG2 osx x64 checked	Log
1535750	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#131660
1535667	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#131654
1535725	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#131691
1535743	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#131692
1535755	dotnet/maui	Controls Shell	Log
1535669	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#131655
1535640	dotnet/runtime	maccatalyst-x64 Release AllSubsets_NativeAOT	Log
1535591	dotnet/maui	Controls CollectionView	Log
1535488	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#131654
3036212	dotnet-maui	RunOniOS_BlazorDebug ARM64	Log
3036222	dotnet-maui	RunOniOS_BlazorRelease_CoreCLR ARM64	Log
1535320	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36986
3036196	dotnet-maui	RunOniOS_BlazorDebug_CoreCLR ARM64
1535241	dotnet/aspnetcore	Tests: macOS	Log
1535067	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#131657
1535048	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#131654
1534997	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36988
1534946	dotnet/maui	Controls CollectionView	Log
1534953	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1534930	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#131652
3035806	dotnet-runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	#63382
1534890	dotnet/maui	Controls Shell	Log	dotnet/maui#36130
1534582	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
3035982	dotnet-maui	MultiProject macOS	Log
1534830	dotnet/runtime	osx-arm64 checked	Log
3035949	dotnet-maui	RunOniOS_MauiRelease_CoreCLR ARM64	Log
1534675	dotnet/maui	Controls Shell	Log
1534642	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1534639	dotnet/runtime	osx-x64 Release Libraries_CheckedCoreCLR	Log
1534629	dotnet/runtime	maccatalyst-x64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#131634
1534609	dotnet/aspnetcore	Tests: macOS	Log
3035829	dotnet-maui	RunOniOS_BlazorRelease ARM64	Log
3035832	dotnet-maui	RunOniOS_MauiRelease ARM64	Log
1534496	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36911
3035830	dotnet-maui	Controls Shell	Log
3035812	dotnet-maui	RunOniOS_BlazorDebug_CoreCLR ARM64	Log
1534472	dotnet/maui	Controls Shell	Log	dotnet/maui#36884
1534459	dotnet/maui	Controls Shell	Log
1534449	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34903
1534435	dotnet/maui	Controls Shell	Log
1534217	dotnet/maui	Controls Shell	Log	dotnet/maui#36935
1534347	dotnet/aspnetcore	Tests: macOS	Log
1534219	dotnet/maui	Controls Shell	Log
1534190	dotnet/aspnetcore	Tests: macOS	Log
1534027	dotnet/maui	Controls CollectionView	Log
1534007	dotnet/maui	Controls Shell	Log
1534015	dotnet/aspnetcore	Tests: macOS	Log
3035503	dotnet-maui	RunOniOS_BlazorDebug_CoreCLR ARM64	Log
1533945	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#131623
1533923	dotnet/maui	Controls Shell	Log	dotnet/maui#36949
3035453	dotnet-maui	Build macOS	Log
3035465	dotnet-maui	RunOniOS_MauiRelease_CoreCLR ARM64	Log
3035467	dotnet-maui	RunOniOS_BlazorDebug_CoreCLR ARM64	Log
1533719	dotnet/runtime	osx-x64 release CdacDumpTest	Log	dotnet/runtime#131024
3035440	dotnet-efcore	macOS	Log
1533768	dotnet/maui	Controls Shell	Log
1533736	dotnet/maui	Controls CollectionView	Log
1533714	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#131024
1533709	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log
1533707	dotnet/aspnetcore	Tests: macOS	Log
1533498	dotnet/runtime	osx-x64 Debug Installer_Build_And_Test	Log	dotnet/runtime#130512
3035380	dotnet-maui	RunOniOS_MauiRelease_CoreCLR ARM64
1533633	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#131307
1533629	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130920
1533450	dotnet/runtime	osx-x64 release CdacDumpTest	Log	dotnet/runtime#131571
1533409	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#131599
1533414	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#129688
1533427	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#131559
1533364	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#131586
1533597	dotnet/runtime	osx-x64 Debug PALTests	Log
1533582	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#55455
1533397	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#131442
1533380	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log
1533564	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#131538
1533545	dotnet/maui	Controls Shell	Log	dotnet/maui#36654
3035331	dotnet-maui	RunOniOS_MauiNativeAOT ARM64	Log
3035329	dotnet-maui	RunOniOS_BlazorDebug_CoreCLR ARM64	Log
3035312	dotnet-maui	RunOniOS_MauiRelease_CoreCLR ARM64	Log
1533533	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#131578
1533324	dotnet/runtime	coreclr Pri1 Runtime Tests Run osx x64 checked	Log
1533508	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#130985
1533442	dotnet/maui	Controls CollectionView	Log
3035268	dotnet-maui	RunOniOS_MauiNativeAOT ARM64	Log
1533385	dotnet/runtime	osx-x64 Release NativeAOT_Libs	Log
1533383	dotnet/runtime	osx-x64 Release Libraries_Release_CoreCLR	Log
3035201	dotnet-maui	RunOniOS_MauiDebug x64	Log
1533386	dotnet/aspnetcore	Tests: macOS	Log
1533309	dotnet/maui	Controls Shell	Log
3035203	dotnet-maui	Controls Layout	Log
1533287	dotnet/runtime	osx-x64 Release NativeAOT_Libs	Log
3035148	dotnet-maui	RunOniOS_BlazorRelease_CoreCLR ARM64	Log
3035183	dotnet-runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	#63384
3035182	dotnet-runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	#63383
3035181	dotnet-runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	#63382
1533253	dotnet/aspnetcore	Tests: macOS	Log
3035122	dotnet-maui	RunOniOS_MauiDebug_CoreCLR ARM64	Log
Displaying 100 of 1381 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
30	288	1381

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.