slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
946582	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111539
946564	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112350
2639134	dotnet-emsdk	macOS_arm64	Log
946529	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
946512	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
946509	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
946429	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
945455	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#112302
946432	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log
946139	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1356
946410	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#112345
946389	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
946378	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112231
946332	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112342
946296	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112318
946326	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#111610
2638956	dotnet-binaryen	macOS_x64	Log
946285	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112337
946280	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112340
946244	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#112338
946218	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112336
946212	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112335
946204	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1357
946254	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log	dotnet/aspnetcore#60291
946096	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112331
946108	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112323
946101	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#112073
946072	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112256
946066	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111598
946035	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#107499
945920	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#112050
946042	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112308
946011	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#110819
946025	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#110272
945989	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112027
946001	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112322
945963	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#111759
945947	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
945950	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#112321
945941	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111646
945894	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112318
945911	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#112073
945901	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112320
2638724	dotnet-runtime	Libraries SuperPMI collection libraries_tests Checked coreclr osx arm64 Release	Log
945859	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112308
945844	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#110945
945826	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112073
945804	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
945782	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
945783	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
945778	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
945740	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
945741	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112027
945700	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112314
945677	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112313
945670	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111646
945666	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112295
945663	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#111598
945653	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#111982
945648	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#111410
2638416	dotnet-arcade-validation	Validate_Signing_MacOS Real_Signing	Log
2638414	dotnet-performance	Performance roslyn windows 22H2 arm64 Ampere 8.0	Log
945601	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112308
945573	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#110819
945570	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#107230
945565	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#112295
945541	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112256
945538	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
945498	dotnet/runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
945499	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
945512	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#112307
945503	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
945465	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112027
945459	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112306
945451	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111598
945437	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112209
945311	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
2637853	dotnet-performance	Performance mlnet ubuntu 2204 arm64 Ampere main	Log
944957	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log	dotnet/aspnetcore#60263
944670	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#110622
2637610	dotnet-efcore	macOS	Log
2637415	dotnet-performance	Performance mlnet ubuntu 2204 arm64 Ampere 8.0	Log
943797	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log
943759	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46527
943627	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log
2636981	dotnet-arcade-validation	MacOS Build_Release	Log
943388	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log	dotnet/aspnetcore#59952
943030	dotnet/performance	Performance imagesharp windows 22H2 x64 Open 9.0	Log
2636745	dotnet-performance	Performance scenarios Affinity_85 windows 22H2 arm64 Ampere 8.0	Log
2636743	dotnet-performance	Performance scenarios Affinity_85 windows 22H2 arm64 Ampere 9.0	Log
2636739	dotnet-performance	Performance scenarios Affinity_85 windows 22H2 arm64 Ampere main	Log
942983	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log	dotnet/aspnetcore#59952
942887	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46602
942871	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#46562
942773	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46575
942737	dotnet/sdk	Darwin Build_Release	Log	dotnet/sdk#46467
942780	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#43073
942777	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46601
942749	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46397
942747	dotnet/sdk	OSX_x64	Log	dotnet/sdk#46397
Displaying 100 of 1391 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
45	284	1391

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.