The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2696700	dotnet-performance	Send job to Helix (Unix)	Log
2696690	dotnet-dotnet	🛡 Guardian: CredScan (Source)	Log
2696687	dotnet-arcade	Publish packages, blobs and symbols	Log
2696686	dotnet-arcade	Publish packages, blobs and symbols	Log
2696685	dotnet-arcade	Publish packages, blobs and symbols	Log
2696680	dotnet-dotnet	🛡 Guardian: PoliCheck	Log
2696568	dotnet-roslyn	Publish Using Darc	Log
2696575	dotnet-roslyn	Publish Using Darc	Log
2696570	dotnet-roslyn	Publish Using Darc	Log
2696657	dotnet-arcade	Publish packages, blobs and symbols	Log
2696656	dotnet-arcade	Publish packages, blobs and symbols	Log
2696654	dotnet-arcade	Publish packages, blobs and symbols	Log
2696653	dotnet-arcade	Publish packages, blobs and symbols	Log
2696645	dotnet-arcade	Publish packages, blobs and symbols	Log
2696649	dotnet-runtime	Send AssemblyNameInfoFuzzer to OneFuzz	Log
2696640	dotnet-arcade	Publish packages, blobs and symbols	Log
2696639	dotnet-arcade	Publish packages, blobs and symbols	Log
2696515	dotnet-sdk	Publish Using Darc	Log
2696634	dotnet-arcade	Publish packages, blobs and symbols	Log
2696592	dotnet-razor	Publish Using Darc	Log
2696594	dotnet-razor	Publish Using Darc	Log
2696595	dotnet-razor	Publish Using Darc	Log
2696590	dotnet-roslyn	Publish Using Darc	Log
2696632	dotnet-arcade	Publish packages, blobs and symbols	Log
2696347	dotnet-diagnostics	Publish Using Darc	Log
2696571	dotnet-razor	Publish Using Darc	Log
2696573	dotnet-razor	Publish Using Darc	Log
2696618	dotnet-arcade	Publish packages, blobs and symbols	Log
2696574	dotnet-razor	Publish Using Darc	Log
1027686	dotnet/runtime	AOT compile CoreCLR tests	Log
2696566	dotnet-fsharp	CmdLine	Log
2696550	dotnet-dotnet	🛡 Guardian: Post Analysis	Log
2696517	dotnet-runtime	Send job to Helix (Windows)	Log
2696530	dotnet-arcade	Publish packages, blobs and symbols	Log
2696522	dotnet-arcade	Publish packages, blobs and symbols	Log
2696464	dotnet-dotnet-monitor	Publish Using Darc	Log
2696513	dotnet-arcade	Publish packages, blobs and symbols	Log
2696445	dotnet-arcade	Publish packages, blobs and symbols	Log
2696435	dotnet-arcade	Publish packages, blobs and symbols	Log
2696382	dotnet-runtime	Send job to Helix (Unix)	Log
2696396	dotnet-performance	Send job to Helix (Unix)	Log
2696404	dotnet-arcade	Publish packages, blobs and symbols	Log
1027561	dotnet/performance	Send job to Helix (Unix)	Log
1027544	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115081
1027554	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115083
1027540	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#114629
1027536	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115068
2696351	dotnet-fsharp	Generate delegation SAS Token for dotnetbuilds/internal	Log
2696353	dotnet-fsharp	Checkout dotnet-fsharp@UpdateNuget_12.0 to s	Log
2696341	dotnet-arcade	Publish packages, blobs and symbols	Log
1027507	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#114675
1027501	dotnet/runtime	Send tests to Helix (Windows)	Log
1027502	dotnet/runtime	AOT compile CoreCLR tests	Log
2696306	dotnet-arcade	Publish packages, blobs and symbols	Log
2696268	dotnet-runtime	Send job to Helix (Windows)	Log
2696282	dotnet-arcade	Publish packages, blobs and symbols	Log
2696279	dotnet-arcade	Publish packages, blobs and symbols	Log
2696263	dotnet-arcade	Publish packages, blobs and symbols	Log
2696214	dotnet-roslyn	Publish Using Darc	Log
2696216	dotnet-arcade	Publish packages, blobs and symbols	Log
2696154	dotnet-runtime	Send job to Helix (Windows)	Log
2696167	dotnet-performance	Send job to Helix (Unix)	Log
1027362	dotnet/runtime	Send to Helix	Log
2696126	dotnet-runtime	Send AssemblyNameInfoFuzzer to OneFuzz	Log
1027318	dotnet/runtime	AOT compile CoreCLR tests	Log
1027308	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#76906
2696034	dotnet-runtime	Send job to Helix (Windows)	Log
2695960	dotnet-runtime	🛡 Guardian: AntiMalware Scanner (Binary)	Log
2695874	dotnet-performance	Send job to Helix (Unix)	Log
1026950	dotnet/runtime	Send to Helix	Log
2695860	dotnet-runtime	Send job to Helix (Windows)	Log
1027037	dotnet/runtime	Send to Helix	Log
2695055	dotnet-efcore	Send job to helix	Log
2695782	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
2695768	dotnet-aspnetcore	Run build.sh	Log	#49512
1026883	dotnet/runtime	AOT compile CoreCLR tests	Log
2695715	dotnet-aspnetcore	🛡 Guardian: AntiMalware Scanner (Binary)	Log	#49512
2695707	dotnet-aspnetcore	🛡 Guardian: Set BinSkim targets (1ES PT)	Log	#49512
2695691	dotnet-aspnetcore	🛡 Generate SBoM Manifest	Log	#49512
2695675	dotnet-aspnetcore	Build Installers	Log	#49512
2695661	dotnet-aspnetcore	🛡 Component Governance	Log	#49512
2695624	dotnet-runtime	Send job to Helix (Unix)	Log
2695652	dotnet-aspnetcore	Validate Hosted Pool Information (1ES PT)	Log	#49512
2695611	dotnet-sdk	Run Tests in Helix and non Helix in parallel	Log	#49522
1026669	dotnet/runtime	AOT compile CoreCLR tests	Log
1026588	dotnet/runtime	Send job to Helix (Unix)	Log	dotnet/runtime#115036
2695471	dotnet-runtime	Send job to Helix (Unix)	Log
1026416	dotnet/runtime	Send to Helix	Log
2695462	dotnet-performance	Send job to Helix (Unix)	Log
1026380	dotnet/runtime	AOT compile CoreCLR tests	Log
2695316	dotnet-runtime	Send job to Helix (Unix)	Log
2694727	dotnet-dotnet	Publish Using Darc	Log
1026259	dotnet/roslyn	Run BuildValidator	Log
2695004	dotnet-dotnet	🛡 Guardian: CredScan (Binary)	Log
2695127	dotnet-runtime	Send job to Helix (Unix)	Log
2695086	dotnet-performance	Send job to Helix (Unix)	Log
2695066	dotnet-runtime	Checkout dotnet-runtime@refs/pull/49479/merge to s	Log	#49479
1026043	dotnet/runtime	AOT compile CoreCLR tests	Log
2695045	dotnet-aspnetcore	Secure Supply Chain Analysis (auto-injected by policy)	Log	#49477
2695030	dotnet-efcore	CmdLine@2	Log
Displaying 100 of 1229 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
57	308	1229

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)