The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2787432	dotnet-dotnet	Build (azurelinuxX64CrossContainer)	Log
1140683	dotnet/runtime	Send tests to Helix (Unix)	Log
1141002	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#79929
2787324	dotnet-sdk	Prepare dotnet/msbuild for formatting	Log	#52957
2787316	dotnet-workload-versions	🛡 Component Governance	Log
1140777	dotnet/runtime	Send to Helix	Log	dotnet/runtime#119413
1140820	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#50445
1140724	dotnet/runtime	Send to Helix	Log	dotnet/runtime#119409
1140788	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#50663
1140696	dotnet/runtime	Send to Helix	Log
1140785	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#50164
1140736	dotnet/runtime	Send to Helix	Log
2787275	dotnet-dotnet	🛡 Guardian: PoliCheck	Log
1140717	dotnet/runtime	Send to Helix (Windows)	Log	dotnet/runtime#119376
1140759	dotnet/performance	Send job to Helix (Windows)	Log	dotnet/performance#4948
1140709	dotnet/runtime	Send to Helix	Log	dotnet/runtime#119408
2787257	dotnet-dotnet	Build (azurelinuxX64CrossContainer)	Log
2782746	dotnet-dotnet	Install MicroBuild plugin	Log
2787073	dotnet-runtime	Run performance job script	Log
2787014	dotnet-windowsdesktop	Build	Log
1140241	dotnet/runtime	Send to Helix	Log
2786981	dotnet-sdk	Run Tests in Helix	Log	#52915
2786926	dotnet-performance	Send job to Helix (Unix)	Log
1140242	dotnet/runtime	Send to Helix	Log
2786904	dotnet-runtime	Send job to Helix (Unix)	Log
2786892	dotnet-aspnetcore	Run build.cmd helix target	Log	#52993
1140137	dotnet/runtime	Send to Helix	Log	dotnet/runtime#119255
2786795	dotnet-runtime	Send tests to Helix (Unix)	Log
2786717	dotnet-dotnet	Build (centOSStreamContainer)	Log
1139914	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#119376
2786665	dotnet-sdk	Run AoT Tests in Helix	Log	#52915
2786653	dotnet-dotnet-buildtools-prereqs-docker	🟪 Build Images	Log
2786666	dotnet-sdk	Run Tests in Helix	Log	#52914
1139884	dotnet/runtime	Send to Helix (Windows)	Log	dotnet/runtime#119369
1139881	dotnet/runtime	Send to Helix (Windows)	Log	dotnet/runtime#118371
1139725	dotnet/runtime	Send to Helix	Log
2786638	dotnet-aspnetcore	Run build.cmd helix target	Log
1139869	dotnet/runtime	Send to Helix	Log	dotnet/runtime#119375
2786616	dotnet-workload-versions	🟣 Build solution	Log
1139806	dotnet/runtime	Send to Helix (Windows)	Log	dotnet/runtime#119355
1139856	dotnet/performance	Send job to Helix (Windows)	Log	dotnet/performance#4939
1139834	dotnet/runtime	Send to Helix	Log	dotnet/runtime#114969
1139809	dotnet/runtime	Send to Helix	Log	dotnet/runtime#119355
2786608	dotnet-efcore	Send job to helix	Log
1139810	dotnet/runtime	Send to Helix	Log	dotnet/runtime#119372
1139797	dotnet/runtime	Send to Helix	Log	dotnet/runtime#119050
1139813	dotnet/runtime	Send to Helix	Log	dotnet/runtime#119373
1139765	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115502
1139850	dotnet/runtime	Checkout dotnet/runtime@refs/pull/118257/merge to s	Log	dotnet/runtime#118257
1139851	dotnet/runtime	Checkout dotnet/runtime@refs/pull/118257/merge to s	Log	dotnet/runtime#118257
1139742	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#119343
1139758	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#119368
1139751	dotnet/runtime	Send to Helix	Log	dotnet/runtime#119288
1139821	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#63511
1139820	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#63511
1139724	dotnet/runtime	Send to Helix	Log
1139727	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#119324
2786590	dotnet-workload-versions	🟣 Build solution	Log
1139787	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#63261
1139784	dotnet/aspnetcore	Run build.cmd helix target	Log
2786519	dotnet-aspnetcore	Validate Hosted Pool Information (1ES PT)	Log
2786514	dotnet-aspnetcore	Run build.cmd	Log
2786499	dotnet-docker-tools	🛡 Guardian: Publish SDL Logs to Artifacts	Log
2786498	dotnet-docker-tools	🟦 Generate EOL Annotations - msdotnetcontainersstagingtest.azurecr.io	Log
2786466	dotnet-runtime	Send job to Helix (Unix)	Log
2786456	dotnet-dotnet	Build (centOSStreamContainer)	Log
2786328	dotnet-runtime	⏬ Pipeline Artifact Download	Log
1139378	dotnet/runtime	Send to Helix	Log
2786335	dotnet-dotnet-buildtools-prereqs-docker	🟪 Build Images	Log
2786313	dotnet-dotnet-buildtools-prereqs-docker	🛡 Generate SBoM Manifest	Log
1139387	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#50636
1139345	dotnet/runtime	Build Tests	Log
1139324	dotnet/runtime	Send to Helix	Log
2786260	dotnet-dotnet-buildtools-prereqs-docker	SDL Binary Analysis Checks (1ES PT)	Log
1139277	dotnet/runtime	Build Tests	Log
1139145	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118810
1139105	dotnet/runtime	Send tests to Helix (Windows)	Log
1139110	dotnet/runtime	Send to Helix	Log	dotnet/runtime#119177
2786143	dotnet-dotnet	🛡 Guardian: PSScriptAnalyzer	Log
2786077	dotnet-sdk	Run TestBuild Tests	Log	#52834
1138051	dotnet/aspnetcore	Run E2E tests	Log	dotnet/aspnetcore#63507
1139037	dotnet/runtime	Send to Helix	Log	dotnet/runtime#119343
1139048	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#50598
2786035	dotnet-dotnet	Build (centOSStreamContainer)	Log
2785982	dotnet-dotnet-monitor	Build	Log
2785949	dotnet-sdk	Run AoT Tests in Helix	Log	#52915
2785948	dotnet-sdk	Run Tests in Helix	Log	#52916
1138938	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#50518
1138891	dotnet/roslyn	Run Unit Tests	Log
2785885	dotnet-runtime	Run performance job script	Log
1138845	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118371
1138721	dotnet/runtime	Send to Helix	Log
1138833	dotnet/runtime	Send to Helix	Log	dotnet/runtime#119334
1138784	dotnet/runtime	Send to Helix	Log
1138791	dotnet/runtime	Send to Helix (Windows)	Log	dotnet/runtime#119335
1138805	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118732
1138780	dotnet/runtime	Send tests to Helix (Unix)	Log
1138781	dotnet/runtime	Send tests to Helix (Unix)	Log
2785864	dotnet-performance	Send job to Helix (Windows)	Log
1138844	dotnet/performance	Send job to Helix (Windows)	Log
Displaying 100 of 875 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
19	218	875

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)

[CarnaViire]

For the "known issues" folks: the timeout here is reported twice, making BA red despite the tracking issue. E.g.:

1️⃣ dotnet/runtime#117255

2️⃣ dotnet/runtime#117800

cc @AlitzelMendez