Skip to content

Builds timing out on CI #38964

Closed
Closed
@safern

Description

@safern

We're seeing multiple builds canceled due to "timeouts" when they're not really timeouts. This seems like a bug in AzDo that it is considering the "queue" time and time spent waiting for an agent as part of the timeout time.

##[error]The job running on agent NetCorePublic-Pool 172 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134
Pool: NetCorePublic-Pool
Agent: NetCorePublic-Pool 172
Started: Today at 1:54 PM
Duration: 15m 2s

As seen, the duration was really 15m 2s which is not over 60 mins.

Runtime build

Build Kind Timeline Record
721340 PR #38938 CoreCLR Product Build Windows_NT x64 release
721340 PR #38938 CoreCLR Product Build Windows_NT arm64 checked
721340 PR #38938 CoreCLR Product Build Windows_NT x86 release
721425 PR #38952 CoreCLR Product Build Windows_NT x64 release
721425 PR #38952 CoreCLR Product Build Windows_NT arm release
721425 PR #38952 CoreCLR Product Build Windows_NT arm64 release
721425 PR #38952 CoreCLR Product Build Windows_NT x86 release
721535 PR #38765 CoreCLR Product Build OSX x64 checked
721294 PR #38948 Mono Product Build Linux x64 debug
721294 PR #38948 CoreCLR Product Build Linux_musl x64 release
721294 PR #38948 CoreCLR Product Build Linux_musl x64 checked
721294 PR #38948 CoreCLR Product Build Linux x64 release
721294 PR #38948 CoreCLR Product Build Linux arm release
721294 PR #38948 Formatting Linux x64
721294 PR #38948 CoreCLR Product Build FreeBSD x64 release
721294 PR #38948 CoreCLR Product Build Linux arm64 release
721294 PR #38948 CoreCLR Product Build Linux_musl arm64 release
721294 PR #38948 Build Linux arm Debug AllSubsets_Mono
721294 PR #38948 Build Linux x64 Debug AllSubsets_Mono_LLVMJIT
721294 PR #38948 Mono Product Build Linux arm64 debug
721294 PR #38948 Mono Product Build Linux x64 release
721294 PR #38948 Build Linux_musl x64 Release AllSubsets_Mono
721294 PR #38948 Mono Product Build Linux arm64 release
721294 PR #38948 CoreCLR Product Build Linux arm64 checked
721294 PR #38948 CoreCLR Product Build Linux_musl arm64 checked
721294 PR #38948 CoreCLR Product Build Linux x64 checked
721294 PR #38948 CoreCLR Product Build Linux arm checked
721294 PR #38948 Build Linux x64 Release AllSubsets_Mono_LLVMAOT
721099 PR #38707 Mono Product Build Linux x64 debug
721099 PR #38707 CoreCLR Product Build Linux_musl x64 release
721099 PR #38707 CoreCLR Product Build Linux x64 release
721099 PR #38707 CoreCLR Product Build Linux arm release
721099 PR #38707 CoreCLR Product Build FreeBSD x64 release
721099 PR #38707 CoreCLR Product Build Linux arm64 release
721099 PR #38707 CoreCLR Product Build Linux_musl arm64 release
721099 PR #38707 Build Linux arm Debug AllSubsets_Mono
721099 PR #38707 Build Linux x64 Release AllSubsets_Mono_LLVMAOT
721099 PR #38707 Build Linux x64 Debug AllSubsets_Mono_LLVMJIT
721099 PR #38707 Mono Product Build Linux arm64 debug
721099 PR #38707 Mono Product Build Linux x64 release
721099 PR #38707 Build Linux_musl x64 Release AllSubsets_Mono
721099 PR #38707 Mono Product Build Linux arm64 release
721062 PR #38935 CoreCLR Product Build Linux_musl x64 release
721062 PR #38935 CoreCLR Product Build Linux x64 release
721062 PR #38935 Build Linux x64 Release AllSubsets_Mono_LLVMAOT
721160 PR #37971 CoreCLR Product Build Windows_NT x86 checked
721161 PR #38884 CoreCLR Product Build Windows_NT x64 release
721161 PR #38884 CoreCLR Product Build Windows_NT arm release
721161 PR #38884 CoreCLR Product Build Windows_NT x86 release
721161 PR #38884 CoreCLR Product Build Windows_NT arm64 release
721009 PR #38815 Build Linux arm Debug AllSubsets_Mono
721042 PR #38933 CoreCLR Product Build Linux arm release
720484 PR #38893 CoreCLR Product Build OSX x64 release
719881 PR #38728 CoreCLR Product Build Windows_NT x86 checked

Evaluated 100 builds
Impacted 12 bulids
Impacted 54 jobs

This is impacting around 12% of builds of runtime and 12% of runtime-live-build.

Runtime-live-build

Build Kind Timeline Record
721591 PR #38547 Build OSX x64 release Runtime_Debug
721451 PR #38894 Build OSX x64 release Runtime_Debug
721456 PR #38616 Build Windows_NT x64 debug Mono_Libraries
721456 PR #38616 Build Windows_NT x86 release Runtime_Debug
721295 PR #38948 Build Linux x64 debug RuntimeFlavor_Mono
721295 PR #38948 Build Linux x64 debug Runtime_Release
721295 PR #38948 Build Linux x64 release SourceBuild
721264 PR #38547 Build Linux x64 debug Runtime_Release
721264 PR #38547 Build Linux x64 release SourceBuild
721270 PR #38946 Build Linux x64 debug RuntimeFlavor_Mono
721270 PR #38946 Build Linux x64 debug Runtime_Release
721241 PR #32520 Build Windows_NT x64 debug Mono_Libraries
721063 PR #38935 Build Windows_NT x64 debug Mono_Libraries
721063 PR #38935 Build Linux x64 debug RuntimeFlavor_Mono
721063 PR #38935 Build Linux x64 debug Runtime_Release
721063 PR #38935 Build Linux x64 release SourceBuild
721100 PR #38707 Build Linux x64 debug RuntimeFlavor_Mono
721100 PR #38707 Build Linux x64 debug Runtime_Release
721043 PR #38933 Build Linux x64 release SourceBuild
721039 PR #37159 Build Linux x64 debug Runtime_Release
720680 PR #38917 Build OSX x64 release Runtime_Debug

Evaluated 100 builds
Impacted 12 bulids
Impacted 21 jobs

@MattGal is looking into this and opening a ticket on AzDo

cc: @dotnet/runtime-infrastructure

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions