Skip to content

fix: Flaky CI tests by increase build timeout - #2854

Merged
timcassell merged 4 commits into
dotnet:masterfrom
filzrev:fix-ci-build-timeout
Nov 6, 2025
Merged

fix: Flaky CI tests by increase build timeout#2854
timcassell merged 4 commits into
dotnet:masterfrom
filzrev:fix-ci-build-timeout

Conversation

@filzrev

@filzrev filzrev commented Nov 5, 2025

Copy link
Copy Markdown
Contributor

This PR intended to fix #2853

What's changed in this PR
Increase BuildTimeout value to value to 240 seconds (Default: 120 seconds)
of following tests.

  • WasmIsSupported
  • UserCanSpecifyCustomBuildConfiguration

Note:
Following FactEnvSpecific is left as is.

[FactEnvSpecific("Flaky, see https://github.com/dotnet/BenchmarkDotNet/issues/2376", EnvRequirement.NonFullFramework)]

Timeout error on NETFX might be resolved by this PR changes.
Though, it'll be handled by #2376

@filzrev

filzrev commented Nov 5, 2025

Copy link
Copy Markdown
Contributor Author

macos (x64) CI failed on following error.

MemoryDiagnoserSupportsModernMono [FAIL]
2025-11-05T04:42:58.2674460Z [xUnit.net 00:35:03.19]       Assert.Equal() Failure: Values differ
2025-11-05T04:42:58.2682030Z [xUnit.net 00:35:03.19]       Expected: 40
2025-11-05T04:42:58.2705840Z [xUnit.net 00:35:03.19]       Actual:   null

Above error seems to be happen randomly. and cause flaky test.

And ubuntu-latest following MSB3277 warning happens.

2025-11-05T04:28:21.5570278Z /home/runner/work/BenchmarkDotNet/BenchmarkDotNet/.dotnet/sdk/8.0.410/Microsoft.Common.CurrentVersion.targets(2412,5): warning MSB3277: Found conflicts between different versions of "System.Runtime" that could not be resolved. [/home/runner/work/BenchmarkDotNet/BenchmarkDotNet/src/BenchmarkDotNet.Diagnostics.dotMemory/BenchmarkDotNet.Diagnostics.dotMemory.csproj::TargetFramework=net6.0]
2025-11-05T04:28:21.5579320Z /home/runner/work/BenchmarkDotNet/BenchmarkDotNet/.dotnet/sdk/8.0.410/Microsoft.Common.CurrentVersion.targets(2412,5): warning MSB3277: There was a conflict between "System.Runtime, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" and "System.Runtime, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a". [/home/runner/work/BenchmarkDotNet/BenchmarkDotNet/src/BenchmarkDotNet.Diagnostics.dotMemory/BenchmarkDotNet.Diagnostics.dotMemory.csproj::TargetFramework=net6.0]
2025-11-05T04:28:21.5587153Z /home/runner/work/BenchmarkDotNet/BenchmarkDotNet/.dotnet/sdk/8.0.410/Microsoft.Common.CurrentVersion.targets(2412,5): warning MSB3277:     "System.Runtime, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" was chosen because it was primary and "System.Runtime, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" was not. [/home/runner/work/BenchmarkDotNet/BenchmarkDotNet/src/BenchmarkDotNet.Diagnostics.dotMemory/BenchmarkDotNet.Diagnostics.dotMemory.csproj::TargetFramework=net6.0]
2025-11-05T04:28:21.5594052Z /home/runner/work/BenchmarkDotNet/BenchmarkDotNet/.dotnet/sdk/8.0.410/Microsoft.Common.CurrentVersion.targets(2412,5): warning MSB3277:     References which depend on "System.Runtime, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" [/home/runner/.nuget/packages/microsoft.netcore.app.ref/6.0.36/ref/net6.0/System.Runtime.dll]. [/home/runner/work/BenchmarkDotNet/BenchmarkDotNet/src/BenchmarkDotNet.Diagnostics.dotMemory/BenchmarkDotNet.Diagnostics.dotMemory.csproj::TargetFramework=net6.0]

I don't know why this warning occurred randomly.
Though it's caused by transitive package reference conflicts between .NET6 / .NET8
So add .NET8 TargetFrameworks to dotTrace/dotMemory project

@timcassell

Copy link
Copy Markdown
Collaborator

Still seeing The configured timeout 00:02:00 was reached! for the modern Mono tests.

@timcassell

Copy link
Copy Markdown
Collaborator

Same with MemoryDiagnoserTests.MemoryDiagnoserSupportsModernMono.

@filzrev

filzrev commented Nov 6, 2025

Copy link
Copy Markdown
Contributor Author

As far as I've confirmed MemoryDiagnoserSupportsModernMono failed on test assertion.
Built timeout message seems not happens on this test.

@timcassell

Copy link
Copy Markdown
Collaborator

As far as I've confirmed MemoryDiagnoserSupportsModernMono failed on test assertion. Built timeout message seems not happens on this test.

It happened on the previous test run.

@filzrev

filzrev commented Nov 6, 2025

Copy link
Copy Markdown
Contributor Author

Is it able to provide log link of test run that failed with timeout?

@timcassell

Copy link
Copy Markdown
Collaborator

Is it able to provide log link of test run that failed with timeout?

https://github.com/dotnet/BenchmarkDotNet/actions/runs/19093373782/job/54598246558

The test says it failed like this

MemoryDiagnoserSupportsModernMono [FAIL]
Assert.Equal() Failure: Values differ
Expected: 40
Actual:   null

But if you scroll further down, you'll see it's because the build timed out.

The configured timeout 00:02:00 was reached!

@filzrev

filzrev commented Nov 6, 2025

Copy link
Copy Markdown
Contributor Author

Thanks for confirmations.
I seem to have missed it because it's not happens build timing and outputted as stdout log.

I've added new commit 515be8d.
It increase timeout value for all MemoryDiagnoserTests to simplify code changes.

@timcassell timcassell left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

@timcassell
timcassell merged commit 24a357b into dotnet:master Nov 6, 2025
9 checks passed
@timcassell timcassell added this to the v0.15.7 milestone Nov 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Build timeout error occurred on CI build (macos (x64))

2 participants