Skip to content

test(assertions): drop flaky wall-clock upper bound on WaitsFor timeout test#5886

Merged
thomhurst merged 1 commit into
mainfrom
fix/ci-waitsfor-budget
May 11, 2026
Merged

test(assertions): drop flaky wall-clock upper bound on WaitsFor timeout test#5886
thomhurst merged 1 commit into
mainfrom
fix/ci-waitsfor-budget

Conversation

@thomhurst
Copy link
Copy Markdown
Owner

Summary

WaitsFor_Fails_When_Timeout_Expires (TUnit.Assertions.Tests/WaitsForAssertionTests.cs:60) asserted that the entire test method completes within 1 s while configuring a 100 ms timeout. On a slow Windows CI worker the total wall-clock — exception construction, assertion-machinery cost, etc. — pushed past 2.1 s, failing the test even though the 100 ms timeout itself was respected (run 24793788489).

This swaps the brittle upper-bound check for the lower-bound invariant the test should actually be guarding: the timeout fires no earlier than its configured value.

Test plan

  • dotnet run --filter WaitsFor_Fails_When_Timeout_Expires* on Windows passes
  • .NET workflow runs green across all three OS legs

Copy link
Copy Markdown
Contributor

@claude claude Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: drop flaky wall-clock upper bound on WaitsFor timeout test

Verdict: Approve. The change is correct, well-motivated, and improves test reliability without sacrificing meaningful coverage.

What the change does

The old assertion (IsLessThan(1s)) was a wall-clock upper bound that included not just the 100 ms timeout, but also exception construction and assertion-machinery overhead. On slow CI workers that overhead exceeded 1 s, causing false failures. The new assertion (IsGreaterThanOrEqualTo(100ms)) checks the only invariant this test actually owns: the timeout did not fire prematurely.

Why this is the right call

The test still verifies its true contract through two complementary assertions:

  1. IsGreaterThanOrEqualTo(100ms) — the timeout was not short-circuited or set to zero.
  2. exception.Message.Contains("assertion did not pass within 100ms") — the configured timeout value was reflected in the failure output correctly.

Those two together are stronger evidence that the mechanism works than a loose wall-clock upper bound ever was.

One thing worth considering

The removed upper bound did serve as a canary for "WaitsFor is hanging catastrophically." That protection is now gone. If it matters, a very wide upper bound — say IsLessThan(TimeSpan.FromSeconds(30)) — would catch a runaway retry loop without being sensitive to CI load. That said, the test-level timeout on the host framework would catch infinite hangs anyway, so this is low priority and strictly out of scope here.

Minor note

WaitsFor_Passes_Immediately_When_Assertion_Succeeds at line 18 still has IsLessThan(TimeSpan.FromMilliseconds(100)), which is subject to the same class of flakiness on a loaded runner. Worth a follow-up issue if that one starts flaking too.

Overall: clean fix, correct diagnosis, no regressions in coverage that matter. The 3-line comment explaining the reasoning is justified given the non-obvious CI constraint it documents.

@codacy-production
Copy link
Copy Markdown

codacy-production Bot commented May 11, 2026

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

…ired lower-bound

WaitsFor_Fails_When_Timeout_Expires asserted the entire test method
completed in <1s while configuring a 100ms timeout. On a slow Windows
CI worker, exception construction + assertion-machinery cost pushed the
total to 2.118s, failing the test even though the 100ms timeout itself
was respected (run 24793788489).

Replace the upper-bound check with the lower-bound invariant the test
should actually be guarding: the timeout fired no earlier than the
configured 100ms.
@thomhurst thomhurst force-pushed the fix/ci-waitsfor-budget branch from 721729a to 91dac14 Compare May 11, 2026 22:31
@thomhurst thomhurst enabled auto-merge (squash) May 11, 2026 22:32
Copy link
Copy Markdown
Contributor

@claude claude Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review (after merge-from-main commit)

The only PR-authored change remains the single assertion swap in WaitsForAssertionTests.cs. The new commit (91dac14) is a merge from main that brings in package version bumps (Mockolate 3.2.0, NUnit 4.6.0, TUnit.Assertions.FSharp 1.44.0, etc.) — no functional changes to the code under review.

Previous findings addressed?

The previous review raised two points:

  1. Dropped upper-bound as a runaway-hang canary — still applies; a very wide sanity bound (e.g., IsLessThan(30s)) would catch catastrophic hangs cheaply. Low priority given host-level test timeouts exist, but worth a follow-up issue.

  2. WaitsFor_Passes_Immediately_When_Assertion_Succeeds has a symmetric flakiness risk (line 18: IsLessThan(100ms)) — still unremediated, which is fine for this PR's scope. Worth a tracked issue.

Neither point is a blocker; both are low-priority follow-ups.

Verdict

The fix is correct, minimal, and well-explained. The comment added alongside the assertion clearly documents the CI constraint so future readers don't reintroduce the upper bound. Approving.

@thomhurst thomhurst merged commit 627179d into main May 11, 2026
13 of 14 checks passed
@thomhurst thomhurst deleted the fix/ci-waitsfor-budget branch May 11, 2026 23:02
github-actions Bot pushed a commit to IntelliTect/CodingGuidelines that referenced this pull request May 19, 2026
Updated [TUnit.Core](https://github.com/thomhurst/TUnit) from 1.44.0 to
1.45.8.

<details>
<summary>Release notes</summary>

_Sourced from [TUnit.Core's
releases](https://github.com/thomhurst/TUnit/releases)._

## 1.45.8

<!-- Release notes generated using configuration in .github/release.yml
at v1.45.8 -->

## What's Changed
### Other Changes
* fix(aspire): route CreateHttpClient through IHttpClientFactory by
@​thomhurst in thomhurst/TUnit#5957
### Dependencies
* chore(deps): update tunit to 1.45.0 by @​thomhurst in
thomhurst/TUnit#5949
* chore(deps): update dependency dompurify to v3.4.5 by @​thomhurst in
thomhurst/TUnit#5951
* chore(deps): update dependency
microsoft.testing.extensions.codecoverage to 18.7.0 by @​thomhurst in
thomhurst/TUnit#5953
* chore(deps): update dependency coverlet.collector to 10.0.1 by
@​thomhurst in thomhurst/TUnit#5952
* chore(deps): update dependency polyfill to 10.6.0 by @​thomhurst in
thomhurst/TUnit#5955
* chore(deps): update dependency polyfill to 10.6.0 by @​thomhurst in
thomhurst/TUnit#5954


**Full Changelog**:
thomhurst/TUnit@v1.45.0...v1.45.8

## 1.45.0

<!-- Release notes generated using configuration in .github/release.yml
at v1.45.0 -->

## What's Changed
### Other Changes
* fix(generator): place CallerArgumentExpression before params in
[GenerateAssertion] emit by @​JohnVerheij in
thomhurst/TUnit#5940
* fix(sourcegen): drop covariant TActual when [GenerateAssertion] method
has its own type parameters by @​JohnVerheij in
thomhurst/TUnit#5935
* feat(assertions): add CancellationToken overload to WaitsFor and
Eventually by @​JohnVerheij in
thomhurst/TUnit#5938
* fix(html-report): extract categories using MTP Key=name convention by
@​thomhurst in thomhurst/TUnit#5946
* feat(html-report): rewrite as split-pane design template by
@​thomhurst in thomhurst/TUnit#5947
### Dependencies
* chore(deps): update microsoft.testing to 2.2.3 by @​thomhurst in
thomhurst/TUnit#5927
* chore(deps): update mstest to 4.2.3 by @​thomhurst in
thomhurst/TUnit#5928
* chore(deps): update tunit to 1.44.39 by @​thomhurst in
thomhurst/TUnit#5929
* chore(deps): update aspire to 13.3.3 by @​thomhurst in
thomhurst/TUnit#5933
* chore(deps): update dependency dompurify to v3.4.4 by @​thomhurst in
thomhurst/TUnit#5944
* chore(deps): update dependency qs to v6.15.2 by @​thomhurst in
thomhurst/TUnit#5941


**Full Changelog**:
thomhurst/TUnit@v1.44.39...v1.45.0

## 1.44.39

<!-- Release notes generated using configuration in .github/release.yml
at v1.44.39 -->

## What's Changed
### Other Changes
* fix(tests): retry trx read to dodge MTP post-exit flush race on
Windows by @​thomhurst in thomhurst/TUnit#5888
* fix(pipeline): timeout + retry InstallPlaywrightModule so a hung
download fails fast by @​thomhurst in
thomhurst/TUnit#5889
* fix(otel): require two consecutive idle windows in DrainAsync to catch
in-transit POSTs by @​thomhurst in
thomhurst/TUnit#5890
* test(assertions): drop flaky wall-clock upper bound on WaitsFor
timeout test by @​thomhurst in
thomhurst/TUnit#5886
* fix(sourcegen): drop spurious ')' in MethodAssertionGenerator
Task<bool> emit by @​JohnVerheij in
thomhurst/TUnit#5920
* fix(sourcegen): merge generic parameter lists in [AssertionExtension]
emit by @​JohnVerheij in thomhurst/TUnit#5921
* fix(aspnetcore): scope correlation processor per-factory to stop
cross-factory tag leak by @​thomhurst in
thomhurst/TUnit#5891
* Changed FSharp.Core version to 10.1.300 by @​licon4812 in
thomhurst/TUnit#5909
* feat(mocks): add Mock.HttpClientFactory() helper by @​thomhurst in
thomhurst/TUnit#5894
* Harden WaitsFor timeout test by @​thomhurst in
thomhurst/TUnit#5926
* fix(sourcegen): emit `default` literal for value-type assertion
parameters by @​JohnVerheij in
thomhurst/TUnit#5919
### Dependencies
* chore(deps): update dependency nunit to 4.6.0 by @​thomhurst in
thomhurst/TUnit#5826
* chore(deps): update tunit to 1.44.0 by @​thomhurst in
thomhurst/TUnit#5882
* chore(deps): update dependency mockolate to 3.2.0 by @​thomhurst in
thomhurst/TUnit#5892
* chore(deps): update dependency yaml to v2.9.0 by @​thomhurst in
thomhurst/TUnit#5887
* chore(deps): update dependency nuget.protocol to 7.6.0 by @​thomhurst
in thomhurst/TUnit#5897
* chore(deps): update dependency microsoft.entityframeworkcore to 10.0.8
by @​thomhurst in thomhurst/TUnit#5898
* chore(deps): update dependency microsoft.templateengine.authoring.cli
to v10.0.300 by @​thomhurst in
thomhurst/TUnit#5899
* chore(deps): update microsoft.extensions by @​thomhurst in
thomhurst/TUnit#5905
* chore(deps): update microsoft.aspnetcore to 10.0.8 by @​thomhurst in
thomhurst/TUnit#5904
* chore(deps): update dependency
microsoft.templateengine.authoring.templateverifier to 10.0.300 by
@​thomhurst in thomhurst/TUnit#5902
* chore(deps): update aspire to 13.3.1 by @​thomhurst in
thomhurst/TUnit#5900
* chore(deps): update dependency system.commandline to 2.0.8 by
@​thomhurst in thomhurst/TUnit#5903
* chore(deps): update dependency azure.storage.blobs to 12.28.0 by
@​thomhurst in thomhurst/TUnit#5910
* chore(deps): update dependency dotnet-sdk to v10.0.300 by @​thomhurst
in thomhurst/TUnit#5901
* chore(deps): update dependency stackexchange.redis to 2.13.1 by
@​thomhurst in thomhurst/TUnit#5906
* chore(deps): update aspire to 13.3.2 by @​thomhurst in
thomhurst/TUnit#5924
* chore(deps): bump mermaid from 11.12.2 to 11.15.0 in /docs by
@​dependabot[bot] in thomhurst/TUnit#5893
* chore(deps): update dependency streamjsonrpc to 2.24.92 by @​thomhurst
in thomhurst/TUnit#5915
* chore(deps): update dependency dompurify to v3.4.3 by @​thomhurst in
thomhurst/TUnit#5913
* chore(deps): update microsoft.build to 18.6.3 by @​thomhurst in
thomhurst/TUnit#5914


**Full Changelog**:
thomhurst/TUnit@v1.44.0...v1.44.39

Commits viewable in [compare
view](thomhurst/TUnit@v1.44.0...v1.45.8).
</details>

[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=TUnit.Core&package-manager=nuget&previous-version=1.44.0&new-version=1.45.8)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This was referenced May 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant