Skip to content

feat: Add validator for benchmarks that contains null runtime #2771

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 15, 2025

Conversation

filzrev
Copy link
Contributor

@filzrev filzrev commented Jun 12, 2025

This PR add validator to handle issue that is reported #2609.

What's changed in this PR

  1. Add RuntimeValidator.cs for benchmarks that have multiple runtimes and contains null.
  2. Add RuntimeValidator to DefaultConfig's validator
  3. Add unit tests.

Background
When benchmark don't contains Runtime characteristic. (e.g. When specify WithToolChain() only)
BenchmarkCase::GetRuntime returns current runtime instead of actual runtime that used by benchmark.

Normally, Runtime column is hidden on table because all benchmarks have same runtime value.
So it's not a problem. But when mixed WithRuntime settings. it display wrong runtime.

So it need to add validator to check benchmarks that contains multiple runtimes and contains null.

RuntimeValidator's error message

// * There are benchmarks that job don't have a Runtime characteristic. It's recommended explicitly specify runtime by WithRuntime

@filzrev filzrev force-pushed the feat-add-runtimevalidator branch 2 times, most recently from 41b066f to 4b61ffb Compare June 12, 2025 11:18
@timcassell
Copy link
Collaborator

I think the UX needs to be improved around this, but this is ok as a stop-gap for now.

Copy link
Collaborator

@timcassell timcassell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment

@filzrev
Copy link
Contributor Author

filzrev commented Jun 15, 2025

I've updated runtime validation logics (457330d)
Additionally, I've added JobId (or Job text representation) to validation error message.

Example validation error messages

// Validating benchmarks:
//    * Job(Toolchain=.NET 10.0) don't have a Runtime characteristic. It's recommended to specify runtime by using WithRuntime explicitly.

//    * Job(Dry) don't have a Runtime characteristic. It's recommended to specify runtime by using WithRuntime explicitly.

//    * Job(abc) don't have a Runtime characteristic. It's recommended to specify runtime by using WithRuntime explicitly.

@filzrev filzrev force-pushed the feat-add-runtimevalidator branch from ef841f8 to ea68e83 Compare June 15, 2025 04:50
@filzrev
Copy link
Contributor Author

filzrev commented Jun 15, 2025

It's not related to this PR thought. AllocationQuantumIsNotAnIssueForNetCore21Plus test seems flaky on CI.
Currently it's confirmed on mac-os (ARM64).

2025-06-15T04:21:35.2938130Z [xUnit.net 00:09:12.36]     AllocationQuantumIsNotAnIssueForNetCore21Plus(toolchain: .NET 8.0) [FAIL]
2025-06-15T04:21:35.2938900Z [xUnit.net 00:09:12.36]       Assert.Equal() Failure: Values differ
2025-06-15T04:21:35.2939300Z [xUnit.net 00:09:12.36]       Expected: 88
2025-06-15T04:21:35.2939630Z [xUnit.net 00:09:12.36]       Actual:   760

@timcassell timcassell merged commit ed5316b into dotnet:master Jun 15, 2025
14 of 16 checks passed
@timcassell timcassell added this to the v0.15.2 milestone Jun 15, 2025
@timcassell
Copy link
Collaborator

Just thought about in-process toolchains after I merged. The current runtime behavior is correct on those toolchains, so they should not be included as part of the validation. @filzrev

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants