[Feature Request] Add custom option to sort benchmark jobs by numeric orders #2768
Description
Currently BenchmarkDotNet's DefaultOrderer don't have parameter to customize Job display orders.
And JobComparer.Instance is always used.
Currently JobComparer.Instance sort jobs based on characteristics.
And compare each characteristics with Ordinal string comparer.
I want to add option to specify JobOrderPolicy that change behavior of JobComparer with custom NumericStringComparer.
(that similar to CompareOptions.NumericOrdering that added to .NET 10)
Background
Currently there is following job display order issues when using DefaultOrderer.
Example. 1: .NET Runtime orders
.NET 10.0 benchmark is shown on top item.
| Method | Runtime | Mean | Error | Ratio |
|---|---|---|---|---|
| Benchmark1 | .NET 10.0 | 16.337 ms | NA | 3.32 |
| Benchmark1 | .NET 8.0 | 4.922 ms | NA | 1.00 |
| Benchmark1 | .NET 9.0 | 16.233 ms | NA | 3.30 |
Example.2: JobId column orders (and NuGetReferences column orders)
Similarly, JobId/NuGetReferences are sorted by ordinal order.
| Type | Method | Job | N | Mean |
|---|---|---|---|---|
| ContainsBenchmark | Contains | v1.4.1 | 1000000 | 29.87 us |
| ContainsBenchmark | Contains | v1.4.10 | 1000000 | 29.90 us |
| ContainsBenchmark | Contains | v1.4.2 | 1000000 | 30.31 us |