Support a "category" attribute for selecting benchmarks

[jskeet]

In Noda Time, I've found it useful to be able to categorize benchmarks. For example, I have a "BCL" category, and a "Text" category. Categories can be applied to classes or methods. From the command line, you can include or exclude specific categories.

Would love to see this in BenchmarkDotNet.

[AndreyAkinshin]

I like this feature, but I'm not sure how to implement it. I have two questions:

1. How we should name the attribute? I like Category, but it could conflict with other Category attributes (it's a common name, a lot of libraries use it for own purposes). Maybe BenchmarkCategory?
2. Should we use categories only in the command line or it should also affect exporers/summary? If we want to have it in the summary, how it should be displayed? Especially, I'm worried about methods which belong to multiple categories.

[jskeet]

I'm fine with BenchmarkCategory. I don't have enough understanding of the details of BenchmarkDotNet to comment, but I know I'd personally be interested in using it for filtering, primarily... but it would be nice to know which category any benchmark is in for filtering post-run as well. ("Show me how all the text handling benchmarks have gone over time.")

[AndreyAkinshin]

Ok, we could use it in Csv and Json exporters and ignore in the summary.

[roji]

+1 for this. One additional idea would be to have the BenchmarkSwitcher support categories as well. Once you start having lots of benchmarks the switcher quickly becomes very hard to work with, the switcher could first present the user with the list of categories, and after category selection, the benchmarks in that category.

[NickCraver]

I have a related use case here that could be satisfied by the same feature, I'd like to run all benchmarks in an assembly and see the results in a single output.

Concrete case: in Dapper I have a class w/ benchmarks for each ORM, this is because their setup is different and one monolithic class with tons of setup for no good reason is very inefficient, slow, and possibly full of side-effects. A small class is easy to add, maintain, etc. and keeps [Setup] simple/efficient as well.

If we could run all of them (or a * category, or give them all the same category, w.r.t. this issue), life would be much easier. For example we could compare all of the "select many", "select single", "select dynamic", etc. methods across benchmarks very cleanly while also having easy-to-maintain code.

Like @roji mentions above, BenchmarkSwitcher support would be huge, even if it were .RunCategories() or some such to not break existing behavior.

[AndreyAkinshin]

Hey @NickCraver. First of all, there is a workaround that you can use right now, here is an example:

[DryJob]
public class A
{
    [Benchmark]
    public void A1() => Thread.Sleep(10);
    
    [Benchmark]
    public void A2() => Thread.Sleep(10);
}

[DryJob]
public class B
{
    [Benchmark]
    public void B1() => Thread.Sleep(10);
    
    [Benchmark]
    public void B2() => Thread.Sleep(10);
}

internal class Program
{
    public static void Main(string[] args)
    {
        var benchmarks = new List<Benchmark>();
        benchmarks.AddRange(BenchmarkConverter.TypeToBenchmarks(typeof(A)));
        benchmarks.AddRange(BenchmarkConverter.TypeToBenchmarks(typeof(B)));
        BenchmarkRunner.Run(benchmarks.Where(b => b.Target.Method.Name.Contains("1")).ToArray(), null);
    }
}

Such approach allows combining specific set of benchmark methods from different types in one summary table.
However, we need an API which provides a nice way to do it. There is no problem to add the BenchmarkCategory attribute. Last time I was stuck with the API design.

My current suggestions:

• We can add a boolean flag to all BenchmarkSwitcher methods which allow combining all the benchmarks in a single summary table
• We can add the following property to IConfig:

IEnumerable<IFilter> GetFilters();

By default, it will return an empty enumerable which will mean that we want to run all the benchmarks. The interface will contain single method:

public interface IFilter
{
    bool Predicate(Benchmark benchmark);
}

And it would be easy to implement CategoryFilter:

public class CategoryFilter: IFilter
{
    public CategoryFilter(params string[] categories)
}

Since it's a part of IConfig, such filter can be added manually or via command line.
Open question:

• Should we run benchmarks which have all the target categories or at least one? Maybe it makes sense to introduce something like enum UnionPolicy { Any, All }.
• Should we run benchmarks which match all the target filters or at least one? Or maybe we should use the same enum here?

@NickCraver, will this be enough? Do you need any additional features here?

@adamsitnik, @mattwarren, @jskeet, what do you think about the suggested API?

[adamsitnik]

I had similar issues when I wanted to benchmark Array vs Span vs List.

I think that if we add [Category] attribute or property to the [Benchmark] attribute we should get what we all want. Benchmark switcher could then offer to choose whole category or given type. Sample:

class ArrayBenchmarks
{
    [Benchmark(Baseline = true, Category = "Indexers"]
    public int IndexerArray() => array[0];
}

class SpanBenchmarks
{
    [Benchmark(Category = "Indexers"]
    public int IndexerSpan() => span[0];
}

And then sample Benchmark Switcher output could be:

Available Benchmarks:
  #0  Category: Indexers
  #1  ArrayBenchmarks
  #2  SpanBenchmarks

Another thing would be new overload BenchmarkRunner.Run(string categoryName) plus support of the --category parameter in console parser.

But I am not sure how to differentiate whole group of benchmarks (for example CollectionsBenchmark category) vs specialized category (for example "IndexerBenchmarks")

[AndreyAkinshin]

@adamsitnik, see original request by Jon Skeet:

Categories can be applied to classes or methods.

The [Benchmark(Category = "Indexers"] approach looks nice, but it can't be applied to classes.

[NickCraver]

@AndreyAkinshin First, thanks for the workaround - that unblocked me in Dapper and it's much appreciated! I like the API ideas, I guess the big question is whether a benchmark is in 1 or many categories. I'd certainly say it's the latter, which would affect attribute names...or maybe not.

What if it was:

[BenchmarkCategory("dynamic","list")]

via a params on the attribute? It wouldn't preclude a property on [Benchmark], but since that both can't be applied to classes and would imply only a single category is available it's confusing in several ways. I'd agree with only having it as a separate attribute.

I also have use cases for the union descriptor in Dapper as well (e.g. run all dynamic classes, or list classes, or the combination, etc.) so +1 to that.

Related: possibly built-in CategoryColumn to go along with this for easy usage?

[AndreyAkinshin]

@NickCraver, I implemented all the features (you can try 0.10.6.186 from our nigtly feed https://ci.appveyor.com/nuget/benchmarkdotnet). Let me know if you need something else. If everything is fine, I will release new version soon. You can find some docs below:

Filters

Sometimes you don't want to run all of your benchmarks.
In this case, you can filter some of them with the help of filters.

Predefined filters:

• SimpleFilter
• NameFilter
• DisjunctionFilter
• CategoryFilter
• AnyCategoriesFilter
• AllCategoriesFilter

Usage examples:

[Config(typeof(Config))]
public class IntroFilters
{
    private class Config : ManualConfig
    {
        // We will benchmark ONLY method with names with names
        // (which contains "A" OR "1") AND (have length < 3)
        public Config()
        {
            // benchmark with names which contains "A" OR "1"
            Add(new DisjunctionFilter(
                new NameFilter(name => name.Contains("A")),
                new NameFilter(name => name.Contains("1"))
            ));

            // benchmark with names with length < 3
            Add(new NameFilter(name => name.Length < 3));
        }
    }

    [Benchmark] public void A1() => Thread.Sleep(10); // Will be benchmarked
    [Benchmark] public void A2() => Thread.Sleep(10); // Will be benchmarked
    [Benchmark] public void A3() => Thread.Sleep(10); // Will be benchmarked
    [Benchmark] public void B1() => Thread.Sleep(10); // Will be benchmarked
    [Benchmark] public void B2() => Thread.Sleep(10);
    [Benchmark] public void B3() => Thread.Sleep(10);
    [Benchmark] public void C1() => Thread.Sleep(10); // Will be benchmarked
    [Benchmark] public void C2() => Thread.Sleep(10);
    [Benchmark] public void C3() => Thread.Sleep(10);
    [Benchmark] public void Aaa() => Thread.Sleep(10);
}

[DryJob]
[CategoriesColumn]
[BenchmarkCategory("Aswesome")]
[AnyCategoriesFilter("A", "1")]
public class IntroCategories
{
    [Benchmark]
    [BenchmarkCategory("A", "1")]
    public void A1() => Thread.Sleep(10); // Will be benchmarked
    
    [Benchmark]
    [BenchmarkCategory("A", "2")]
    public void A2() => Thread.Sleep(10); // Will be benchmarked

    [Benchmark]
    [BenchmarkCategory("B", "1")]
    public void B1() => Thread.Sleep(10); // Will be benchmarked
    
    [Benchmark]
    [BenchmarkCategory("B", "2")]
    public void B2() => Thread.Sleep(10);
}

Command line examples:

--category=A
--allCategories=A,B
--anyCategories=A,B

If you are using BenchmarkSwitcher and want to run all the benchmarks with a category from all types and join them into one summary table, use the --join option (or BenchmarkSwitcher.RunAllJoined):

* --join --category=MyAwesomeCategory