PowerUtils.BenchmarkDotnet.Reporter

Tool to analyze and compare .NET benchmark reports

PowerUtils.BenchmarkDotnet.Reporter is a command-line tool used to analyze and compare .NET benchmark reports generated by BenchmarkDotNet. This tool is designed to run locally or in CI/CD pipelines, providing a simple way to visualize and compare benchmark results.

• Prepare environment
  • Install Tool
  • Update Tool
  • List installed tools
  • Uninstall Tool
• Export reports using BenchmarkDotNet
• How to use
  • Run the tool
  • Commands
    • compare
      • Options:
      • Example of usage:
      • Error Handling Options
        • Exit Codes
• GitHub Actions Setup
• Acknowledgments
• Contribution

Prepare environment

Install Tool

This package is available through Nuget Repository: https://www.nuget.org/packages/PowerUtils.BenchmarkDotnet.Reporter

Locally

# Create a tool manifest file in the current directory
dotnet new tool-manifest
# Install the tool in the current directory
dotnet tool install PowerUtils.BenchmarkDotnet.Reporter

Globally

dotnet tool install --global PowerUtils.BenchmarkDotnet.Reporter

Update Tool

Locally

dotnet tool update PowerUtils.BenchmarkDotnet.Reporter

Update Tool

dotnet tool update --global PowerUtils.BenchmarkDotnet.Reporter

List installed tools

Locally

dotnet tool list

Globally

dotnet tool list --global

Uninstall Tool

Locally

dotnet tool uninstall PowerUtils.BenchmarkDotnet.Reporter

Globally

dotnet tool uninstall --global PowerUtils.BenchmarkDotnet.Reporter

Export reports using BenchmarkDotNet

To use the PowerUtils.BenchmarkDotnet.Reporter tool, you first need to generate benchmark reports using BenchmarkDotNet and export them in JSON Full format. You can do this by looking at the documentation of BenchmarkDotNet here or following the examples below:

// TheBenchmark.cs
using BenchmarkDotNet.Attributes;

[MemoryDiagnoser]
[JsonExporterAttribute.Full]
public class TheBenchmark
{
    [Benchmark]
    public void TheMethod()
    {
        // Your benchmark code here
    }
}


// Program.cs
using BenchmarkDotNet.Running;

BenchmarkRunner.Run<TheBenchmark>();

Or

// TheBenchmark.cs
using BenchmarkDotNet.Attributes;

[MemoryDiagnoser]
public class TheBenchmark
{
    [Benchmark]
    public void TheMethod()
    {
        // Your benchmark code here
    }
}


// Program.cs
using BenchmarkDotNet.Running;

var config = ManualConfig
    .Create(DefaultConfig.Instance)
    .AddExporter(JsonExporter.Full);

BenchmarkRunner.Run<TheBenchmark>(config);

How to use

Run the tool

Locally

dotnet pbreporter [command] [options]

Globally

pbreporter [command] [options]

Commands

compare

Compares two benchmark reports and generates a report with the differences.

Example:

pbreporter compare -b baseline-full.json -t target-full.json

Options:

• (-b, --baseline) <baseline>: Path to the folder or file with Baseline report. [Required]
• (-t, --target) <target>: Path to the folder or file with target reports. [Required]
• (-tm, --threshold-mean) <threshold-mean>: Throw an error when the mean threshold is met. Examples: 5%, 10ms, 10μs, 100ns, 1s.
• (-ta, --threshold-allocation) <threshold-allocation>: Throw an error when the allocation threshold is met. Examples: 5%, 10b, 10kb, 100mb, 1gb.
• (-f, --format) <console|hit-txt|json|markdown>: Output format for the report. [default: console]
• (-o, --output) <output>: Output directory to export the diff report. Default is current directory. [default: ./BenchmarkReporter]
• (-fw, --fail-on-warnings): Exit with error code when any threshold is hit during comparison. [default: disabled]
• (-ft, --fail-on-threshold-hit): Exit with error code when any threshold is hit during comparison. [default: disabled]
• (-?, -h, --help): Show help and usage information

Example of usage:

Simple usage

pbreporter compare -b baseline-full.json -t target-full.json

With output format and directory

pbreporter compare -b baseline-full.json -t target-full.json -f json -f markdown -o ./out

With thresholds

pbreporter compare -b baseline-full.json -t target-full.json -tm 5% -ta 12b

With thresholds and output threshold report

pbreporter compare -b baseline-full.json -t target-full.json -tm 5% -f hit-txt

Note: The hit-txt format will only generate when at least one threshold is hit.

With console output

pbreporter compare -b baseline-full.json -t target-full.json -f console

Note: The console format displays the comparison report directly in the terminal instead of creating a file.

With Markdown output

pbreporter compare -b baseline-full.json -t target-full.json -f markdown

Note: The markdown format is ideal for generating reports to upload to GitHub or other platforms that support Markdown rendering.

With multiple formats

pbreporter compare -b baseline-full.json -t target-full.json -f json -f markdown -f console

Error Handling Options

The tool provides options to control exit codes for CI/CD integration and automated quality gates.

Fail on warnings

pbreporter compare -b baseline-full.json -t target-full.json -fw

Note: Exits with code 1 if any warnings are generated during comparison (e.g., environment differences between baseline and target).

Fail on threshold hits

pbreporter compare -b baseline-full.json -t target-full.json -tm 5% -ta 10% -ft

Note: Exits with code 2 if any performance thresholds are exceeded during comparison.

Both error handling options

pbreporter compare -b baseline-full.json -t target-full.json -tm 5% -fw -ft

Note: If both conditions are met, warnings take priority and the tool exits with code 1.

Exit Codes

• 0: Success - No issues detected
• 1: Warnings detected (when --fail-on-warnings is enabled)
• 2: Performance thresholds exceeded (when --fail-on-threshold-hit is enabled)

GitHub Actions Setup

PowerUtils.BenchmarkDotnet.Reporter can be easily integrated into your CI/CD pipeline to automatically analyze and compare benchmark performance. For detailed instructions on setting up GitHub Actions workflows, see the GitHub Actions Setup Guide.

Acknowledgments

This project was inspired by and based on the ResultsComparer tool from the .NET performance repository.

Contribution

If you have any questions, comments, or suggestions, please open an issue or create a pull request.

Want to contribute to this tool? Check out the Test Data Documentation to see sample benchmark reports available for testing your changes and developing new features.