Skip to content

Initial ScottPlotExporter with just Bar Plot and Unit Tests - #2560

Merged
AndreyAkinshin merged 6 commits into
dotnet:masterfrom
FlatlinerDOA:exporters-plot
Jul 20, 2024
Merged

Initial ScottPlotExporter with just Bar Plot and Unit Tests#2560
AndreyAkinshin merged 6 commits into
dotnet:masterfrom
FlatlinerDOA:exporters-plot

Conversation

@FlatlinerDOA

Copy link
Copy Markdown
Contributor

This adds a new project called BenchmarkDotNet.Exporters.Plotting.

Currently just supports Bar Plots with configurable options for Width, Height and whether to RotateLabels (for large bar labels defaults to true).

Example Bar Plot:
MockSummary-Escape_ParamsAndArguments-barplot

NOTE: Currently build gets an error about strong naming?

@FlatlinerDOA

Copy link
Copy Markdown
Contributor Author

@dotnet-policy-service agree

Comment thread src/BenchmarkDotNet.Exporters.Plotting/BenchmarkDotNet.Exporters.Plotting.csproj Outdated
Comment thread src/BenchmarkDotNet.Exporters.Plotting/BenchmarkDotNet.Exporters.Plotting.csproj Outdated
@FlatlinerDOA

Copy link
Copy Markdown
Contributor Author

Hey @timcassell and @AndreyAkinshin just checking in for a status update on this one. I'm happy for any feedback. My plan was to add the rest of the charts that the R plot has once this PR was accepted.

@timcassell timcassell left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but @AndreyAkinshin controls the Nuget packages, so he'll need to approve this and get it on Nuget.

@AndreyAkinshin
AndreyAkinshin merged commit e933bb0 into dotnet:master Jul 20, 2024
@AndreyAkinshin

Copy link
Copy Markdown
Member

@FlatlinerDOA thanks!

@AndreyAkinshin AndreyAkinshin added this to the v0.14.0 milestone Jul 20, 2024
@nietras

nietras commented Aug 11, 2024

Copy link
Copy Markdown
Contributor

I have been trying out this new ScottPlotExporter but getting some weird results. Perhaps I am using it wrong?

var exporter = new ScottPlotExporter();
var logger = new AccumulationLogger();
var summaries = BenchmarkSwitcher.FromAssembly(Assembly.GetExecutingAssembly()).Run(args);
foreach (var summary in summaries)
{
    var filePaths = exporter.ExportToFiles(summary, logger).ToList();
    WriteLine(string.Join(Environment.NewLine, filePaths));
}
BenchmarkDotNet v0.14.0, Windows 10 (10.0.19044.3086/21H2/November2021Update)
AMD Ryzen 9 5950X, 1 CPU, 32 logical and 16 physical cores
.NET SDK 8.0.303
  [Host]     : .NET 8.0.7 (8.0.724.31311), X64 RyuJIT AVX2 [AttachedDebugger]
  DefaultJob : .NET 8.0.7 (8.0.724.31311), X64 RyuJIT AVX2


| Method                                  | Mean       | Error     | StdDev    | Ratio | Gen0   | Allocated | Alloc Ratio |
|---------------------------------------- |-----------:|----------:|----------:|------:|-------:|----------:|------------:|
| PlainStruct_DictionaryGet               | 171.754 ns | 1.1416 ns | 1.0678 ns |  1.00 | 0.0110 |     184 B |        1.00 |
| EquatableStruct_DictionaryGet           |  35.244 ns | 0.0589 ns | 0.0551 ns |  0.21 | 0.0019 |      32 B |        0.17 |
| HashStruct_DictionaryGet                | 131.498 ns | 0.6998 ns | 0.6546 ns |  0.77 | 0.0091 |     152 B |        0.83 |
| HashEquatableStruct_DictionaryGet       |   5.831 ns | 0.0057 ns | 0.0050 ns |  0.03 |      - |         - |        0.00 |
| ValueTuple_DictionaryGet                |  14.810 ns | 0.0231 ns | 0.0216 ns |  0.09 |      - |         - |        0.00 |
| RecordStruct_DictionaryGet              |   7.299 ns | 0.0215 ns | 0.0191 ns |  0.04 |      - |         - |        0.00 |
| HashEquatableRecordStruct_DictionaryGet |   5.916 ns | 0.0143 ns | 0.0134 ns |  0.03 |      - |         - |        0.00 |

DictionaryBench-20240811-134553-DictionaryBench-barplot

@FlatlinerDOA

Copy link
Copy Markdown
Contributor Author

Thanks @nietras, it's plotting the mean of all measurements (including warmup). I've logged an issue #2611 and created a PR to fix it.

@nietras

nietras commented Aug 12, 2024

Copy link
Copy Markdown
Contributor

@FlatlinerDOA not sure I fully follow given what I show is way off from plot, e. g. PlainStruct has mean +170 ns but nothing in plot is anywhere near that, all values around 10-12ns... that is they are much lower.

liuhaoyang pushed a commit to dotnetcore/mocha that referenced this pull request Jul 27, 2026
Updated [BenchmarkDotNet](https://github.com/dotnet/BenchmarkDotNet)
from 0.13.12 to 0.15.8.

<details>
<summary>Release notes</summary>

_Sourced from [BenchmarkDotNet's
releases](https://github.com/dotnet/BenchmarkDotNet/releases)._

## 0.15.8

This release adds OpenMetrics exporter support for Prometheus-compatible
metrics export, improves the Roslyn analyzers with multi-target support
and better type checking, and fixes several bugs including process
deadlocks and WASM trimming issues.

## Features

- Add OpenMetrics exporter for Prometheus-compatible metrics output
(#​2801)
- Add Job info to DisassemblyDiagnoser report headers to distinguish
assemblies when using multiple coreruns (#​2884, fixes #​2573)
- Add `NO_COLOR` environment variable support for disabling console
colors (#​2870)

## Improvements

- Multi-target analyzers with improved type assignability checking using
semantic model (#​2866)
- Add new analyzer diagnostic BDN1503 for better argument/params
validation (#​2865, fixes #​2864)
- Use PolySharp for `[DynamicallyAccessedMembers]` attribute polyfill
(#​2883)
- Refactor to use `AsyncProcessOutputReader` for cleaner process output
handling (#​2878)

## Bug Fixes

- Fix process deadlock issue when reading process output (#​2877)
- Fix WASM generated project being trimmed out (#​2872)
- Allow filters to filter out every benchmark from a type without errors
(#​2879, fixes #​2860)
- Fix unhandled exception when running `BenchmarkRunner.Run<T>()` with
arguments on invalid benchmark type (#​2880, fixes #​2724)

## Internal

- Update release workflow for analyzers (#​2882)
- Improve docs building workflow
- Cleanup `#if`-`#endif` preprocessor directives using PolySharp
polyfills (#​2881)

**Full Changelog**:
dotnet/BenchmarkDotNet@v0.15.7...v0.15.8


## 0.15.7

This release introduces Roslyn analyzers to catch incorrect
BenchmarkDotNet usage at compile time, improves .NET Framework version
detection, and updates OS detection support.

## Features

- Add Roslyn analyzers to detect incorrect usage of BenchmarkDotNet at
compile-time (#​2837)
- Validates benchmark class structure (public, non-sealed, generic
constraints)
- Checks `[Arguments]`, `[Params]`, and `[ParamsAllValues]` attribute
usage
  - Verifies `[GenericTypeArguments]` requirements
  - Ensures only one baseline method per category
  - Validates `BenchmarkRunner.Run` invocations

## Improvements

- Improve .NET Framework version detection by retrieving version from
`TargetFrameworkAttribute` (#​2682)
- Bump Perfolizer 0.6.0 → 0.6.1, bringing updated Windows and macOS
version detection in `OsBrandHelper`

## Bug Fixes

- Fix null reference handling and exception logging in `TestCaseFilter`
for the test adapter
- Fix flaky CI tests by increasing build timeout values (#​2854)

## Internal

- Improve release workflow in `release.yaml`

**Full Changelog**:
dotnet/BenchmarkDotNet@v0.15.6...v0.15.7


## 0.15.6

## v0.15.6

This release adds ref struct parameter support for `[ArgumentsSource]`,
fixes Native AOT runtime moniker resolution, and upgrades to Perfolizer
0.6.0 with the new Pragmastat statistical engine.

### Features

- Add ref struct parameter support for `[ArgumentsSource]` attribute,
enabling `Span<T>` and `ReadOnlySpan<char>` parameters (#​2849)

### Bug Fixes

- Fix runtime moniker normalization for Native AOT targets (#​2852)

### Improvements

- Upgrade to Perfolizer 0.6.0 with Pragmastat statistical engine
integration

### Documentation

- Add documentation for breaking changes related to disassembler native
dependencies (#​2836)

### Internal

- Introduce GitHub Actions release workflow

**Full Changelog**:
dotnet/BenchmarkDotNet@v0.15.5...v0.15.6


## 0.15.5

This release fixes job naming consistency when using `--runtimes`,
clamps histogram bin bounds to avoid confusing negative values, and
reduces output directory clutter by filtering unnecessary runtime and
satellite assembly files.

## Features

- Add custom MSBuild targets to remove unnecessary files from the bin
directory (#​2737)
  - Filters out Capstone native binaries for non-target platforms
  - Removes satellite assemblies from Microsoft.CodeAnalysis packages

## Bug Fixes

- Fix job names consistency between `SimpleJobAttribute` and
`--runtimes` CLI option (#​2841)
- Jobs now use runtime names as IDs consistently across all runtime
monikers
- Clamp histogram bin lower bounds to non-negative values (#​1821)
- Prevents confusing negative values in histogram output for
non-negative measurements

## Internal

- Bump Perfolizer: 0.5.3 → 0.5.4 (#​2773)
- Update changelog and GitHub Pages generation workflows
- Enable `workflow_dispatch` for test workflow (#​2835)

**Full Changelog**:
dotnet/BenchmarkDotNet@v0.15.4...v0.15.5


## 0.15.4

This release fixes issues with `ParamsSource` attribute resolution in
inheritance scenarios and corrects a MSBuild syntax error in the
TestAdapter.

## Bug Fixes

- Allow `[ParamsSource]` to resolve overridden methods and properties in
derived classes (#​2832)
- Fix MSBuild condition syntax for `TestTfmsInParallel` property that
prevented Visual Studio from loading projects (#​2831)

**Full Changelog**:
dotnet/BenchmarkDotNet@v0.15.3...v0.15.4


## 0.15.3

This release brings .NET 10 NativeAOT instruction set support, improved
CPU detection on Windows when WMIC is unavailable, test adapter
filtering, and numerous bug fixes.

## Breaking Changes

- Deprecated `.WithNuget()` job extension in favor of
`.WithMsBuildArguments()` (#​2812)

## Features

- Add VS Test Adapter filter support for running specific benchmarks
(#​2788)
- Update NativeAOT instruction set support for .NET 10+ (#​2828)

## Improvements

- Add PowerShell-based CPU detection fallback for Windows when WMIC is
unavailable (#​2749)
- Improve `IsNetCore` and `IsNativeAOT` detection for single-file apps
without AOT (#​2799)
- Use `--nodeReuse:false` for dotnet CLI commands to improve build
isolation (#​2814)
- Enable assembly signing for debug builds (#​2774)

## Bug Fixes

- Fix `ArgumentsSource` on external types not working if the argument
type is not primitive (#​2820)
- Fix workload warmup mode not working correctly
- Fix `EtwProfiler` for file paths slightly under 260 characters
(#​2808)
- Fix console logs being output twice when using TestAdapter (#​2790)
- Fix `EventProcessor.OnEndValidationStage` not being called when
critical validation errors occur (#​2816)
- Fix `XmlException` thrown when `TextReader.Null` is passed to
`AppConfigGenerator` (#​2817)
- Fix case sensitivity issue in `NativeMemoryLogParser` program name
matching (#​2795)
- Fix typo in `BuildPlots.R`

## Internal

- Replace StyleCop.Analyzers with unstable version for improved analysis
(#​2796)
- Add workflow to run selected tests (#​2797)
- Fix flaky `MemoryDiagnoser` tests on macOS (#​2813)
- Fix x86 disassembler tests for net462 (#​2792)
- Split `TimeConsumingBenchmark` class to reduce test time
- Update `BenchmarkDotNetDiagnosers` package version (#​2805)
- Fix comment in package props about `GenerateProgramFile` (#​2802)

**Full Changelog**:
dotnet/BenchmarkDotNet@v0.15.2...v0.15.3


## 0.15.2

This release improves memory allocation measurement accuracy and adds
new features for job ordering and runtime validation.

## Features

- Add `JobOrderPolicy` option to sort jobs in numeric order instead of
ordinal order (#​2770)
- Add `RuntimeValidator` to detect benchmarks with null runtime
configuration (#​2771)

## Improvements

- Improve memory diagnoser accuracy with better allocation measurement
isolation, warm-up phase, and handling of tiered JIT (#​2562)

## Bug Fixes

- Auto-generate unique job IDs between benchmark runs to prevent ID
collisions
- Skip null runtime validation for in-process toolchain (#​2780)
- Fix flaky memory allocation test (#​2782)
- Fix benchmark test adapter enumeration issues (#​2766)

## Internal

- Modify macOS runner image for CI (#​2775)
- Add setting to skip test reports when original workflow is cancelled
(#​2772)
- Suppress xunit non-serializable data warnings (#​2769)
- Enable `--force-clone` for docs-fetch in generate-gh-pages workflow
- Allow `workflow_dispatch` for publish-nightly workflow
- Enhance docs-fetch command with additional options
- Remove `docs/_changelog` folder from main branch (migrated to
docs-changelog branch)

**Full Changelog**:
dotnet/BenchmarkDotNet@v0.15.1...v0.15.2


## 0.15.1

A maintenance release with improved cross-platform compatibility, a new
feature for referencing external types in source attributes, and several
bug fixes for ARM CPUs and unsupported operating systems.

## Features

- Allow `[ArgumentsSource]` and `[ParamsSource]` to reference methods in
other types via new constructor overload:
`[ArgumentsSource(typeof(MyClass), nameof(MyClass.Values))]` (#​2748)

## Bug Fixes

- Fix WakeLock P/Invoke compatibility with ARM CPUs by refactoring
`REASON_CONTEXT` to use proper union structure (#​2745, #​2756)
- Fix `Console.CancelKeyPress` crash on platforms that don't support it
(Android, iOS, tvOS, WASM) (#​2739, #​2741)
- Fix CPU detection crash on unsupported operating systems by returning
`CpuInfo.Unknown` (#​2740)
- Support `.slnx` solution file format when searching for solution files
(#​2764)

## Improvements

- Bump Perfolizer: 0.5.2 → 0.5.3
- Make `ExporterBase.GetArtifactFullName` accessibility modifier more
permissive

## Internal

- Update .NET SDK version to 8.0.410 (#​2762)
- Update Microsoft.NET.Test.Sdk and other package dependencies (#​2750,
#​2755)
- Rework changelog generation to use `docs-changelog` branch
(#​93d12c42)
- Fix line-endings to LF in several files
- Update GitHub Actions workflows

**Full Changelog**:
dotnet/BenchmarkDotNet@v0.15.0...v0.15.1


## 0.15.0

BenchmarkDotNet v0.15.0 brings .NET 10 support, a new WakeLock feature
to prevent system sleep during benchmarks, improved engine internals for
more consistent measurements, and numerous bug fixes and improvements.

## Features

- **WakeLock support**: New `[WakeLock]` attribute and `--wakeLock` CLI
option to prevent the system from entering sleep mode while benchmarks
are running (#​2670)
- **.NET 10 support**: Added `RuntimeMoniker.Net10`, `NativeAot10`, and
`Mono10` with full toolchain support (#​2642)
- **Box plots in ScottPlotExporter**: New box plot visualization for
benchmark results with improved font sizing
- **RiscV64 platform support**: Added `Platform.RiscV64` for RISC-V
64-bit architecture (#​2644, #​2647)
- **Required properties support**: Benchmark classes can now use C# 11
`required` properties (#​2579)
- **HostSignal.AfterProcessStart**: New signal allows diagnosers to
obtain the process ID of a benchmark process started in suspended state
(#​2674)
- **Parallel build control**: New `ConfigOptions.DisableParallelBuild`
option to force sequential builds (#​2725)
- **Auto-hide empty metric columns**: `ThreadingDiagnoser` and
`ExceptionDiagnoser` now support configuration to hide columns when
metrics have no values (#​2673)
- **Measurements in DiagnoserResults**: Custom diagnosers can now access
measurements for calculations (#​2731)

## Improvements

- **Constant stack size engine refactoring**: Engine stages refactored
to use `IEngineStageEvaluator` for more consistent instruction location
and simpler code (#​2688)
- **Use ArtifactsPath instead of IntermediateOutputPath**: Improved
build artifact handling for SDK 8+ (#​2676)
- **InProcessNoEmitRunner NativeAOT support**: Basic support for running
`InProcessNoEmitRunner` with NativeAOT (#​2702)
- **Allow ParamsAttribute values from derived classes**:
`ParamsAttribute.Values` setter is now protected instead of private
(#​2716)
- **Updated clrmd to 3.1**: Disassembler now uses `ClrMdV3Disassembler`
(#​2488)
- **Updated ScottPlot to 5.0.54**: Plotting exporter updated to latest
ScottPlot version (#​2709)
- **Perfolizer upgraded to 0.5.2**: CPU/OS detection logic moved to
Perfolizer with new `Perfonar` exporters replacing `Phd` exporters
- **Log warnings for empty benchmarks**: Clear warning messages when
running benchmarks that match no methods (#​2718)
- **Validation for sealed benchmark classes**: Compiler now warns when
benchmark classes are sealed (#​2660)
- **Improved baseline warning message**: More descriptive warning when
baseline benchmarks are misconfigured (#​2650)

## Bug Fixes

- **Fix async GlobalSetup/GlobalCleanup with InProcessEmit**: Async
setup and cleanup methods are now properly awaited (#​2109)
- **Fix Windows path too long**: Handle Windows MAX_PATH limitations in
build paths (#​2681)
- **Fix builds with --keepFiles**: Include auto-incremented ID in build
artifacts directory to avoid conflicts (#​2423)
- **Fix Ctrl-C handling**: System state (power management, console
title) is now properly reverted on process termination (#​2483, #​2661)
- **Fix dotnet command failure detection**: Commands now always fail
when dotnet returns non-zero exit code (#​2535)
- **Fix deadlock in GetDotNetSdkVersion**: Resolved potential deadlock
in SDK version detection (#​2622)
- **Fix lscpu CPU frequency parsing**: Corrected frequency parsing from
lscpu output
- **Make lscpu call language-invariant**: CPU detection now works
correctly regardless of system locale (#​2577)
- **Clean up unsupported Native AOT flags**: Removed obsolete
`IlcGenerateCompleteTypeMetadata` and updated flag names (#​2616)
- **Native AOT projects copy SettingsWeWantToCopy**: Build settings are
now properly propagated (#​2665)

## Breaking Changes

- **Removed ConfigCompatibilityValidator**: No longer validates config
compatibility between runs (#​2599)
- **Phd exporters renamed to Perfonar**: `[PhdExporter]` →
`[PerfonarExporter]`, `PhdJsonExporter` → `PerfonarJsonExporter`,
`PhdMdExporter` → `PerfonarMdExporter`
- **Removed netstandard1.0 target from Annotations package**: Minimum
target is now netstandard2.0

## Documentation

- Added Visual Studio Profiler documentation with samples (#​2672)
- Added WakeLock documentation and samples (#​2670)
 ... (truncated)

## 0.14.0

Full changelog: https://benchmarkdotnet.org/changelog/v0.14.0.html

## Highlights

* Introduce `BenchmarkDotNet.Diagnostics.dotMemory`
[#​2549](dotnet/BenchmarkDotNet#2549): memory
allocation profile of your benchmarks using
[dotMemory](https://www.jetbrains.com/dotmemory/), see
@​BenchmarkDotNet.Samples.IntroDotMemoryDiagnoser
* Introduce `BenchmarkDotNet.Exporters.Plotting`
[#​2560](dotnet/BenchmarkDotNet#2560): plotting
via [ScottPlot](https://scottplot.net/) (initial version)
* Multiple bugfixes
* The default build toolchains have been updated to pass
`IntermediateOutputPath`, `OutputPath`, and `OutDir` properties to the
`dotnet build` command. This change forces all build outputs to be
placed in a new directory generated by BenchmarkDotNet, and fixes many
issues that have been reported with builds. You can also access these
paths in your own `.csproj` and `.props` from those properties if you
need to copy custom files to the output.

## Bug fixes

* Fixed multiple build-related bugs including passing MsBuildArguments
and .Net 8's `UseArtifactsOutput`.

## Breaking Changes

* `DotNetCliBuilder` removed `retryFailedBuildWithNoDeps` constructor
option.
* `DotNetCliCommand` removed `RetryFailedBuildWithNoDeps` property and
`BuildNoRestoreNoDependencies()` and `PublishNoBuildAndNoRestore()`
methods (replaced with `PublishNoRestore()`).

Commits viewable in [compare
view](dotnet/BenchmarkDotNet@v0.13.12...v0.15.8).
</details>

[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=BenchmarkDotNet&package-manager=nuget&previous-version=0.13.12&new-version=0.15.8)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants