Add BenchmarkDotNet performance benchmarks with automated historical tracking

[Copilot]

Description

Adds a Dapr.Benchmarks project that uses BenchmarkDotNet with Dapr.Testcontainers to benchmark SDK operations against a real Dapr sidecar — no gRPC mocking required. Includes a GitHub Actions workflow that automatically captures and publishes resource utilization metrics (latency, memory allocations, package size) on a per-release basis via an interactive GitHub Pages dashboard.

Benchmark project (test/Dapr.Benchmarks/)

• StateStoreBenchmarks — Save/Get/Delete with Small/Medium/Large payloads against Redis state store
• PubSubBenchmarks — Publish and end-to-end Publish+Receive via RabbitMQ
• WorkflowBenchmarks — Simple single-activity and fan-out (5 parallel activities) workflow execution
• DaprBenchmarkBase — Shared infrastructure managing DaprTestEnvironment lifecycle for [GlobalSetup]/[GlobalCleanup]
• All benchmarks use [MemoryDiagnoser] for allocation tracking
• JSON exporter enabled for CI consumption by tracking tools

Automated metrics tracking (.github/workflows/benchmarks.yml)

• Integrates benchmark-action/github-action-benchmark for historical trend visualization
• Metrics captured per tagged release:

  Metric	Source	Unit
  Latency (mean)	BenchmarkDotNet	ns/μs/ms
  Memory allocations	[MemoryDiagnoser]	bytes/op
  Package size	dotnet pack output	KB per .nupkg

• Publishes interactive charts to gh-pages branch at https://dapr.github.io/dotnet-sdk/
• Each framework (net8.0/9.0/10.0) gets its own chart panel; package sizes tracked separately per package
• Alerts on >150% latency regression or >120% package size growth (non-blocking)
• Triggers scoped to tagged release builds only — tag pushes matching v* (GA) and v*-rc* (release candidates), release: published, weekly cron for baseline drift detection, and workflow_dispatch for ad-hoc runs. Master pushes and PRs do not trigger benchmarks (the per-test sidecar startup cost makes per-PR runs prohibitive, and release-cadence data is what drives the per-release improvement narrative).
• Matrix: net8.0, net9.0, net10.0
• Uploads raw BenchmarkResults/ as artifacts (90-day retention)
• Not a release blocker — fully decoupled from sdk_build.yml

The project is IsPackable=false and the name Dapr.Benchmarks intentionally does not match the Dapr.*.Test.csproj / Dapr.IntegrationTest.*.csproj CI test discovery globs, so it won't be picked up by existing test jobs.

Compatibility with latest master — updated to work with the restructured Dapr.Workflow meta-package (explicitly references Dapr.Workflow.Abstractions since it's bundled with PrivateAssets="all") and resolved namespace collision between Dapr.Benchmarks.Workflow and Dapr.Workflow types via a using alias.

# Run all benchmarks
dotnet run -c Release --project test/Dapr.Benchmarks -- --filter '*'

# Single category on a specific TFM
dotnet run -c Release -f net10.0 --project test/Dapr.Benchmarks -- --filter '*StateStore*'

Issue reference

Informational benchmarking infrastructure — no prior issue.

Checklist

• Code compiles correctly
• Created/updated tests
• Extended the documentation

[WhitWaldo]

@copilot Can you please resolve the merge conflict?

[Copilot]

@copilot Can you please resolve the merge conflict?

Resolved in de024a3. The conflict was in all.sln where both this branch and master (#1789 WorkflowRetryPolicy example) added project entries at the same location. Both entries are now included.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 47.06%. Comparing base (d433287) to head (7f2e858).
⚠️ Report is 2 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (d433287) and HEAD (7f2e858). Click for more details.

HEAD has 3 uploads less than BASE

Flag	BASE (d433287)	HEAD (7f2e858)
net10.0	14	11

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #1793       +/-   ##
===========================================
- Coverage   63.54%   47.06%   -16.49%     
===========================================
  Files         313      259       -54     
  Lines        9431     7945     -1486     
  Branches     1106      914      -192     
===========================================
- Hits         5993     3739     -2254     
- Misses       3175     4030      +855     
+ Partials      263      176       -87     

Flag	Coverage Δ
net10.0	47.06% <ø> (-16.48%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[WhitWaldo]

@copilot Can you please resolve the git merge conflicts?

[Copilot]

@copilot Can you please resolve the git merge conflicts?

Resolved in aa8f98c. The conflict was in all.sln where master added Dapr.IntegrationTest.Examples (#1792) and Dapr.IntegrationTest.Actors.Generators (#1791) at the same location as our Dapr.Benchmarks entry. All three project entries are now included.

[WhitWaldo]

@copilot Several packages have been added to this since - can you please revise the PR to reflect the new packages as well?

[Copilot]

@copilot Several packages have been added to this since - can you please revise the PR to reflect the new packages as well?

Merged latest master (including StateManagement, SecretsManagement, Workflow meta-package restructuring, IntegrationTest.Examples, IntegrationTest.Actors.Generators, etc.) and fixed the resulting build errors in the benchmark project. The Dapr.Workflow.Abstractions reference is now explicit (needed since the meta-package bundles it with PrivateAssets="all") and the namespace collision with Dapr.Benchmarks.Workflow is resolved via a using alias.

[WhitWaldo]

@copilot One of the big goals of this PR is to enable me to capture and publish resource utilization metrics (package size, memory usage, latency) on a per-release basis so I might demonstrate continued package improvement from one release to the next. Ideally, this is validating down to the per-integration test level so I can demonstrate improvement over consistent test code, but if it makes more sense to build out a separate application for benchmarking each of the deployed packages, that's fine.

Can you dig into how to best capture and present this sort of information and tie it into the CI pipeline so it's automated?

[Copilot]

@copilot One of the big goals of this PR is to enable me to capture and publish resource utilization metrics (package size, memory usage, latency) on a per-release basis so I might demonstrate continu...

Implemented in the latest commits. Here's the approach:

Metrics captured per commit/release:

Metric	Source	Unit
Latency (mean)	BenchmarkDotNet	ns/μs/ms
Memory allocations	[MemoryDiagnoser]	bytes/op
Package size	dotnet pack output	KB per .nupkg

How it works:

1. BenchmarkDotNet now exports full JSON results (JsonExporter.Full added in Program.cs)
2. The workflow feeds those results + package size data into benchmark-action/github-action-benchmark
3. The action commits historical data points to the gh-pages branch and serves an interactive chart dashboard at https://dapr.github.io/dotnet-sdk/
4. Each framework (net8.0/9.0/10.0) gets its own chart panel; package sizes get a separate panel
5. Runs on every master push, published releases, and weekly — so every release tag gets a data point

Alerting:

• 150% latency regression → comment on commit (non-blocking)

• 120% package size growth → comment on commit (non-blocking)

The package-size job measures every .nupkg produced by dotnet pack, so all deployed packages (Dapr.Client, Dapr.Workflow, Dapr.Messaging, etc.) are tracked individually. This gives per-package granularity for demonstrating improvements release-over-release.

[WhitWaldo]

@copilot How readily can I go through and essentially ensure that every single integration test in a package matching Dapr.IntegrationTest.* is benchmarked? Ideally, this would be resilient enough that as I add tests, it identifies that the tests are new and as I remove them, it can identify that they've been discontinued. For everything else, it shows commit-by-commit changes.

[Copilot]

@copilot How readily can I go through and essentially ensure that every single integration test in a package matching Dapr.IntegrationTest.* is benchmarked? Ideally, this would be resilient enough t...

Pretty readily, but not by wrapping every [Fact] in a BenchmarkDotNet [Benchmark] — that path is impractical and I'd recommend against it. Here's why and what I'd propose instead.

Why not BenchmarkDotNet-wrap every test

Across the Dapr.IntegrationTest.* projects there are ~66 [Fact]/[Theory] methods today. Each one provisions its own DaprTestEnvironment (sidecar + placement + scheduler + Redis/RabbitMQ container) inside the test body — that's ~3–8 s of setup before any assertion runs. BenchmarkDotNet wants warmup + ≥3 iterations per case to produce statistically meaningful numbers; multiplied by 66 cases × 3 TFMs that's hours per CI run, plus the setup cost dwarfs anything we'd measure. You'd be benchmarking Testcontainers startup, not the SDK.

What I'd build instead — "tests-as-metrics" harness

A single new job that treats each integration test as a one-shot timed run and emits its duration + allocations as a metric series. This satisfies your three requirements naturally:

Requirement	How it's met
New tests auto-tracked	New series name appears in the next data point — benchmark-action/github-action-benchmark charts it from that commit forward
Removed tests auto-discontinued	The series simply stops getting new points; existing history stays viewable on the dashboard
Commit-by-commit changes	Same mechanism as the existing benchmarks — one data point per master push / release tag

Mechanism

1. Run tests with structured output: dotnet test <project> --logger "trx;LogFileName=results.trx" --collect:"XPlat Code Coverage" against each Dapr.IntegrationTest.* project. TRX records per-test duration; we run sequentially (-- xUnit.ParallelizeTestCollections=false) for stable timings.
2. Capture allocations per test: small xUnit v3 IAfterTestAttribute/IBeforeTestAttribute pair (or BeforeAfterTestAttribute shim) that brackets each test with GC.GetTotalAllocatedBytes(precise: true) and writes <TestFullName>,<bytes> to a side file. Add it via [assembly: ...] so it covers every test without touching individual files.
3. Convert to benchmark-action JSON: a small tool/script reads the TRX files + allocation log and produces a customSmallerIsBetter JSON like:

   [
     { "name": "Dapr.IntegrationTest.DaprClient.DaprBulkDeleteStateApiKeyTests.DeleteBulkStateAsync_WithDaprApiToken (ms)", "unit": "ms", "value": 4123 },
     { "name": "Dapr.IntegrationTest.DaprClient.DaprBulkDeleteStateApiKeyTests.DeleteBulkStateAsync_WithDaprApiToken (bytes)", "unit": "bytes", "value": 184320 },
     ...
   ]

4. Feed into the existing dashboard: same benchmark-action/github-action-benchmark@v1 step we already use, written under a new benchmark-data-dir-path: 'integration-tests/<framework>'. Add/remove handled by the action automatically.

Trade-offs you should know going in

• Single-shot timings have variance (~10–30 % run-to-run on shared GitHub runners). Good for spotting big regressions, noisy for ≤10 % shifts. Alert threshold would need to be wide (e.g. 175–200 %).
• Total runtime grows with the test suite. Today ~66 tests × ~5 s avg ≈ 5–6 min per TFM. Acceptable on a non-blocking workflow; would need a self-hosted runner long-term.
• Allocations measured per test include test framework + Testcontainers overhead, so absolute numbers aren't pure SDK cost — but the delta between commits is meaningful, which is what you actually care about for regression tracking.
• The richer per-operation, multi-iteration numbers stay in the existing Dapr.Benchmarks project for representative workloads.

Scope of changes if you greenlight this

• New test/Dapr.IntegrationTest.Benchmarks.Shared/ project with the allocation-tracking xUnit attribute
• [assembly: TrackAllocations] added to each Dapr.IntegrationTest.* project (one line each)
• New tools/integration-test-metrics/ console converter (TRX + alloc log → benchmark-action JSON)
• New integration-test-metrics job in .github/workflows/benchmarks.yml

Want me to go ahead with this design, or would you rather I prototype against one project (say Dapr.IntegrationTest.DaprClient) first so you can see the dashboard output before rolling it out across the suite?

[WhitWaldo]

@copilot At the same time, I don't think it makes much sense to run this for every single PR, but rather it's something worth running for release candidate and GA release builds (e.g. any tagged release).

[Copilot]

@copilot At the same time, I don't think it makes much sense to run this for every single PR, but rather it's something worth running for release candidate and GA release builds (e.g. any tagged relea...

Agreed. Scoped the workflow to tagged release builds (v* and v*-rc* tag pushes + release: published) in b7c33b6. Master pushes are no longer a trigger. The weekly cron stays for baseline drift detection between releases, and workflow_dispatch remains for ad-hoc runs.

This trigger policy will also apply to the per-integration-test metrics job from the previous comment once that's built — same gate, so RC + GA each get one data point per test.

Should I go ahead and build the integration-test-as-metrics harness now under this tag-only trigger policy, or prototype it on Dapr.IntegrationTest.DaprClient first?

[WhitWaldo]

@copilot Currently, integration tests often fail due to transient connection issues due to Testcontainers and port issues on the GitHub actions test runners. Until this can be worked out, does it make more sense to build a small benchmark project with some curated "tests" for benchmarking (as these can represent the whole of the project's capabilities but be less dependent on changes from one version to the next? It'd shrink the overall benchmark method count as well. It'd give us some tests that are fairly small and lightweight with others that are more comprehensive and have multiple steps.

[Copilot]

@copilot Currently, integration tests often fail due to transient connection issues due to Testcontainers and port issues on the GitHub actions test runners. Until this can be worked out, does it make...

Yes — that aligns with what this PR already does and I'd recommend staying on this path. The curated approach in test/Dapr.Benchmarks/ sidesteps the Testcontainers/port-flakiness problem the integration suite hits because:

• It runs one shared DaprTestEnvironment per benchmark class via [GlobalSetup]/[GlobalCleanup] instead of standing up a fresh sidecar per test. Fewer container starts = fewer port-collision/handshake-race opportunities.
• It only triggers on tagged releases (v*, v*-rc*) + weekly cron + workflow_dispatch, so transient flakes don't gate PRs and a failed run is just a missing data point, not a release blocker.
• BenchmarkDotNet's warmup + multi-iteration model masks single-call jitter that would torpedo one-shot integration-test timings.

The current three suites are intentionally a mix of lightweight and comprehensive:

Suite	Scope	Style
StateStoreBenchmarks	Save / Get / Delete × Small/Medium/Large payloads (Redis)	Lightweight, single-op
PubSubBenchmarks	Publish + end-to-end Publish→Receive (RabbitMQ)	Mid-weight, two-step
WorkflowBenchmarks	Single-activity + 5-way fan-out (Workflow)	Comprehensive, multi-step

If you want broader coverage of "the whole of the project's capabilities" without growing the method count much, the natural next additions (one method each, same pattern) would be:

1. Actors — Dapr.Actors invoke round-trip + state save/get (covers Dapr.Actors, Dapr.Actors.AspNetCore)
2. Secrets — GetSecretAsync / GetBulkSecretAsync against the local file secret store (covers Dapr.Secrets.Extensions)
3. Configuration — GetConfigurationAsync + a brief SubscribeConfigurationAsync against an in-memory config store
4. Cryptography — EncryptAsync / DecryptAsync round-trip with a small payload (covers Dapr.Cryptography)
5. Jobs (optional) — schedule + receive a one-shot job (covers Dapr.Jobs)

That would land at ~8 benchmark classes / ~15–20 methods total — still well under the integration-test count, with each deployed package represented. Want me to add some/all of these in this PR, or land what's here first and follow up?

[WhitWaldo]

I'm going to close this as unnecessary for now and instead focus on benchmarking between the older and newer SDK packages