Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 73 additions & 21 deletions docs/docs/benchmarks/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@ sidebar_position: 1

# Performance Benchmarks

:::info Benchmark Data Loading
Benchmarks are automatically updated daily from CI runs. This page will be populated with real data after the first Speed Comparison workflow completes.
:::info Last Updated
These benchmarks were automatically generated on **2025-11-07** from the latest CI run.

In the meantime, check out the [Calculator](/docs/benchmarks/calculator) and [Methodology](/docs/benchmarks/methodology) pages.
**Environment:** Ubuntu Latest • .NET SDK 10.0.100-rc.2.25502.107
:::

## 🎯 Executive Summary
Expand All @@ -20,48 +20,100 @@ TUnit demonstrates significant performance advantages across all testing scenari

### Average Performance vs Other Frameworks

- **2.6x faster** than xUnit v3
- **4.5x faster** than NUnit
- **5.1x faster** than MSTest
- **1.3x faster** than xUnit v3
- **1.2x faster** than NUnit
- **1.3x faster** than MSTest

</div>

---

## 🚀 Runtime Performance

Detailed benchmark results will appear here automatically after the Speed Comparison workflow runs.

### Running Benchmarks Manually
### results

You can trigger benchmarks manually:
:::tip Native AOT Performance
TUnit with Native AOT compilation is **11.65x faster** than regular JIT!
:::

**Performance:** **1.26x faster** than xUnit • **1.24x faster** than NUnit • **1.28x faster** than MSTest

| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| 🏆 **TUnit** | 1.0.0 | 487.37 ms | 487.27 ms | 2.639 ms |
| NUnit | 4.4.0 | 602.15 ms | 601.33 ms | 9.904 ms |
| MSTest | 4.0.1 | 623.12 ms | 621.63 ms | 9.691 ms |
| xUnit3 | 3.2.0 | 614.68 ms | 613.67 ms | 11.426 ms |
| 🏆 **TUnit (AOT)** | 1.0.0 | 41.85 ms | 41.53 ms | 3.056 ms |

1. Go to [GitHub Actions - Speed Comparison](https://github.com/thomhurst/TUnit/actions/workflows/speed-comparison.yml)
2. Click "Run workflow"
3. Wait for completion
4. Results will be committed to this page automatically

---

## 📊 Methodology
## 🔨 Build Performance

Compilation time comparison across frameworks:

These benchmarks compare TUnit against the most popular .NET testing frameworks using [BenchmarkDotNet](https://benchmarkdotnet.org/).
| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| 🏆 **TUnit** | 1.0.0 | 1.707 s | 1.707 s | 0.0314 s |
| Build_NUnit | 4.4.0 | 1.465 s | 1.466 s | 0.0170 s |
| Build_MSTest | 4.0.1 | 1.530 s | 1.529 s | 0.0144 s |
| Build_xUnit3 | 3.2.0 | 1.469 s | 1.474 s | 0.0277 s |

For complete methodology details, see the [Methodology](/docs/benchmarks/methodology) page.

---

## Interactive Tools
## 📊 Methodology

These benchmarks compare TUnit against the most popular .NET testing frameworks:

| Framework | Version Tested |
|-----------|----------------|
| **TUnit** | 1.0.0 |
| **xUnit v3** | 3.2.0 |
| **NUnit** | 4.4.0 |
| **MSTest** | 4.0.1 |

### Test Scenarios

The benchmarks measure real-world testing patterns:

- **DataDrivenTests**: Parameterized tests with multiple data sources
- **AsyncTests**: Realistic async/await patterns with I/O simulation
- **ScaleTests**: Large test suites (1000+ tests) measuring scalability
- **MatrixTests**: Combinatorial test generation and execution
- **MassiveParallelTests**: Parallel execution stress tests

### Environment

- **OS**: Ubuntu Latest (GitHub Actions)
- **Runtime**: .NET 10.0.0 (10.0.0-rc.2.25502.107, 10.0.25.50307), X64 RyuJIT x86-64-v3
- **SDK**: .NET SDK 10.0.100-rc.2.25502.107
- **Hardware**: GitHub Actions Standard Runner (Ubuntu)
- **Tool**: BenchmarkDotNet v0.15.6, Linux Ubuntu 24.04.3 LTS (Noble Numbat)

### Why These Numbers Matter

- **No Mocking**: All tests use realistic patterns, not artificial micro-benchmarks
- **Equivalent Logic**: Each framework implements identical test scenarios
- **Warm-Up Excluded**: Measurements exclude JIT warm-up overhead
- **Statistical Rigor**: Multiple iterations with outlier detection

### Source Code

All benchmark source code is available in the [`tools/speed-comparison`](https://github.com/thomhurst/TUnit/tree/main/tools/speed-comparison) directory.

### Interactive Comparison

- **[Benchmark Calculator](/docs/benchmarks/calculator)** - Calculate potential time savings for your test suite
- **[Methodology](/docs/benchmarks/methodology)** - Learn how performance is measured
Want to estimate performance for your test suite? Try the [Benchmark Calculator](/docs/benchmarks/calculator) to see potential time savings.

---

:::note Continuous Benchmarking
These benchmarks run automatically daily via [GitHub Actions](https://github.com/thomhurst/TUnit/actions/workflows/speed-comparison.yml).

Each benchmark runs multiple iterations with statistical analysis to ensure accuracy.
Each benchmark runs multiple iterations with statistical analysis to ensure accuracy. Results may vary based on hardware and test characteristics.
:::

*This page will be automatically updated with real data after the first workflow run.*
*Last generated: 2025-11-07T18:03:14.595Z*
13 changes: 12 additions & 1 deletion docs/static/benchmarks/historical.json
Original file line number Diff line number Diff line change
@@ -1 +1,12 @@
[]
[
{
"date": "2025-11-07",
"averageSpeedups": {
"vsXUnit": "1.3",
"vsNUnit": "1.2",
"vsMSTest": "1.3",
"count": 1
},
"environment": "Ubuntu"
}
]
127 changes: 84 additions & 43 deletions docs/static/benchmarks/latest.json
Original file line number Diff line number Diff line change
@@ -1,69 +1,110 @@
{
"timestamp": "2025-01-07T00:00:00.000Z",
"timestamp": "2025-11-07T18:03:14.595Z",
"environment": {
"os": "Ubuntu Latest",
"sdk": ".NET 10",
"host": ".NET 10.0"
"benchmarkDotNetVersion": "BenchmarkDotNet v0.15.6, Linux Ubuntu 24.04.3 LTS (Noble Numbat)",
"sdk": ".NET SDK 10.0.100-rc.2.25502.107",
"host": ".NET 10.0.0 (10.0.0-rc.2.25502.107, 10.0.25.50307), X64 RyuJIT x86-64-v3"
},
"categories": {
"AsyncTests": [
"results": [
{
"Method": "TUnit",
"Version": "0.x.x",
"Mean": "350.0 ms",
"Median": "348.0 ms",
"StdDev": "12.0 ms"
},
{
"Method": "xUnit3",
"Version": "3.x.x",
"Mean": "910.0 ms",
"Median": "905.0 ms",
"StdDev": "25.0 ms"
"Version": "1.0.0",
"Mean": "487.37 ms",
"Error": "2.977 ms",
"StdDev": "2.639 ms",
"Median": "487.27 ms"
},
{
"Method": "NUnit",
"Version": "4.x.x",
"Mean": "1575.0 ms",
"Median": "1570.0 ms",
"StdDev": "35.0 ms"
"Version": "4.4.0",
"Mean": "602.15 ms",
"Error": "11.172 ms",
"StdDev": "9.904 ms",
"Median": "601.33 ms"
},
{
"Method": "MSTest",
"Version": "4.x.x",
"Mean": "1785.0 ms",
"Median": "1780.0 ms",
"StdDev": "40.0 ms"
"Version": "4.0.1",
"Mean": "623.12 ms",
"Error": "10.932 ms",
"StdDev": "9.691 ms",
"Median": "621.63 ms"
},
{
"Method": "xUnit3",
"Version": "3.2.0",
"Mean": "614.68 ms",
"Error": "12.215 ms",
"StdDev": "11.426 ms",
"Median": "613.67 ms"
},
{
"Method": "TUnit_AOT",
"Version": "0.x.x",
"Mean": "100.0 ms",
"Median": "98.0 ms",
"StdDev": "8.0 ms"
"Version": "1.0.0",
"Mean": "41.85 ms",
"Error": "1.042 ms",
"StdDev": "3.056 ms",
"Median": "41.53 ms"
}
]
},
"build": {
"BuildTime": [
{
"Method": "Build_TUnit",
"Version": "1.0.0",
"Mean": "1.707 s",
"Error": "0.0306 s",
"StdDev": "0.0314 s",
"Median": "1.707 s"
},
{
"Method": "Build_NUnit",
"Version": "4.4.0",
"Mean": "1.465 s",
"Error": "0.0182 s",
"StdDev": "0.0170 s",
"Median": "1.466 s"
},
{
"Method": "Build_MSTest",
"Version": "4.0.1",
"Mean": "1.530 s",
"Error": "0.0163 s",
"StdDev": "0.0144 s",
"Median": "1.529 s"
},
{
"Method": "Build_xUnit3",
"Version": "3.2.0",
"Mean": "1.469 s",
"Error": "0.0282 s",
"StdDev": "0.0277 s",
"Median": "1.474 s"
}
]
},
"build": {},
"comparisons": {
"AsyncTests": {
"tunitMean": 350.0,
"tunitAOTMean": 100.0,
"vsXUnit": "2.6",
"vsNUnit": "4.5",
"vsMSTest": "5.1",
"aotSpeedup": "3.5"
"results": {
"tunitMean": 487.37,
"tunitAOTMean": 41.85,
"vsXUnit": "1.26",
"vsNUnit": "1.24",
"vsMSTest": "1.28",
"aotSpeedup": "11.65"
}
},
"averageSpeedups": {
"vsXUnit": "2.6",
"vsNUnit": "4.5",
"vsMSTest": "5.1"
"vsXUnit": "1.3",
"vsNUnit": "1.2",
"vsMSTest": "1.3",
"count": 1
},
"stats": {
"runtimeCategories": 1,
"buildCategories": 0,
"totalBenchmarks": 1,
"lastUpdated": "2025-01-07T00:00:00.000Z"
"buildCategories": 1,
"totalBenchmarks": 2,
"lastUpdated": "2025-11-07T18:03:14.594Z"
}
}
}