feat: add [PairwiseDataSource] for all-pairs test case generation

docs/docs/writing-tests/data-driven-overview.md

@@ -11,6 +11,7 @@ TUnit offers several ways to provide data to your tests. Use this guide to pick
 | Shared object with lifecycle | `[ClassDataSource<T>]` | [Class Data Source](class-data-source.md) |
 | Reusable data rows | `[TestDataRow<T>]` | [Test Data Row](test-data-row.md) |
 | All parameter combinations | `[MatrixDataSource]` | [Matrix Tests](matrix-tests.md) |
+| Every pair of parameter values (fewer cases) | `[PairwiseDataSource]` | [Pairwise Tests](pairwise-tests.md) |
 | Multiple sources on one method | Combined attributes | [Combined Data Sources](combined-data-source.md) |
 | Hierarchical injection | Nested properties | [Nested Data Sources](nested-data-sources.md) |
 | Custom generic attributes | `[GenerateGenericTest(typeof(...))]` | [Generic Attributes](generic-attributes.md) |
@@ -78,4 +79,5 @@ public async Task Multiply(
 - **`[MethodDataSource]`** is best for computed or complex data.
 - **`[ClassDataSource<T>]`** manages object lifecycles (initialization, disposal, sharing across tests).
 - **`[MatrixDataSource]`** generates the Cartesian product of all `[Matrix]` parameter values.
+- **`[PairwiseDataSource]`** uses the same `[Matrix]` parameters but only covers every pair of values, which is far fewer test cases when there are many parameters.
 - Attributes can be combined on a single method — see [Combined Data Sources](combined-data-source.md).

docs/docs/writing-tests/matrix-tests.md

@@ -6,6 +6,7 @@ The Matrix data source is a way to specify different arguments per parameter, an
 As your number of arguments and/or parameters increase, the number of test cases will grow exponentially.  
 For example, 3 parameters with 10 values each will generate 1,000 test cases (10 × 10 × 10).  
 Use with caution to avoid very large test suites.
+If you don't need every combination, [Pairwise Tests](pairwise-tests.md) use the same `[Matrix]` parameter attributes but only generate enough test cases to cover every pair of values.
 :::
 
 For our arguments, we'll add a `[Matrix]` attribute. Instead of this being added to the test method, it's added to the parameters themselves.

docs/docs/writing-tests/pairwise-tests.md

@@ -0,0 +1,176 @@
+# Pairwise Tests
+
+[Matrix tests](matrix-tests.md) run every combination of parameter values, so the number of test cases grows exponentially with the number of parameters.
+Pairwise testing (also known as *all-pairs* testing) is a well-known way to keep that under control: instead of every combination, it generates a much smaller set of test cases in which **every pair of values, across every pair of parameters, appears at least once**.
+
+Most bugs are triggered by a single value or by the interaction of two parameters, so pairwise testing catches a large share of them with far fewer test cases.
+
+To use it, swap `[MatrixDataSource]` for `[PairwiseDataSource]`. The parameters are declared exactly as for matrix tests:
+
+```csharp
+using TUnit.Assertions;
+using TUnit.Assertions.Extensions;
+using TUnit.Core;
+
+namespace MyTestProject;
+
+public class MyTestClass
+{
+    [Test]
+    [PairwiseDataSource]
+    public async Task MyTest(
+        bool largeArrayPool,
+        bool excessSpan,
+        [Matrix(0, 99, 100, 101)] int sizeHint,
+        [Matrix(1, 2, 3, 100)] int stepCount)
+    {
+        var total = sizeHint + stepCount;
+
+        await Assert.That(total).IsPositive();
+    }
+}
+```
+
+`[MatrixDataSource]` would generate 2 × 2 × 4 × 4 = 64 test cases for this test. `[PairwiseDataSource]` generates 16, and every combination of values for any two of the parameters (for example `excessSpan: true` with `sizeHint: 101`) is still tested.
+
+The savings grow quickly with more parameters:
+
+| Parameter value counts | `[MatrixDataSource]` | `[PairwiseDataSource]` |
+|---|---:|---:|
+| 2 × 2 × 4 × 4 | 64 | 16 |
+| 2 × 2 × 3 × 3 × 2 | 72 | 10 |
+| 3 × 3 × 3 × 3 | 81 | 9 |
+| 3 × 4 × 5 × 6 | 360 | 30 |
+| 10 × 10 × 10 | 1,000 | 106 |
+| 2 (×10 parameters) | 1,024 | 9 |
+
+:::info
+With two parameters, pairwise and matrix generate the same test cases, because every combination *is* a pair.
+Pairwise starts to pay off from three parameters.
+:::
+
+:::warning
+Pairwise testing does not test every combination. If a bug only shows up for a specific combination of three or more parameter values, a pairwise test may not hit it.
+Use `[MatrixDataSource]` when you need exhaustive coverage.
+:::
+
+## Parameter values
+
+`[PairwiseDataSource]` reads parameter values the same way as `[MatrixDataSource]`, so everything described in [Matrix Tests](matrix-tests.md) works here too:
+
+- `[Matrix(...)]` and `[Matrix<T>(...)]` with explicit values, including `Excluding = [...]`
+- `[MatrixRange<T>(...)]` for numeric ranges
+- `[MatrixMethod<T>(...)]` for values returned by a method
+- `bool` and `enum` parameters (and their nullable forms) expand to all of their values automatically. For nullable types, `null` is added after the other values.
+
+```csharp
+using TUnit.Assertions;
+using TUnit.Assertions.Extensions;
+using TUnit.Core;
+
+namespace MyTestProject;
+
+public enum Color
+{
+    Red,
+    Green,
+    Blue
+}
+
+public class MyTestClass
+{
+    [Test]
+    [PairwiseDataSource]
+    public async Task MyTest(
+        Color color,
+        bool? flag,
+        [MatrixRange<int>(1, 5)] int number,
+        [Matrix("x", "y")] string text,
+        [MatrixMethod<MyTestClass>(nameof(Sizes))] int size)
+    {
+        await Assert.That(number).IsBetween(1, 5);
+    }
+
+    public static IEnumerable<int> Sizes()
+    {
+        yield return 1;
+        yield return 10;
+        yield return 100;
+    }
+}
+```
+
+This generates 17 test cases instead of 270.
+
+## Exclusions
+
+`[MatrixExclusion(...)]` works with `[PairwiseDataSource]` as well. Exclusions are applied *while* the test cases are generated, not by filtering afterwards, so every pair of values that can still appear in an allowed test case remains covered. Pairs that no allowed test case can contain (because your exclusions rule out every row with that pair) are skipped silently, so over-excluding produces fewer test cases rather than an error.
+
+```csharp
+using TUnit.Assertions;
+using TUnit.Assertions.Extensions;
+using TUnit.Core;
+
+namespace MyTestProject;
+
+public class MyTestClass
+{
+    [Test]
+    [PairwiseDataSource]
+    [MatrixExclusion(1, 1, 1)]
+    [MatrixExclusion(2, 2, 2)]
+    [MatrixExclusion(3, 3, 3)]
+    public async Task MyTest(
+        [Matrix(1, 2, 3)] int value1,
+        [Matrix(1, 2, 3)] int value2,
+        [Matrix(1, 2, 3)] int value3)
+    {
+        var allEqual = value1 == value2 && value2 == value3;
+
+        await Assert.That(allEqual).IsFalse();
+    }
+}
+```
+
+## Determinism and seeds
+
+The generated test cases are deterministic: the same parameter values always produce the same test cases, so test IDs are stable between runs and between source-generated and reflection mode.
+
+The algorithm uses a seeded pseudo-random number generator to pick test cases. You can change the seed with the `Seed` property to get a different, equally valid, set of test cases:
+
+```csharp
+using TUnit.Assertions;
+using TUnit.Assertions.Extensions;
+using TUnit.Core;
+
+namespace MyTestProject;
+
+public class MyTestClass
+{
+    [Test]
+    [PairwiseDataSource(Seed = 42)]
+    public async Task MyTest(
+        [Matrix(1, 2, 3)] int a,
+        [Matrix(1, 2, 3, 4)] int b,
+        [Matrix(1, 2, 3, 4, 5)] int c)
+    {
+        await Assert.That(a + b + c).IsPositive();
+    }
+}
+```
+
+:::note
+Changing the parameter values (or their order) can change which test cases are generated, not just add or remove the cases for the changed value.
+:::
+
+## Limitations
+
+- Like `[MatrixDataSource]`, `[PairwiseDataSource]` only supports test method parameters. It can't be used for class constructor parameters.
+- Pairwise coverage only guarantees that every *pair* of values is tested. It does not try to cover combinations of three or more values.
+- When many values are combined with heavy `[MatrixExclusion]` sets, discovery may still probe a large search space for unreachable pairs. Prefer fewer values or fewer exclusions when generation is slow.
+- Do not apply `[MatrixDataSource]` and `[PairwiseDataSource]` to the same method or class. Each attribute is its own data source, so you would get both the full Cartesian product and a pairwise covering set rather than one coherent strategy.
+
+## Algorithm
+
+The pairwise algorithm is based on Bob Jenkins' [jenny](https://burtleburtle.net/bob/math/jenny.html) tool, by way of NUnit's `PairwiseStrategy` and [Xunit.Combinatorial](https://github.com/AArnott/Xunit.Combinatorial)'s `[PairwiseData]`.
+It uses the same default seed as Xunit.Combinatorial, so tests migrated from `[PairwiseData]` generate the same number of test cases.

docs/sidebars.ts

@@ -31,6 +31,7 @@ const sidebars: SidebarsConfig = {
             'writing-tests/test-data-row',
             'writing-tests/defer-enumeration',
             'writing-tests/matrix-tests',
+            'writing-tests/pairwise-tests',
             'writing-tests/combined-data-source',
             'writing-tests/nested-data-sources',
             'writing-tests/generic-attributes',

src/TUnit.Analyzers.CodeFixers/Base/TestAttributeEnsurer.cs

@@ -15,7 +15,8 @@ public class TestAttributeEnsurer : CSharpSyntaxRewriter
         "Arguments",
         "MethodDataSource",
         "ClassDataSource",
-        "MatrixDataSource"
+        "MatrixDataSource",
+        "PairwiseDataSource"
     ];
 
     public override SyntaxNode? VisitMethodDeclaration(MethodDeclarationSyntax node)

src/TUnit.Analyzers.CodeFixers/MatrixDataSourceCodeFixProvider.cs

@@ -12,7 +12,8 @@ namespace TUnit.Analyzers.CodeFixers;
 [ExportCodeFixProvider(LanguageNames.CSharp, Name = nameof(MatrixDataSourceCodeFixProvider)), Shared]
 public class MatrixDataSourceCodeFixProvider : CodeFixProvider
 {
-    private const string Title = "Add [MatrixDataSource]";
+    private const string MatrixTitle = "Add [MatrixDataSource]";
+    private const string PairwiseTitle = "Add [PairwiseDataSource]";
 
     public sealed override ImmutableArray<string> FixableDiagnosticIds { get; } =
         ImmutableArray.Create(DiagnosticIds.MatrixDataSourceAttributeRequired);
@@ -39,19 +40,35 @@ public sealed override async Task RegisterCodeFixesAsync(CodeFixContext context)
 
             context.RegisterCodeFix(
                 CodeAction.Create(
-                    title: Title,
-                    createChangedDocument: c => AddMatrixDataSourceAsync(context.Document, target, c),
-                    equivalenceKey: Title),
+                    title: MatrixTitle,
+                    createChangedDocument: c => AddDataSourceAsync(context.Document, target, "MatrixDataSource", c),
+                    equivalenceKey: MatrixTitle),
                 diagnostic);
+
+            // [PairwiseDataSource] implements IAccessesInstanceData, which TestBuilder rejects for
+            // class (constructor) data sources, so only offer it for test methods.
+            if (target is MethodDeclarationSyntax)
+            {
+                context.RegisterCodeFix(
+                    CodeAction.Create(
+                        title: PairwiseTitle,
+                        createChangedDocument: c => AddDataSourceAsync(context.Document, target, "PairwiseDataSource", c),
+                        equivalenceKey: PairwiseTitle),
+                    diagnostic);
+            }
         }
     }
 
-    private static async Task<Document> AddMatrixDataSourceAsync(Document document, SyntaxNode target, CancellationToken cancellationToken)
+    private static async Task<Document> AddDataSourceAsync(
+        Document document,
+        SyntaxNode target,
+        string attributeName,
+        CancellationToken cancellationToken)
     {
         var editor = await DocumentEditor.CreateAsync(document, cancellationToken).ConfigureAwait(false);
         var attributeList = SyntaxFactory.AttributeList(
             SyntaxFactory.SingletonSeparatedList(
-                SyntaxFactory.Attribute(SyntaxFactory.IdentifierName("MatrixDataSource"))));
+                SyntaxFactory.Attribute(SyntaxFactory.IdentifierName(attributeName))));
 
         SyntaxNode updated = target switch
         {

src/TUnit.Analyzers/CombinedDataSourceAnalyzer.cs

@@ -105,9 +105,9 @@ private void CheckCombinedDataSourceErrors(SymbolAnalysisContext context,
                 }
             }
 
-            // Rule 3: Warn if mixing CombinedDataSources with MatrixDataSource
-            var hasMatrixDataSource = attributes.Any(x => x.IsMatrixDataSourceAttribute(context.Compilation));
-            if (hasMatrixDataSource)
+            // Rule 3: Warn if mixing CombinedDataSources with MatrixDataSource or PairwiseDataSource
+            var hasMatrixValuesDataSource = attributes.Any(x => x.IsMatrixValuesDataSourceAttribute(context.Compilation));
+            if (hasMatrixValuesDataSource)
             {
                 context.ReportDiagnostic(
                     Diagnostic.Create(Rules.CombinedDataSourceConflictWithMatrix,

src/TUnit.Analyzers/Extensions/AttributeExtensions.cs

@@ -97,6 +97,22 @@ public static bool IsMatrixDataSourceAttribute(this AttributeData attributeData,
             TUnitSymbols.For(compilation).MatrixDataSourceAttribute);
     }
 
+    public static bool IsPairwiseDataSourceAttribute(this AttributeData attributeData, Compilation compilation)
+    {
+        return SymbolEqualityComparer.Default.Equals(attributeData.AttributeClass,
+            TUnitSymbols.For(compilation).PairwiseDataSourceAttribute);
+    }
+
+    /// <summary>
+    /// Returns true for data sources that generate test cases from the [Matrix] values on parameters
+    /// ([MatrixDataSource] and [PairwiseDataSource]).
+    /// </summary>
+    public static bool IsMatrixValuesDataSourceAttribute(this AttributeData attributeData, Compilation compilation)
+    {
+        return attributeData.IsMatrixDataSourceAttribute(compilation)
+            || attributeData.IsPairwiseDataSourceAttribute(compilation);
+    }
+
     public static bool IsCombinedDataSourceAttribute(this AttributeData attributeData, Compilation compilation)
     {
         return SymbolEqualityComparer.Default.Equals(attributeData.AttributeClass,

src/TUnit.Analyzers/Helpers/TUnitSymbols.cs

@@ -38,6 +38,7 @@ private TUnitSymbols(Compilation compilation)
         BaseTestAttribute = compilation.GetTypeByMetadataName("TUnit.Core.BaseTestAttribute");
         MatrixAttribute = compilation.GetTypeByMetadataName(WellKnown.AttributeFullyQualifiedClasses.Matrix.WithoutGlobalPrefix);
         MatrixDataSourceAttribute = compilation.GetTypeByMetadataName(WellKnown.AttributeFullyQualifiedClasses.MatrixDataSourceAttribute.WithoutGlobalPrefix);
+        PairwiseDataSourceAttribute = compilation.GetTypeByMetadataName(WellKnown.AttributeFullyQualifiedClasses.PairwiseDataSourceAttribute.WithoutGlobalPrefix);
         CombinedDataSourceAttribute = compilation.GetTypeByMetadataName(WellKnown.AttributeFullyQualifiedClasses.CombinedDataSourceAttribute.WithoutGlobalPrefix);
         DataSourceAttributeInterface = compilation.GetTypeByMetadataName(WellKnown.AttributeFullyQualifiedClasses.IDataSourceAttribute.WithoutGlobalPrefix);
     }
@@ -60,6 +61,8 @@ private TUnitSymbols(Compilation compilation)
 
     public INamedTypeSymbol? MatrixDataSourceAttribute { get; }
 
+    public INamedTypeSymbol? PairwiseDataSourceAttribute { get; }
+
     public INamedTypeSymbol? CombinedDataSourceAttribute { get; }
 
     public INamedTypeSymbol? DataSourceAttributeInterface { get; }

src/TUnit.Analyzers/Helpers/WellKnown.cs

@@ -8,6 +8,7 @@ public static class AttributeFullyQualifiedClasses
         public static readonly FullyQualifiedTypeName Explicit = GetTypeName("ExplicitAttribute");
         public static readonly FullyQualifiedTypeName Matrix = GetTypeName("MatrixAttribute");
         public static readonly FullyQualifiedTypeName MatrixDataSourceAttribute = GetTypeName("MatrixDataSourceAttribute");
+        public static readonly FullyQualifiedTypeName PairwiseDataSourceAttribute = GetTypeName("PairwiseDataSourceAttribute");
         public static readonly FullyQualifiedTypeName CombinedDataSourceAttribute = GetTypeName("CombinedDataSourcesAttribute");
 
         public static readonly FullyQualifiedTypeName BeforeAttribute = GetTypeName("BeforeAttribute");

src/TUnit.Analyzers/MatrixAnalyzer.cs

@@ -66,7 +66,7 @@ private void CheckMatrixErrors(SymbolAnalysisContext context, ImmutableArray<Att
             return;
         }
 
-        if (!attributes.Any(x => x.IsMatrixDataSourceAttribute(context.Compilation)))
+        if (!attributes.Any(x => x.IsMatrixValuesDataSourceAttribute(context.Compilation)))
         {
             context.ReportDiagnostic(
                 Diagnostic.Create(Rules.MatrixDataSourceAttributeRequired,

src/TUnit.Analyzers/Resources.resx

@@ -346,13 +346,13 @@
         <value>Test methods must not be static</value>
     </data>
     <data name="TUnit0049Description" xml:space="preserve">
-        <value>[MatrixDataSourceAttribute] is required if using [Matrix] values on your parameters.</value>
+        <value>[MatrixDataSource] or [PairwiseDataSource] is required if using [Matrix] values on your parameters.</value>
     </data>
     <data name="TUnit0049MessageFormat" xml:space="preserve">
-        <value>[MatrixDataSourceAttribute] is required if using [Matrix] values on your parameters.</value>
+        <value>[MatrixDataSource] or [PairwiseDataSource] is required if using [Matrix] values on your parameters.</value>
     </data>
     <data name="TUnit0049Title" xml:space="preserve">
-        <value>[MatrixDataSourceAttribute] is required</value>
+        <value>[MatrixDataSource] or [PairwiseDataSource] is required</value>
     </data>
     <data name="TUnit0050Description" xml:space="preserve">
         <value>Too many arguments provided.</value>
@@ -481,10 +481,10 @@
         <value>Parameter missing data source attribute</value>
     </data>
     <data name="TUnit0072Description" xml:space="preserve">
-        <value>Using [CombinedDataSources] together with [MatrixDataSource] is not recommended as they serve different purposes and may cause confusion.</value>
+        <value>Using [CombinedDataSources] together with [MatrixDataSource] or [PairwiseDataSource] is not recommended as they serve different purposes and may cause confusion.</value>
     </data>
     <data name="TUnit0072MessageFormat" xml:space="preserve">
-        <value>[CombinedDataSources] should not be used with [MatrixDataSource]. Use one or the other.</value>
+        <value>[CombinedDataSources] should not be used with [MatrixDataSource] or [PairwiseDataSource]. Use one or the other.</value>
     </data>
     <data name="TUnit0072Title" xml:space="preserve">
         <value>Conflicting data source attributes</value>