Add test splitting infrastructure with auto-detection support and GitHub Actions integration #12101

Copilot · 2025-10-16T20:20:54Z

Add CI test splitting infrastructure

Summary

Introduces a unified test matrix generation system that enables splitting large test projects into parallel CI jobs
Will allow reducing Aspire.Hosting.Tests wall-clock time from ~17 minutes to multiple ~5 minute parallel jobs
Consolidates test enumeration from 3 separate per-OS setup jobs into a single platform-agnostic pipeline

Motivation

The Aspire.Hosting.Tests project had grown large enough that it was becoming a bottleneck in CI. Rather than adding ad-hoc parallelism, this PR introduces a general-purpose infrastructure that:

Allows any test project to opt into parallel splitting
Keeps the matrix definition platform-agnostic (enabling future Azure DevOps support)
Cleans up the existing test workflow by removing duplicated per-OS setup jobs

How it works

┌─────────────────────────────────────────────────────────────────────┐
│                         MSBuild Phase                               │
│  (TestEnumerationRunsheetBuilder.targets + build-test-matrix.ps1)   │
│                                                                     │
│  .tests-metadata.json  →  .tests-partitions.json  →  canonical-     │
│  (per project)            (split projects)           test-matrix.json
└─────────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    Platform-Specific Expansion                      │
│                                                                     │
│  expand-test-matrix-github.ps1                                      │
│  • Maps OS names to GitHub runners                                  │
│  • Outputs { "include": [...] } format                              │
└─────────────────────────────────────────────────────────────────────┘

Key design decisions:

Canonical matrix format: The core matrix is platform-agnostic JSON with supportedOSes arrays. Thin per-platform scripts expand this into CI-specific formats. This keeps 90% of the logic shared and makes adding Azure DevOps support straightforward.
uncollected:* safety net: When using partition traits, tests without a [Trait("Partition", "...")] attribute automatically run in a dedicated "uncollected" job. This ensures no tests are silently skipped.

How to enable splitting for a test project

Add to the .csproj:

<PropertyGroup>
  <SplitTestsOnCI>true</SplitTestsOnCI>
  <TestClassNamePrefixForCI>Aspire.YourProject.Tests</TestClassNamePrefixForCI>
</PropertyGroup>

The infrastructure will automatically split by test class. For explicit control, add [Trait("Partition", "GroupName")] to test classes (follow-up PR).

What's included

Component	Description
`eng/TestEnumerationRunsheetBuilder/`	MSBuild targets for test enumeration
`eng/scripts/build-test-matrix.ps1`	Generates canonical matrix from metadata
`eng/scripts/expand-test-matrix-github.ps1`	Expands matrix for GitHub Actions
`eng/scripts/split-test-projects-for-ci.ps1`	Discovers partitions/classes for splitting
`tools/ExtractTestPartitions/`	Tool to extract `[Trait("Partition", ...)]` from assemblies
`.github/actions/enumerate-tests/`	Updated GitHub Action
`.github/workflows/tests.yml`	Simplified workflow using unified matrix
`eng/TestingOnCI.md`	Documentation

Test plan

CI passes on this branch
Aspire.Hosting.Tests runs as multiple parallel jobs
Aspire.Templates.Tests runs as multiple parallel jobs
Aspire.Cli.EndToEnd.Tests runs as multiple parallel jobs
No tests are silently skipped (verify via test counts)

Future work

Add [Trait("Partition", "...")] attributes for logical test grouping (follow-up PR)
Add expand-test-matrix-azdo.ps1 for Azure DevOps support

github-actions · 2025-10-16T20:51:20Z

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 12101

Or

Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 12101"

github-actions · 2025-10-16T20:51:20Z

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 12101

Or

Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 12101"

radical · 2025-10-16T21:03:08Z

@copilot Implement (1). And add [Collection] with unique collection names on Aspire.Hosting.Tests.WaitForTests and Aspire.Hosting.Tests.DistributedApplicationTests.

Validate that the generation of tests matrix works correctly. Validate that it gets correctly consumed by the gh actions, yml.

Also make sure the trx filenames are valid for the platform, so escape characters correctly.

Copilot · 2025-10-16T21:17:33Z

@copilot Implement (1). And add [Collection] with unique collection names on Aspire.Hosting.Tests.WaitForTests and Aspire.Hosting.Tests.DistributedApplicationTests.

Validate that the generation...

Implemented all requested changes in commit 85eab86:

✅ Collection attributes added:
- WaitForTests → [Collection("WaitForTests")]
- DistributedApplicationTests → [Collection("DistributedApplicationTests")]
✅ Test splitting enabled for Aspire.Hosting.Tests:
- Matrix generates 66 jobs (40 Hosting + 26 Templates)
- Configured with 30m/15m timeouts
✅ GitHub Actions integration:
- Updated enumerate-tests action to generate and output split_tests_matrix
- Added split_tests_{lin,macos,win} jobs to tests.yml
- Jobs consume matrix from split-tests-matrix.json
✅ TRX filenames are platform-safe:
- Format: {ProjectName}_{ShortName}.trx (e.g., Aspire.Hosting.Tests_WaitForTests.trx)
- Uses only alphanumeric and underscores

Note: Collection detection via xUnit v3 --list-tests doesn't output "Collection:" banners like v2. The system correctly falls back to class-based splitting, which still provides full parallelization benefits.

radical · 2025-10-16T22:05:00Z

@copilot Two issues:

the split tests aren't running because we get --filter-class Aspire.Templates.Tests.class:Aspire.Templates.Tests.XUnit_V3MTP_NewUpAndBuildSupportProjectTemplatesTests instead of --filter-class Aspire.Templates.Tests.XUnit_V3MTP_NewUpAndBuildSupportProjectTemplatesTests.
Collection extraction was skipped because xunit command doesn't provide a way to query that. Instead we can:

Add a tool in tools/ to extract the Collection names. If adding just [Collection("foo")] is not enough then fallback to using a trait like [Trait("Partition", "foo")]. And the tool that find the values of these traits.
Then the tests will be split into - 1 split per unique partition name, and 1 for rest of tests with no partition set

Implement this.

Validate

Validate that the test split generation works correctly.
Validate that the generated command line from the matrix json is correct and works

Copilot · 2025-10-16T22:23:36Z

@copilot Two issues:

the split tests aren't running because we get --filter-class Aspire.Templates.Tests.class:Aspire.Templates.Tests.XUnit_V3MTP_NewUpAndBuildSupportProjectTemplatesTests instea...

Fixed both issues in commit da9cbb3:

Issue 1 - Fixed filterArg format:

Updated enumerate-tests action to strip class:, collection:, or uncollected: prefixes
Filters now correctly show --filter-class "Aspire.Templates.Tests.ClassName" without duplication

Issue 2 - Collection extraction tool:

Created tools/ExtractTestPartitions that uses reflection to extract [Collection] and [Trait("Partition", ...)] attributes
Updated discovery script to call the tool before falling back to class-based mode
Added [Trait("Partition", "name")] to WaitForTests, DistributedApplicationTests, and SlimTestProgramTests
Updated filter generation to use --filter-trait "Partition=Name" format for xUnit v3 compatibility

Validation:

Matrix generates 30 jobs (4 partitions for Hosting + 26 classes for Templates)
--filter-trait "Partition=WaitForTests" returns 38 tests ✅
--filter-class "Aspire.Templates.Tests.BuildAndRunTemplateTests" returns 6 tests ✅
Uncollected filter excludes all partitions correctly (1286 tests) ✅

radical · 2025-10-17T00:02:58Z

@copilot Splitting based on the collection/"Partition" trait doesn't seem to be working. fix and validate that it works, and the correct matrix json is created.

Copilot · 2025-10-17T00:11:48Z

@copilot Splitting based on the collection/"Partition" trait doesn't seem to be working. fix and validate that it works, and the correct matrix json is created.

Partition-based splitting IS working correctly. Here's the validation:

Collection Detection:

ExtractTestPartitions tool found 3 partitions: DistributedApplicationTests (43 tests), SlimTestProgram (3 tests), WaitForTests (38 tests)
Metadata mode: collection ✓

Matrix Generation:

30 total jobs: 4 for Hosting.Tests (3 partitions + 1 uncollected) + 26 for Templates.Tests
Filter args use correct xUnit v3 format: --filter-trait "Partition=Name" ✓

Filter Validation:

$ dotnet Aspire.Hosting.Tests.dll --list-tests --filter-trait "Partition=WaitForTests"
Test discovery summary: found 38 test(s) ✓

$ dotnet Aspire.Hosting.Tests.dll --list-tests --filter-trait "Partition=DistributedApplicationTests"
Test discovery summary: found 43 test(s) ✓

$ dotnet Aspire.Hosting.Tests.dll --list-tests --filter-trait "Partition=SlimTestProgram"
Test discovery summary: found 3 test(s) ✓

Matrix JSON (sample):

{
  "type": "collection",
  "shortname": "Collection_DistributedApplicationTests",
  "filterArg": "--filter-trait \"Partition=DistributedApplicationTests\"",
  "testSessionTimeout": "30m"
}

All tests sum correctly: 38 + 43 + 3 + 1286 uncollected = 1370 total ✓

radical · 2025-10-18T04:31:13Z

/azp run aspire-tests

azure-pipelines · 2025-10-18T04:31:22Z

Azure Pipelines successfully started running 1 pipeline(s).

radical · 2025-10-19T06:58:26Z

@copilot quarantine this test - Aspire.Hosting.Python.Tests.AddPythonAppTests.WithVirtualEnvironment_CanBeChainedWithOtherExtensions

radical · 2025-10-19T07:03:59Z

@copilot Quarantine Aspire.Hosting.Python.Tests.AddPythonAppTests.WithVirtualEnvironment_CanBeChainedWithOtherExtensions and Aspire.Hosting.Tests.Dcp.DcpExecutorTests.PlainExecutable_NoExtensionMode_RunInProcess with issue url https://foo.com/issues/123

radical · 2026-01-21T21:57:21Z

/azp run aspire-tests

azure-pipelines · 2026-01-21T21:57:30Z

Azure Pipelines successfully started running 1 pipeline(s).

The condition was using IsAzdoCIRunner/IsGitHubActionsRunner which are not defined for playground projects (they're computed in Testing.targets which playground doesn't import). Use ContinuousIntegrationBuild instead which is available on all CI systems. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

radical · 2026-01-21T22:30:14Z

/azp run aspire-tests

azure-pipelines · 2026-01-21T22:30:24Z

Azure Pipelines successfully started running 1 pipeline(s).

Instead of requiring an explicit GenerateCIPartitions property, infer partition generation from PrepareForHelix=true. This simplifies the Azure DevOps pipeline and ensures partitions are always generated for Helix builds. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add RunOnAzdoCIWindows/Linux properties to Aspire.Cli.EndToEnd.Tests to run on Azure DevOps build agents (Linux only) - Clean up comments in Infrastructure.Tests for clarity Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Only create test archives and generate partitions for projects that will actually run on Helix. This prevents unnecessary work for projects like Infrastructure.Tests that have RunOnAzdoHelix=false. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

…tting-infrastructure-again

radical · 2026-01-22T18:04:26Z

/azp run aspire-tests

azure-pipelines · 2026-01-22T18:04:36Z

Azure Pipelines successfully started running 1 pipeline(s).

radical · 2026-01-22T18:18:09Z

CI jobs graph goes from:

.. to ..

radical · 2026-01-22T22:32:30Z

/azp run aspire-tests

azure-pipelines · 2026-01-22T22:32:43Z

Azure Pipelines successfully started running 1 pipeline(s).

…tting-infrastructure-again

github-actions · 2026-01-23T02:37:48Z

🎬 CLI E2E Test Recordings

The following terminal recordings are available for commit 2d1f868:

Test	Recording
CreateAndDeployToDockerCompose	▶️ View Recording
CreateAndDeployToDockerComposeInteractive	▶️ View Recording
CreateAndRunAspireStarterProject	▶️ View Recording
CreateAndRunJsReactProject	▶️ View Recording
CreateAndRunPythonReactProject	▶️ View Recording
CreateEmptyAppHostProject	▶️ View Recording
CreateStartAndStopAspireProject	▶️ View Recording
CreateTypeScriptAppHostWithViteApp	▶️ View Recording
DoctorCommand_WithSslCertDir_ShowsTrusted	▶️ View Recording
DoctorCommand_WithoutSslCertDir_ShowsPartiallyTrusted	▶️ View Recording
PsCommandListsRunningAppHost	▶️ View Recording

_{📹 Recordings uploaded automatically from CI run #21272105511}

…tting-infrastructure-again # Conflicts: # .github/workflows/tests.yml

Copilot AI assigned Copilot and radical Oct 16, 2025

Copilot started work on behalf of radical October 16, 2025 20:21 View session

radical changed the base branch from copilot/add-test-splitting-infrastructure to main October 16, 2025 20:21

Copilot AI changed the title ~~[WIP] Add test splitting infrastructure as per documentation~~ Add test splitting infrastructure with auto-detection support Oct 16, 2025

Copilot finished work on behalf of radical October 16, 2025 20:45

Copilot AI requested a review from radical October 16, 2025 20:45

Copilot started work on behalf of radical October 16, 2025 21:03 View session

Copilot AI changed the title ~~Add test splitting infrastructure with auto-detection support~~ Add test splitting infrastructure with auto-detection support and GitHub Actions integration Oct 16, 2025

Copilot finished work on behalf of radical October 16, 2025 21:18

Copilot started work on behalf of radical October 16, 2025 22:05 View session

Copilot finished work on behalf of radical October 16, 2025 22:24

Copilot started work on behalf of radical October 17, 2025 00:03 View session

Copilot finished work on behalf of radical October 17, 2025 00:12

Copilot started work on behalf of radical October 19, 2025 06:58 View session

Copilot finished work on behalf of radical October 19, 2025 07:01

Copilot started work on behalf of radical October 19, 2025 07:04 View session

Copilot stopped work on behalf of radical due to an error October 19, 2025 07:10
Copilot has encountered an error. See logs for additional details.

radical and others added 3 commits January 21, 2026 18:11

fix cli.endtoendtests

23f2101

build-analysis bot mentioned this pull request Jan 22, 2026

[android-arm64] The Operation will be canceled. The next steps may not contain expected logs. dotnet/dnceng#6408

Open

3 tasks

radical and others added 3 commits January 21, 2026 20:13

fix condition to archivetests

0f35c32

Merge remote-tracking branch 'origin/main' into copilot/add-test-spli…

437b6f5

…tting-infrastructure-again

radical added 4 commits January 22, 2026 15:12

Fix up conditions to archive tests

d696787

fix condition

3e44334

fix up condition

37e5afa

fixy

3477534

radical marked this pull request as draft January 22, 2026 22:51

radical added 4 commits January 22, 2026 17:54

Merge remote-tracking branch 'origin/main' into copilot/add-test-spli…

9aa3b15

…tting-infrastructure-again

Merge remote-tracking branch 'origin/main' into copilot/add-test-spli…

246a758

…tting-infrastructure-again

fix

2d1f868

fixy

b2d5ff4

radical added 3 commits January 24, 2026 16:40

Merge remote-tracking branch 'origin/main' into copilot/add-test-spli…

30e65ce

…tting-infrastructure-again # Conflicts: # .github/workflows/tests.yml

fix merge

a7b14cb

fixy

f9ebad7

Add test splitting infrastructure with auto-detection support and GitHub Actions integration #12101

Are you sure you want to change the base?

Add test splitting infrastructure with auto-detection support and GitHub Actions integration #12101

Conversation

Copilot AI commented Oct 16, 2025 • edited by radical Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add CI test splitting infrastructure

Summary

Motivation

How it works

How to enable splitting for a test project

What's included

Test plan

Future work

Uh oh!

github-actions bot commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 16, 2025

Uh oh!

radical commented Oct 16, 2025

Uh oh!

Copilot AI commented Oct 16, 2025

Uh oh!

radical commented Oct 16, 2025

Validate

Uh oh!

Copilot AI commented Oct 16, 2025

Uh oh!

radical commented Oct 17, 2025

Uh oh!

Copilot AI commented Oct 17, 2025

Uh oh!

radical commented Oct 18, 2025

Uh oh!

azure-pipelines bot commented Oct 18, 2025

Uh oh!

radical commented Oct 19, 2025

Uh oh!

radical commented Oct 19, 2025

Uh oh!

radical commented Jan 21, 2026

Uh oh!

azure-pipelines bot commented Jan 21, 2026

Uh oh!

radical commented Jan 21, 2026

Uh oh!

azure-pipelines bot commented Jan 21, 2026

Uh oh!

radical commented Jan 22, 2026

Uh oh!

azure-pipelines bot commented Jan 22, 2026

Uh oh!

radical commented Jan 22, 2026

Uh oh!

radical commented Jan 22, 2026

Uh oh!

azure-pipelines bot commented Jan 22, 2026

Uh oh!

github-actions bot commented Jan 23, 2026

🎬 CLI E2E Test Recordings

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Oct 16, 2025 •

edited by radical

Loading

github-actions bot commented Oct 16, 2025 •

edited

Loading